Optimizing Data Infrastructure for a Fintech Leader

Fintech

BUSINESS CHALLENGE

The client is cited as the industry’s largest commercially available database of securitized mortgages.

• Since they collect data from various providers, they needed a scalable infrastructure to process these data streams to create client-based products that could serve their customers through API and BI Tools.
• This called for a scalable platform to store all the structured and unstructured data.
• The client was looking for a cost-effective solution for the implementation of a new pipeline.

OUR SOLUTION

We made quality deployments in time while monitoring and maintaining the running services and tackled the use cases with innovative workflows.

• We adopted Data Lake with AWS Lake Formation and Glue to eliminate the need to model data into an enterprise-wide schema. With the increase in data volume, data quality, and metadata, the quality of analyses also increases.
• Many existing processes were run through spark jobs on AWS EMR, leading to increased server costs. So, we switched to a newer graviton-based instance type which uses 64-bit Arm Neoverse cores to provide the highest price performance and scalability with minimal latency and cost.
• With the introduction of new processes, we had to keep up with more Jenkins job creations and executions. This created a cluttered Jenkins Server and became a bottleneck for multiple urgent deployments due to the server’s limited computing resources. So, we completely migrated all the existing Jenkins jobs into Github action workflows, automating the deployments after every commit. This made deployment, monitoring and debugging a lot easier and this reduced the time for making quality deployment releases by 40-50%.
• To maintain data security, alerts are monitored and resolved by Trusted Advisor, and sensitive information is stored in AWS Parameter Store.