Muthoot Fincorp Reduces Platform Costs by 61% with Ganit and AWS

Muthoot Fincorp provides customers with more than 20 different financial services, and the data behind these offerings lives on Amazon Web Services (AWS). In Muthoot Fincorp’s previous infrastructure, data was pushed to Amazon Simple Storage Service (Amazon S3) from an on-premises SQL Server database using AWS Database Migration Service (AWS DMS). Multiple AWS Glue ETL jobs processed the data to read it in Amazon Athena, write it to Amazon S3, then push it to Amazon QuickSight dashboards for data analysis.

However, with this workflow, 25 years’ worth of data was being overwritten on a daily basis, which took hours to refresh. AWS Glue jobs were rewriting the entire dataset during processing cycles. This led to high spikes in Amazon Athena and Amazon S3 costs each time queries scanned the dataset during a refresh cycle. As data, usage, and users increased, so did costs, and Muthoot Fincorp realized it needed a way to scale operations to manage data more efficiently and cost-effectively.

Muthoot Fincorp reached out to AWS Advanced Services Partner Ganit to review its data architecture and recommend opportunities for optimization. A thorough evaluation from Ganit determined that appropriate AWS resources were being leveraged, but processes could be better optimized to take advantage of cloud capabilities for scalability, storage, and automation.

Ganit discovered the existing Amazon QuickSight configuration for data retrieval was causing latency when loading reports. The dashboards were reading data from Amazon S3 through Amazon Athena, and large datasets were causing huge read volumes. The previous setup with AWS DMS allowed for only a full load of data, and this data was then processed using AWS Glue. This often caused bottlenecks and outages due to the large data processing jobs timing out. Instead of moving all its on-premises data to the cloud on a daily basis, Ganit configured AWS Glue workflows to load incremental data only. This dramatically reduced the overall load. The new AWS Glue jobs minimized the amount of manual data transformation, reducing processing resources and the potential for data load failures. The incremental data stored in Amazon S3 was then partitioned and saved by date into subfolders, making it easier to quickly find and read.

A key piece of Ganit’s solution was leveraging Amazon QuickSight’s Super-fast, Parallel, In-memory Calculation Engine (SPICE). Muthoot Fincorp utilizes nearly 400 dashboards in Amazon QuickSight to view activities such as total business volume and services used by customers. By summarizing and moving Muthoot Fincorp’s datasets into SPICE, report queries processed faster, and data could be reused without incurring additional costs. Previously, complex joins from reports were accessing tables in Amazon Athena, which took about 10 minutes to load. Once the datasets were summarized in SPICE, loading reports took only a few seconds. SPICE also enabled automated scanning whenever new data was populated or refreshed. Sony Soman Pillai, head of analytics at Muthoot Fincorp, noted, “The SPICE functionality in Amazon QuickSight was a gamechanger. The service scales up or down as needed to accommodate fluctuating volumes of new data, and removed the need for our engineering teams to manually refresh and load data.”