Muthoot Fincorp Reduces Platform Costs by 61% with Ganit and AWS

Muthoot Fincorp provides customers with more than 20 different financial services, and the data behind these offerings lives on Amazon Web Services (AWS). In Muthoot Fincorp’s previous infrastructure, data was pushed to Amazon Simple Storage Service (Amazon S3) from an on-premises SQL Server database using AWS Database Migration Service (AWS DMS). Multiple AWS Glue ETL jobs processed the data to read it in Amazon Athena, write it to Amazon S3, then push it to Amazon QuickSight dashboards for data analysis.

However, with this workflow, 25 years’ worth of data was being overwritten on a daily basis, which took hours to refresh. AWS Glue jobs were rewriting the entire dataset during processing cycles. This led to high spikes in Amazon Athena and Amazon S3 costs each time queries scanned the dataset during a refresh cycle. As data, usage, and users increased, so did costs, and Muthoot Fincorp realized it needed a way to scale operations to manage data more efficiently and cost-effectively.

Muthoot Fincorp reached out to AWS Advanced Services Partner Ganit to review its data architecture and recommend opportunities for optimization. A thorough evaluation from Ganit determined that appropriate AWS resources were being leveraged, but processes could be better optimized to take advantage of cloud capabilities for scalability, storage, and automation.

Ganit discovered the existing Amazon QuickSight configuration for data retrieval was causing latency when loading reports. The dashboards were reading data from Amazon S3 through Amazon Athena, and large datasets were causing huge read volumes. The previous setup with AWS DMS allowed for only a full load of data, and this data was then processed using AWS Glue. This often caused bottlenecks and outages due to the large data processing jobs timing out. Instead of moving all its on-premises data to the cloud on a daily basis, Ganit configured AWS Glue workflows to load incremental data only. This dramatically reduced the overall load. The new AWS Glue jobs minimized the amount of manual data transformation, reducing processing resources and the potential for data load failures. The incremental data stored in Amazon S3 was then partitioned and saved by date into subfolders, making it easier to quickly find and read.

A key piece of Ganit’s solution was leveraging Amazon QuickSight’s Super-fast, Parallel, In-memory Calculation Engine (SPICE). Muthoot Fincorp utilizes nearly 400 dashboards in Amazon QuickSight to view activities such as total business volume and services used by customers. By summarizing and moving Muthoot Fincorp’s datasets into SPICE, report queries processed faster, and data could be reused without incurring additional costs. Previously, complex joins from reports were accessing tables in Amazon Athena, which took about 10 minutes to load. Once the datasets were summarized in SPICE, loading reports took only a few seconds. SPICE also enabled automated scanning whenever new data was populated or refreshed. Sony Soman Pillai, head of analytics at Muthoot Fincorp, noted, “The SPICE functionality in Amazon QuickSight was a gamechanger. The service scales up or down as needed to accommodate fluctuating volumes of new data, and removed the need for our engineering teams to manually refresh and load data.”

By changing the way data was loaded into AWS, Muthoot Fincorp saw a major reduction in processing times. Amazon Athena no longer reads Muthoot Fincorp’s entire historical data, reducing data processing time by 76%. By loading only incremental data, workflows are completed in much less time. Previously, data engineers spent seven hours loading all new data each day. After implementing Ganit’s solution, the same process takes just one and a half hours.

Utilizing SPICE for Amazon QuickSight not only helped process data faster, but also dramatically reduced platform costs. Because data was being processed more efficiently and at much lower volumes via SPICE, Muthoot Fincorp saw platform costs drop by 61%. The company was no longer constantly overwriting its entire stored dataset, and processing only a single day’s worth of data reduced daily data writes to the data storage layer by 99%.

The new architecture enabled faster reporting so that Muthoot Fincorp executives could develop additional reports, such as comprehensive views of a customer’s business data. Employees can use these reports during customer interactions to provide a personalized view of customer activity, informing conversations and improving customer experiences. Faster reporting also facilitated an increase in the volume and frequency of marketing campaigns.

Muthoot Fincorp continues to work with Ganit to further the evolution of its data processing capabilities. It’s currently working on rearchitecting its data pipeline with AWS services. The company continues to explore alternative database options to enhance speed and scalability and incorporate advanced analytics. As Pillai concludes, “Working with Ganit feels like a truly collaborative effort. They’re so engaged and have helped us achieve an optimized experience without compromising performance.”