Intuit Migrates Apache Cassandra Workloads to Amazon Keyspaces with High Availability and Reliability
The Intuit data exchange (IDX) platform team acquires and consolidates data from thousands of financial institutions to support data needs across Intuit products for millions of customers. The team previously relied on Apache Cassandra as its near-real-time data store and had a few key goals outlined around operational efficiency gains and business agility for its target state. Additionally, it wanted the data store to scale elastically for additional workloads and peak traffic conditions with minimal latency to minimize performance impact for customers.
The team chose to boost speed and productivity by migrating to a managed solution from Amazon Web Services (AWS). That solution was Amazon Keyspaces (for Apache Cassandra), a scalable, highly available, and managed Apache Cassandra–compatible database service. By strategically migrating to Amazon Keyspaces, the IDX team delivered high availability and reliability to its users both during and after the migration. Also, using Amazon Keyspaces, the team has increased staff productivity and enhanced its elastic scaling and data recovery capabilities.
In our prior state, if we had to scale out our cluster for more capacity, we would need a lead time of a few weeks. Now, using Amazon Keyspaces, we can accomplish this in 1 day.”
Software Engineering Leader, Intuit
Migrating to Simplify Maintenance and Enhance Productivity
Intuit is a global technology platform that helps consumers and small businesses overcome their most important financial challenges. At Intuit, the IDX team builds and delivers foundational platform capabilities for all the company’s products, such as TurboTax, QuickBooks, and Mint. Since 2017, the IDX team has relied on Apache Cassandra to store data for financial services and entities. Previously, this team relied on full-time engineers with expertise around Apache Cassandra administration and cluster operations to manage the underlying infrastructure. The IDX team was keen to remove effort around managing database infrastructure both for productivity gains and to focus more on delivering business value from its data.
The team at Intuit chose Amazon Keyspaces so that it could meet its operational efficiency goals while retaining its Apache Cassandra–based functionality. “We were keen to implement an innovative, cloud-native managed service for persistence and so that we can seamlessly scale up or down for variance in traffic using features that are on par with Apache Cassandra,” says Manoj Mohan, software engineering leader at Intuit. “That’s where using Amazon Keyspaces made sense.”
The Intuit team put together a robust plan for functional and performance evaluation followed by a complete implementation plan for migrating their Apache Cassandra workloads. The team came up with innovative strategies including implementing a dual-write approach: it continued persisting data to Apache Cassandra while also persisting to Amazon Keyspaces asynchronously in the background. Using this approach, the team was able to simulate its actual production workloads and optimize its serverless capacity needs on the new solution without affecting the customer experience. “To be able to pull off such a complex initiative involving large-scale data migration as well as database overhaul with zero downtime and without any adverse customer impact has been truly phenomenal,” says Mohan. “We were able to solve issues proactively and encapsulate all of this complexity under the hood with the engine running!”
During the entire life cycle of the project, Intuit’s team worked with AWS engineers and the Amazon Keyspaces team from AWS to make the migration successful. Working alongside AWS, the Intuit team was able to develop feature parity, plan onboarding, and receive prompt support when needed—as well as receive two performance-related enhancements delivered by the Amazon Keyspaces team. “From beginning to end, the support from AWS was great,” says Jason Lashmet, staff software engineer at Intuit.
Delivering Efficiency and Reliability While Accelerating Data Recovery
With a small team of engineers, Intuit seamlessly migrated more than 120 TB of data distributed across a 66-node Apache Cassandra cluster. In March 2021, Intuit’s solution on Amazon Keyspaces went live. On Amazon Keyspaces, Intuit was able to expand reliability and data availability for its customers, whereas the legacy system had experienced occasional time-outs. On Amazon Keyspaces, Intuit has delivered users a consistent experience without any production incidents. Furthermore, Intuit’s team members no longer need to dedicate time for Apache Cassandra maintenance and operations (including handling lower-level database management tasks, data compaction, read repair, and tombstone removal).
On Amazon Keyspaces, the IDX team has enhanced the elasticity of its workloads, facilitating the ability to bring on additional workloads and get them to market quickly. “In our prior state, if we had to scale out our cluster for more capacity, we would need a lead time of a few weeks,” says Mohan. “Now, using Amazon Keyspaces, we can accomplish this in 1 day.” Intuit’s expanded scaling capabilities also make adding workloads simpler for the team, and it extends those growth benefits to users.
The Intuit team also relies on features of Amazon Keyspaces to deliver security and to enhance its data recovery process. By using the service, which encrypts data by default, Intuit can facilitate data encryption at rest and in transit. Also, to improve its recovery times, the data exchange team uses point-in-time recovery from Amazon Keyspaces. Using this feature, which provides continuous backups, the team can immediately restore its Amazon Keyspaces data to the second.
Expanding Database Solutions on AWS
Now that it has boosted its operational efficiency and scaling capabilities, Intuit plans to continue using Amazon Keyspaces and adopt it for similar use cases. For data entities that are better suited to a relational database service, Intuit is exploring other solutions on AWS. “From our experience, this went about as well as a project could possibly go in terms of the migration itself and the outcomes,” says Lashmet. “For any suitable use case, I would absolutely recommend Amazon Keyspaces.”
Serving more than 100 million customers worldwide with TurboTax, QuickBooks, Mint, Credit Karma, and Mailchimp, Intuit is a global technology platform that helps consumers and small businesses overcome their most important financial challenges.
Benefits of AWS
- Migrated more than 120 TB of data without production impact
- Delivered highly available data services with no errors or production incidents
- Removed the need for full-time engineers for infrastructure maintenance
- Scales out capacity in 1 day rather than a few weeks
- Restores data from backups virtually instantly
AWS Services Used
Amazon Keyspaces (for Apache Cassandra) is a scalable, highly available, and managed Apache Cassandra–compatible database service. With Amazon Keyspaces, you can run your Cassandra workloads on AWS using the same Cassandra application code and developer tools that you use today.
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.