Marqeta: Scaling MySQL Databases to 256K IOPS and 64TB across Regions Using Amazon EBS

Marqeta’s mission is to set the global standard for modern card issuing and processing. Partnering with leading mobile payment and delivery service companies, Marqeta focuses on simplifying payments for businesses with its innovative open-API solution. Customers can use the solution to customize virtual or physical cards, provision cards into digital wallets, and manage users all in one place.

Marqeta was launched in 2010, with primarily on-premises infrastructure during its early years. In 2018, Marqeta’s team started migrating from on-premises infrastructure to the cloud. By 2020, the majority of Marqeta’s databases ran on Amazon Relational Database Service (Amazon RDS), but some of the largest ones still ran on premises. As the company was experiencing a period of hyperscaling, the performance and availability of its databases needed to be improved to meet the requirements of its growing customer base. However, both the existing RDS and on-premises databases were approaching their IOPS and storage limits.

AWS solutions architects advised the company of the upcoming release of Amazon EBS io2 Block Express volumes. The volume specifications, at 256K IOPS and 64TB, met Marqeta’s performance and capacity requirements. After several rounds of prototyping and benchmarking, the volume specifications were validated. So, Marqeta began to migrate its most critical MySQL databases to Amazon Elastic Block Storage (Amazon EBS) io2 Block Express volumes.

“EBS io2 Block Express gave us the runway we needed to re-architect our application for scalability,” says Jason Zhao, principal engineer at Marqeta. “And it provided the building blocks to implement a multi-region architecture.”

After migrating the largest of its databases at the start of 2022, Marqeta saw significant improvements in that database’s overall performance and availability. Subsequently, the company migrated its remaining large databases to this new stack. The stack runs on Amazon EBS io2 Block Express volumes that are attached to EBS-optimized Amazon Elastic Compute Cloud (Amazon EC2) R5b Instances. To provide high availability and disaster recovery, Marqeta utilized multiple Availability Zones in the optimal regions for its database replicas. In 2023, Marqeta enhanced high availability and disaster recovery further by implementing multi-region architecture for its application and database stacks. Communication between its application stack and database stack is facilitated by Amazon Network Load Balancer (NLB) and ProxySQL. ProxySQL specifically handles the MySQL protocol.

As a provider of payment solutions that handles a high volume of transactions, Marqeta must uphold stringent uptime and accuracy standards. Since adopting the new solution, Marqeta has effectively mitigated database issues because of its improved IOPS performance, increased storage capacity, and multi-region high availability. “Since transitioning to EBS io2 Block Express, our on-call engineers no longer receive replication lag alerts, and we’ve mitigated the risk of hitting capacity limits,” says Zhao. “Also, since implementing multi-region, our availability number is projected to increase to 99.995%.”

Another notable outcome of this migration is that Marqeta no longer operates any database on premises. This boosts its platform engineering team’s autonomy and productivity. Previously, the team relied on other departments to provision infrastructure. The team now manages its entire stack end to end using infrastructure as code (IAC).

Next on Marqeta’s road map is the decomposition of its monolithic application and database. This is projected to be a multiyear project. “The AWS experience has been key to solving our scaling problems quickly, providing the runway to invest in re-architecture,” says Zhao. “Without it, navigating this would have been much harder.”