Mercado Libre Scales Its Business, Improves Reliability in Migrating 5,000 Databases to Amazon DynamoDB
With nearly 79 million active users in 18 different countries using its businesses, Mercado Libre needed to standardize and simplify its data management for thousands of core applications. To support a continuous increase in traffic and storage and further its commitment to being a data-driven organization, Mercado Libre decided that Amazon Web Services (AWS) was, for thousands of databases, its best choice to achieve performance at scale, especially because of the managed database services that AWS offers.
As part of the company’s overhaul, a Mercado Libre team of six engineers migrated 5,000 databases to Amazon DynamoDB—a fast, flexible NoSQL database service delivering single-digit millisecond performance at any scale. Using the new solution, the company saved developer time, improved reliability, and reduced costs.
When we analyzed alternatives to our previous setup, for what we needed, Amazon DynamoDB was a clear winner for us. And we continue to get value from it."
Director of IT and Head of Core Services, Site Reliability Engineering, and Database Administration, Mercado Libre
Building a More Reliable Infrastructure on AWS
Founded in Argentina in 1999, Mercado Libre features two main businesses: ecommerce and fintech. Within those realms, the company offers many services, including an online marketplace, a digital payment system, advertising, logistics, and credit offerings. Its underlying purpose is to democratize commerce, payments, and money to make a difference in the lives of millions of people in Latin America.
Mercado Libre had originally built its solutions on a virtual private cloud, with each internal team responsible for handling its own infrastructure, applications, and maintenance. Individual teams determined, for example, which types of databases to use. Although this was key for fast growth at first, as the number of teams grew and the traffic increased, the variety of technology stacks made it difficult to sustain growth. The company built its teams a new solution that included a proprietary software development kit and custom APIs. For NoSQL databases, instead of providing direct access to any solution, the company built a layer of abstraction so that all teams were speaking the same language. Initially, Mercado Libre built its key-value main database abstraction on top of a self-managed, open-source database engine, a solution that grew to 5,000 databases.
But over time, the company realized it wasn’t getting all the features it needed from the selected engine. Plus, managing over 30 clusters required frequent maintenance, resulting in timeouts, errors, and downtime incidents. In early 2019, Mercado Libre began its migration to Amazon DynamoDB. “With the reliability of Amazon DynamoDB, we can avoid the cost of downtime,” says Oscar Mullin, Mercado Libre’s director of IT and head of core services, site reliability engineering, and database administration. Using the common abstraction to its advantage, the company migrated 5,000 databases without requiring anyone in the organization to change a single line of code or alter the business road map.
Mercado Libre used Amazon DynamoDB alongside other AWS services that Mercado Libre was using to run its mission- and business-critical applications, such as Amazon Elastic Compute Cloud (Amazon EC2), a web service that provides secure, resizable compute capacity in the cloud, and Amazon Simple Storage Service (Amazon S3), an object storage service that offers industry-leading scalability, data availability, security, and performance.
Simplifying Data Management Using Amazon DynamoDB
Using the combined power of AWS solutions, Mercado Libre was able to handle all of its scaling needs without impacting performance. Previously, the company spent significant time preparing for scaling events such as holidays, and scaling was complex. And as a result of unexpected events such as the COVID-19 pandemic, more people have turned to ecommerce and digital payments, which significantly increased traffic for Mercado Libre. As part of its mission of inclusion, Mercado Libre aimed to use AWS to extend its offerings to more small businesses. “Having our core applications using our abstraction based on Amazon DynamoDB was critical,” says Mullin. “Coping with that traffic increase while keeping the level of availability would have been extremely complex otherwise.” The automatic scaling capabilities of Amazon DynamoDB also proved invaluable when the company needed to downscale, helping it prevent unused capacity, reduce waste, and cut costs. “Unlike on our previous solutions, downscaling on Amazon DynamoDB is extremely simple,” says Mullin.
The multi–AWS Region architecture supported by Amazon DynamoDB also solves availability issues for Mercado Libre. Instead of having to transfer data found in separate tables, the company uses Amazon DynamoDB global tables, which replicate tables automatically across a company’s choice of Regions for multi-Region fault tolerance. This fully managed, multi-Region, multiactive database helps deliver fast, local read/write performance to millions of globally dispersed customers. “We have the ability to be multi-Region with a single click,” says Mullin. “And all our historical data moves to the other Regions automatically.” Changes to one table seamlessly propagate to the other Regions, saving time for the company. No matter where users are located, they can access data with low latency. In fact, the ninety-ninth percentile of combined latency of API and Amazon DynamoDB reads is just 6 ms for read requests. And even as requests reach a rate of 300 million per minute, latency doesn’t suffer. Plus, Mercado Libre no longer has to navigate complex processes just to get a daily backup because Amazon DynamoDB offers point-in-time recovery for continual, automatic backups.
Mercado Libre’s approach to services includes building interrelated services that provide its developers with transparent integrations that are as simple as possible. To achieve this integration with the key-value store, Mercado Libre turned to Amazon DynamoDB Streams, which captures a time-ordered sequence of item-level modifications in any Amazon DynamoDB table and stores the information in a log for 24 hours. Also, DynamoDB Streams works with Amazon DynamoDB in automatically removing irrelevant information from tables, all for no additional cost. Mercado Libre can use Amazon DynamoDB to delete these expired items using time-to-live expiration, which removes unwanted or irrelevant items in the background without consuming any write throughput. And by using DynamoDB Streams, the company can capture these deletions and back up or otherwise process data as it expires. “DynamoDB Streams is a powerful feature,” says Mullin. “Using its automated capabilities removes the need for us to do many tasks ourselves, a process that was complex and error prone.”
Driving Growth for Regional Businesses
Although Mercado Libre originally selected Amazon DynamoDB as a way to off-load operational overhead and support massive scale, its developers are continuing to unlock new capabilities, driving the company’s ability to innovate and continue to grow in Latin America. “When we analyzed alternatives to our previous setup, for what we needed, Amazon DynamoDB was a clear winner for us,” says Mullin. “And we continue to get value from it by building new abstractions like secondary indexes or security capabilities.”
Figure 1: Mercado Libre Migration Flows
Click to enlarge for fullscreen viewing.
Figure 2: Mercado Libre Traffic Growth
Click to enlarge for fullscreen viewing.
About Mercado Libre
Mercado Libre hosts the largest online commerce and payments ecosystem in 18 countries in Latin America. Mercado Libre’s efforts are centered on enabling ecommerce and digital and mobile payments on behalf of its customers by delivering a suite of technology solutions across the complete value chain of commerce.
Benefits of AWS
- Reduced incidents by providing automatic scaling features
- Reduced costs by providing simple downscaling features
- Generated combined latency of 6 ms for the ninety-ninth percentile of read requests
- Gained ability to propagate time-to-live expirations in a simple way using Amazon DynamoDB
- Upgraded reliability of databases
- Improved availability for customers by using multi-Region architecture
AWS Services Used
Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.
Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.
Companies of all sizes across all industries are transforming their businesses every day using AWS. Contact our experts and start your own AWS Cloud journey today.