idealo Increases Traffic 6x for Black Friday Using MongoDB Atlas on AWS

German-based idealo launched in 2000 to provide price comparison services to consumers. Using a combination of site scraping and data provided by merchants, the company grew substantially to where it was presenting 2.2 billion offers each month to its 72 million monthly visitors. These offers featured the goods and services of more than 50,000 merchants and included household goods, electronics, flights, and hotel stays. Handling that traffic was becoming increasingly challenging. The company was running its systems on premises, using MongoDB to help it serve millions of visits per day. But it was clear that its on-premises system could not scale to keep pace with demand, as it was processing 160,000 requests per second.

There were three important components that all needed to be optimized to support idealo’s continued growth. First was compute resources, second was the ability to move information around within its data systems, and third was storage of that data. The company was operating an on-premises data center and running on MongoDB because one of idealo’s main requirements for a database was scale. Relational databases could not provide it with the scale and flexibility needed for its business. On the other hand, MongoDB, as a NoSQL and document-based database was better suited, especially its sharding capabilities, which provided idealo with resilience and capacity. “Sharding can be thought of as a horizontal scaling, where multiple nodes are created to store and distribute data, providing redundancy and easing congestion,” says Robin Kaluzny, head of technology at idealo. “It’s very powerful and worked well for us.”

idealo’s traffic has increased by 6 times in the last 4 years, with an expected continued growth of 20 percent per year. As a result, it was looking at having to renew and expand its on-premises data center. Because idealo helps consumers find the best shopping deals, it experiences spikes in demand, such as major holidays, especially on Black Friday. Scaling its self-hosted MongoDB environments to meet that demand was taking 2–3 months. The company wanted to reduce both the cost of acquiring and running hardware, and the maintenance burden on staff. The company also had the challenge of recruiting and retaining enough qualified staff and committing them to running the data center and servers, rather than focusing on the core business. These reasons all supported a migration to AWS.

idealo saw the migration to AWS as an opportunity to focus on its core business of providing great offers to consumers and providing businesses with increased sales. In addition to migrating to the cloud, it wanted to reallocate its database administration team to other activities. So when it migrated to AWS, it also migrated to MongoDB Atlas, an integrated suite of data services centered around a cloud database designed to accelerate and simplify how organizations build with data, and chose AWS Partner MongoDB to manage it. “Because we couldn’t commit the resources to code and operate the huge clusters we need for our data, we were looking for partners,” says Kaluzny. “Obviously MongoDB Atlas is the best choice in that field. They had the product we needed and the expertise to manage it for us.”

The company worked with MongoDB to create a MongoDB Atlas solution that provided the compute, communication, and storage resources required to meet current business needs with the capability to scale as needed. “It was important to understand idealo’s needs as a business so the migration would deliver benefits,” says Oliver Wedell, regional vice president central europe at MongoDB. “We worked with them to migrate to MongoDB Atlas on AWS and AWS was involved to support the use of its services. Everyone contributed to the success of the migration.” Through the AWS ISV Workload Migration Program, MongoDB and AWS simplified the migration process and reduced costs for migrating on-premises workloads to the cloud.

While MongoDB Atlas is at the heart of idealo’s business, it uses AWS services as well, including Amazon Elastic Kubernetes Service (Amazon EKS)—a managed Kubernetes service to run Kubernetes in the AWS cloud and on-premises data centers—to reduce costs with efficient compute resource provisioning and automatic Kubernetes application scaling. It also uses AWS Cloud Development Kit (AWS CDK), which accelerates cloud development using common programming languages to model its applications. “The professionalism of MongoDB was outstanding,” says Kaluzny. “The move from the data center to the cloud with multiple clusters, integrated with AWS, was something we didn’t have to worry about. MongoDB was just what we wanted in a technology partner.”

The migration to MongoDB Atlas has significantly reduced the resources idealo needs to operate its service. It was able to reduce the number of shards it needed from 25 to 12 and reduce the size of each node by about 66 percent. “We cut the resources needed to run the service by about 84 percent. It’s remarkable what MongoDB has been able to help us do,” says Kaluzny.

In addition to reducing the number of shards needed and the size of the nodes, the system gained efficiency by implementing bulk operations on documents, rather than single-document processing. idealo engineers also optimized schema and indexes to further improve performance. At peak load, it can support up to 200,000 queries per second and 60,000 updates per second. It has been measured supporting 150,000 queries per second for more than 14 hours straight. The system is processing the same number of offers post migration as it did before but with optimized performance and reduced cost through refactoring and help from the MongoDB Support as well as the MongoDB Professional Services team.

idealo has improved scalability and can spin up and spin down resources in response to demand, rather than overprovisioning a data center to handle peak demand. It has decommissioned its on-premises data center and is 100 percent on AWS. “An on-premises data center doesn’t make sense,” says Kaluzny. “Part of it is the cost of acquisition and maintenance, but for us the real deciding factor was that on premises was getting in the way of our growth strategy. With MongoDB Atlas on AWS, we only pay for what we need when we need it, we’re getting better insight on workloads, and it has eliminated any performance bottleneck.”

The migration to MongoDB Atlas has positioned idealo to scale to meet its projected 20 percent annual growth. It has also simplified the training of staff who benefit from automation of repetitive tasks to focus on higher-value activities. The company also benefits from the strong security and compliance support in MongoDB Atlas to ensure it meets all requirements in the different countries in which it operates. “Finding the right product for our business needs was important, but the expertise and professionalism of MongoDB managing it has taken it to a higher level,” says Kaluzny. “We can just focus on running the business instead of managing infrastructure. That’s worth a lot.”