How Woot uses Amazon DocumentDB (with MongoDB compatibility) to scale deals

Woot! is the original daily deals site, founded in 2004 and snatched up by Amazon in 2010. Originally, Woot offered just one product per day until it sold out. Today, Woot offers special daily deals and other limited-time offers across seven categories. At Woot, my team is responsible for the Retail Catalog service, which provides deals to Woot’s various front-end services and user interfaces. As Woot has grown over the years, many of our problems have come from aging legacy services that are hitting scaling limits.

In this post, I’m going to share with you how we migrated our self-hosted MongoDB database that powers our product catalog and the deals you see on Woot to Amazon DocumentDB (with MongoDB compatibility). I also cover how Amazon DocumentDB helped us improve our performance, scale, and agility while decreasing costs.

Application architecture

The Woot Retail Catalog service uses Amazon DocumentDB as its primary database. Within Amazon DocumentDB, the data is organized among collections for items (products that Woot sells), offers (items for sale, limited by time or quantity), and events (groups of offers marketed to customers). Our backend systems primarily use C#/.Net, a mix of legacy .Net Framework Windows services, and a serverless .Net Core microservice stack. A Windows IIS API currently provides CRUD operations on top of the Amazon DocumentDB cluster, and we’re in the process of migrating that service to an AWS AppSync GraphQL instance, as shown in the following diagram.

Why Amazon DocumentDB?

Up until the Spring of 2019, Woot self-hosted a MongoDB cluster on Amazon Elastic Compute Cloud (Amazon EC2). Our migration goal was to get to a fully managed database solution and reduce costs. We chose Amazon DocumentDB because our data maps naturally to the document model, we had already invested in the MongoDB API, and we didn’t want to rewrite our application.

One of the key motivations for the migration was to reduce costs, and we had a tight time frame to migrate before we would be charged a licensing fee for the next year. Given that Amazon DocumentDB is MongoDB API compatible, we could complete the migration with little changes to our application, and we hit that deadline with time to spare. We were able to migrate with no downtime, which made migrating to Amazon DocumentDB the cheap and easy solution.

Benefits

The savings of migrating to Amazon DocumentDB were substantial because we could save significantly on infrastructure and operational costs. With Amazon DocumentDB, we cut our infrastructure costs for the database by 82%, including license costs. Plus, we no longer have to negotiate and revisit licensing fees every year, which saves a lot of hassle and approvals.

When we were self-managing our MongoDB cluster, we had challenges and occasional disruptions to the service when we needed to scale. With Amazon DocumentDB, we have noticed that scaling read throughput is significantly easier and more reliable. We can scale up our cluster in less than 10 minutes, which used to take days to complete. We also have the flexibility to easily scale back down when resources are no longer needed, which further helped us optimize for costs.

Additionally, our self-hosted MongoDB cluster wasn’t being backed up adequately, and we previously suffered an extended outage when an important collection was accidentally dropped with no recent backup to roll back to. With Amazon DocumentDB point-in-time recovery (PITR) enabled by default, we have the peace of mind to know that if a situation like that happens again, we can roll it back quickly because we extended our PITR window to 35 days.

Performance has also been excellent, and we’ve seen improved latency and scalability. On our legacy MongoDB cluster, we had to be careful about how code changes and new features would affect the load on the database, and often relied on hacky solutions to avoid changing the database layer. With Amazon DocumentDB, we are able to achieve better performance, optimize our overall database utilization, and develop and deploy new features faster, which enabled us to innovate faster on behalf of our customers.

Migration lessons learned

In preparation for the migration, we over-scaled our Amazon DocumentDB cluster to err on the safe side. We chose a larger instance than we needed, knowing it would be easy to scale it back down to the right size after migration. To perform the migration, we made changes to our application to perform dual writes on both the source MongoDB database and target Amazon DocumentDB cluster. Doing so enabled us run both environments in parallel for an extended period (a few weeks) to validate our migration. We were able to migrate successfully with no downtime.

If we were to perform the migration again, we would likely use Amazon Database Migration Service (AWS DMS) instead. Although we had no issues with our migration, AWS DMS would have reduced the development effort required because it supports ongoing replication of data, something we manually coded.

Summary

By migrating our self-managed MongoDB database to Amazon DocumentDB, Woot cut costs by 82%, increased performance and scalability, and reduced operations. With Amazon DocumentDB, we can iterate faster and deliver more value to our customers.

About the Author

Sam Giagtzoglou is a Software Engineer at Woot! He has experience designing, building and guiding design for high-availability, low-maintenance retail solutions on top of AWS serverless services. He is currently leading a huge redesign of Woot’s Catalog backend systems. He holds a B.A. in Computer Science from Wesleyan University.