Depop on How It Evolved from a Data Swamp to a Data Lake on AWS

Guest post by Alexej Tessaro, Lead Platform Engineer at Depop

Founded by Simon Beckerman, the co-founder of PIG magazine and RETROSUPERFUTURE sunglasses, in 2011, London-based startup Depop provides a social marketplace where customers can buy, sell and discover unique items based on what their friends and favorite influencers are digging at the moment.

Such a network naturally generates a large amount of data. Every second on the app, users follow and message each other, new products are published, sold, commented and liked, and we at Depop need to be able to process that data quickly and efficiently as each event provides the company with invaluable intelligence necessary for making informed decisions around business strategy, operations, marketing, product development, and more.

To do this, we’ve needed to define a data architecture that supports our business structure and needs, and built a reliable, resilient and high-performing data pipeline that does not affect performance and availability of those components that make Depop function.

As background, we originally only had primary data stores, specifically relational and key-value databases. Every team in the organization that needed production information, was running

heavy queries against such databases. Our marketing, operations and product teams were monitoring users’

activity and gathering statistical insights, using business intelligence tools that were throwing heavy and complex SQL queries while at the same time our users were enjoying our iOS and Android apps and our website.

This was a recipe for failure. In fact, we had quite a few outages related to this (non)architecture and with a constantly-growing user base, scaling access to our data layer became even more challenging. The team ultimately decided to go with the concept for a Data Lake after we decided we didn’t want to manage schema updates. You can read more about what we learned while defining a data architecture that supports our business structure and needs here.