605 LLC Scales Analytics Applications to Measure 21 Million US Households Using Amazon FSx for Lustre

Independent TV measurement and analytics firm 605 LLC (605) drives insights from historical data to help networks, programmers, brands, advertisers, and agencies understand engagement and measure advertising effectiveness. By 2019, 605 had collected so much data that its database couldn’t efficiently scale to ingest new data, inhibiting the company’s ability to provide near-real-time insights to customers. So 605 decided to optimize its cloud architecture on Amazon Web Services (AWS) to improve efficiency and scalability and reduce costs.

Since its 2016 founding, 605 has used AWS as its primary compute and data storage environment. Originally, it stored and analyzed a smaller amount of historical data using Amazon Redshift, a cloud data warehouse, and Amazon Elastic Compute Cloud (Amazon EC2) I3en Instances, which offer low-price solid state drive instance storage. Because 605 adds hundreds of millions of rows of viewership and ad exposure data daily, it needed a more scalable solution.

To improve backend compute performance, 605 migrated its storage to Amazon FSx for Lustre, which provides cost-effective, scalable storage for compute workloads. 605 was also able to save significantly by scaling down from Amazon EC2 I3en Instances to Amazon EC2 R5 Instances, which are well suited for memory-intensive applications like near-real-time big data analytics. The company also built a data lake that holds over 4 PB of data on Amazon Simple Storage Service (Amazon S3), which offers industry-leading scalability, data availability, security, and performance. Further, to reduce deployment times, 605 migrated to Amazon Elastic Kubernetes Service (Amazon EKS), a managed container service. Migrating to this new architecture without downtime took about 2 months. Upon completion, 605 immediately saw cost savings and efficiency gains.

Using Amazon FSx for Lustre, 605 adds terabytes of data every few days. “Being able to increase high performance storage capacity by clicking a button in Amazon FSx for Lustre and being able to scale efficiently are huge benefits,” says Jared Williams, director of DevOps at 605. Further, 605 reduced deployment times from 2 hours to minutes using Amazon EC2. And by using Amazon EMR—a solution for rapidly processing, analyzing, and applying machine learning to big data—605 can spin up thousands of Amazon EC2 instances when it needs to reprocess huge datasets.

On AWS, 605 cut costs by 50 percent without decreasing performance. Using Amazon EKS, the company can scale simply and ingest data into its backend storage more quickly, using this historical data to prove to its customers that their advertising is working. “The ability to cost-effectively deliver at scale on AWS is existential for us,” says Dan Berg, chief technology officer of 605. Because AWS manages the infrastructure, 605’s engineers can focus on innovating, and the 605 team can compete with larger companies. Additionally, 605 has gained a more granular look at its cloud environment, which improves security and cost management.

On AWS, 605 enhanced performance and scalability while cutting costs, driving better insights for its customers. The company also uses machine learning to better pinpoint data drops and other anomalies in its incoming source data. “A key part of what we offer to customers is sophisticated data analytics and measurement,” says Berg. “Without the ability to scale infrastructure dynamically on AWS, we wouldn’t be able to do that.”