Modernizing Big Data Processing Using Amazon EMR with Yahoo Advertising

Yahoo Advertising is the advertising division of Yahoo, a trusted guide for hundreds of millions of people globally. Founded in 1994, Yahoo is well known as one of the pioneers of the early internet era and has since grown to offer a range of products and services, including digital advertising solutions. Renowned for its work in big data, the company also created Apache Hadoop, an open-source framework used to process large datasets and perform complex computations.

In 2022, Yahoo Advertising chose AWS as its preferred cloud provider, marking the start of a major digital transformation. Yahoo Advertising plans to migrate all its advertising technology workloads—including its analytics, identity solutions, and media-buying environments—to AWS over the next few years. For big data processing, the company chose to migrate its analytics pipelines from on-premises infrastructure to Amazon EMR, a cloud big data solution for petabyte-scale data processing, interactive analytics, and machine learning.

Using Amazon EMR, Yahoo Advertising realized that it could scale its big data processing capabilities to manage larger amounts of data. And, without the limitations of on-premises hardware, adopting this technology would facilitate innovation and exploration of new use cases.

“Because we have multiple teams running on AWS, we have already used Amazon EMR even before the digital transformation,” says Zilvinas Saltys, senior principal architect at Yahoo. “We knew that this technology was sensible, it worked for us, and it was cost-effective.”

The contextual targeting team at Yahoo Advertising was one of the first to complete its migration. This unit is responsible for verifying that advertisers deliver ads to the right audience, at the right time. “Our job is to identify the websites that are suitable for displaying ads and to build data pipelines that gather and compile information about these websites for our advertising network,” says Saltys. “This process involves handling large amounts of data. We receive samples of this data, not the entirety, which can amount to hundreds of gigabytes every 5 minutes. This data is crucial for our contextual targeting pipelines.”

Instead of opting for a lift-and-shift migration, the contextual targeting team completely optimized and modernized its environment on AWS. It uses Amazon EMR to run various types of jobs based on open-source frameworks, such as Apache Flink and Apache Spark. To schedule Amazon EMR jobs and facilitate cluster creation, the team adopted Amazon Managed Workflows for Apache Airflow (Amazon MWAA), a secure and highly available managed workflow orchestration service for Apache Airflow.

The contextual targeting team stores its data on Amazon Simple Storage Service (Amazon S3), an object storage service built to retrieve virtually any amount of data from anywhere. All inputs and outputs from the pipelines are also directed toward Amazon S3. To query and interface with this data, the team relies on Amazon Athena, a serverless, interactive analytics service, and AWS Glue, a serverless data integration service—effectively replacing its on-premises Apache Hive database.

By migrating to AWS, the contextual targeting team unlocked several benefits. Team members now have access to modern frameworks and recent versions of their software on AWS. Upgrades and updates, which previously might have been lengthy and complex processes, can now be completed in about 1 hour.

The migration has also contributed to a more dynamic work environment. On AWS, team members no longer need to manage or configure hardware infrastructure. This migration empowers them to work more independently, manage their own product development, and decide on their required capacity and software versions—driving innovation and efficiency.

“It’s an absolutely different dynamic of how our data pipelines work,” says Saltys. “On AWS, we’re able to work more independently and effectively without centralized teams, empowering us to upgrade, scale, and resolve issues more quickly.”

As Yahoo Advertising finalizes the migration of its advertising analytics pipelines to AWS, the impact of this modernization will be significant. More teams will continue to incorporate Amazon EMR and other technologies into their workflows, which will set the tone for how the organization processes, visualizes, and works with big data as a whole.

“Migrating to AWS has introduced new technologies and new ways of working, and it has made existing processes simpler and more visible,” says Adesh D’Silva, senior software engineer at Yahoo. “This migration has been of great benefit to our team.”