Hudl Scales Video Processing and Boosts Reliability by Optimizing on Amazon EC2 Spot Instances

To continue its mission to “make every moment count,” Hudl, a global sports video and analytics company, needs to provide its customers with a reliable video uploading experience. In March 2021, Hudl set out to modernize its almost 10-year-old ingest and transcoding infrastructure. Hudl had been using Amazon Web Services (AWS) since 2009, so when it decided to migrate from its self-managed databases to a more scalable, fault-tolerant infrastructure, the company knew it would find the right solution using AWS. Now Hudl is realizing increased reliability and scalability as well as cost savings on video encoding processes.

More than 180,000 sports teams depend on Hudl to deliver video and analytics, including all 30 NBA teams, 20 teams in the Premier League, and 18 teams in Bundesliga, so it’s important that Hudl’s upload infrastructure provides a consistent experience to customers—including during peak times. The company needs 10,000 servers for 4 hours every Friday night during high school football season; however, the scaling protocols of Hudl’s self-managed databases weren’t handling the job at the level that Hudl needed. “They weren’t scaling,” says Andy Pryor, director of platform engineering at Hudl. “We were bursting at the seams.”

Hudl began to explore ways it could better control its operational burdens and costs. The company decided to rearchitect its ingest and transcoding infrastructure. “We wanted to migrate our transcoding workloads to relieve some of the operational burden around deciding how many servers to spin up at any given time and which kinds of servers to use,” says Jordan Degner, director of engineering for the core video team at Hudl. “We wanted to make things very interrupt friendly.”

Hudl reached out to AWS for support optimizing its use of AWS solutions. After weighing the options, the company decided to adopt infrastructure as code. “We discovered we could save a lot of money and also solve this operational problem,” Pryor says. “We can now provision the servers at the scale we want globally and automatically deliver video faster to users.”

Hudl relies on Amazon Elastic Compute Cloud (Amazon EC2), which provides secure, resizable compute capacity in the cloud. At the heart of its scalable, fault-tolerant system are Amazon EC2 Spot Instances, which let users take advantage of unused Amazon EC2 capacity on AWS and are available at up to a 90 percent discount compared to Amazon EC2 On-Demand Pricing. To get the most out of Spot Instances, Hudl takes advantage of all Availability Zones within the AWS Region, using multiple instance families and generations to maintain the availability and capacity it needs. “When we added instances, and especially when we started weighting those instances, we opened ourselves up to using more than just compute-optimized instances,” says Degner. “Being willing to run workloads on a large number of different instance types made placement a lot easier. We’ve seen only positive
things from expanding that.” By using Spot Instances instead of On-Demand instances, Hudl is saving approximately $100,000 per month.

Originally self-managing its instance provisioning, Hudl began using Amazon EC2 Auto Scaling, which companies can use to maintain application availability and automatically add or remove Amazon EC2 instances according to conditions they define. Now, instead of manually selecting which instances to run and attempting to self-optimize, Hudl can rely on Amazon EC2 Auto Scaling to evaluate instances and optimize compute. “Putting all that in Auto Scaling groups meant we were able to basically snap our fingers and spin up that many instances,” Pryor says. “We want the fault tolerance and reliability baked in. Supporting Spot Instances fits naturally into that.” To optimize its usage of Spot Instances, Hudl uses the capacity-optimized allocation strategy, which chooses instances with the lowest interruption rate and most interrupt-friendly configuration—delivering the company additional savings of 28 percent.

Hudl has over 52 PB of video, and it will ingest 192 hours of video every minute at its projected peak rate for the season. The company has approximately 10,000 video encoding servers running during these peak times. “Looking back at one night when we had all those servers running—we didn’t have a single interruption,” says Degner. “Being able to run all those servers without a single interruption is a huge testament to how much data AWS has put out there surrounding interruption rates for different instance types, which helps us to use those instance types without issue.”

As a result of managing Spot Instances using Amazon EC2 Auto Scaling and the capacity-optimized allocation strategy for its ingest and transcoding infrastructure, Hudl’s total video compute costs have been reduced by approximately $450,000 annually. The company has seen additional benefits as well. “The cost aspect was huge, but a big motivator for using Auto Scaling groups for us was being able to use target tracking scaling,” says Degner. “Using Amazon EC2 Auto Scaling, we gain proportional scaling that’s managed by AWS, whereas previously we would have had to build fairly complex in-house scaling algorithms to accomplish the same thing. Moving away from maintaining these pieces of scaling logic that we had to tweak as our usage and customers grew to having that managed by the Auto Scaling groups using AWS was huge for us.”

The success Hudl has seen using AWS for its video ingestion and transcoding has inspired the company to consider expanding its use of AWS solutions. “We’ve started thinking about how we can use these solutions to scale efficiently in other areas as we add new users and new products,” Degner says. And as part of its mission to future-proof its compute in a way that’s also cost efficient, Hudl has begun working on optimizing its storage tiering using Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance.

“Now, instead of thinking about how to spin up compute, we can focus on improving video quality,” says Pryor. “Using AWS keeps us focused on our core mission—providing a better experience to users.”