Unitary Scales AI Moderation to 26 Million Videos Daily with Amazon EKS

Founded in 2019, Unitary specializes in multimodal ML models for identifying harmful text, images, and videos amid seas of digital content. “Neither human nor AI moderation alone can do the whole job, and platforms are forced to combine the two,” says Matt Camp, principal architect at Unitary. “We provide a blended solution that catches harmful content in less time, with more accuracy, and at a lower cost.”

The first version of Unitary’s product was originally built on Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload. But scaling to handle a few million videos daily would have required self-managing more than 1,200 instances. That would have been an overwhelming challenge for the three-person platform team because of the added operational overhead and complexity of self-management at high volume.

As Unitary’s AI models improved, the volume of content that the company needed to process soon outpaced what its initial infrastructure could handle. In 2021, Unitary decided to build a new microservice-based, event-driven architecture on Amazon EKS. For efficient node management, the AWS team recommended adopting Karpenter, an open-source Kubernetes node lifecycle manager, alongside Amazon EKS to scale infrastructure on demand.

Unitary uses an API to ingest up to 26 million videos daily, which represent several billion frames of content and tens of millions of hours of audio. When Unitary receives a video to be analyzed, multiple microservices that run on Amazon EKS split the video into frames, extract the audio, and then classify the video’s content using ML. Afterward, the extracted parts of the videos are stored. Then, Unitary runs a series of AI and ML models—some of which can be trained using a customer’s safety policies—to detect toxic or unsafe content in the combined multimodal context of visuals, audio, and text. Finally, the models generate a summary that includes a safety score, which Unitary delivers to customers so that they can make informed moderation decisions.

Using Amazon EKS and Karpenter for scaling, Unitary was able to unlock the flexibility and speed it needed to accommodate unpredictable traffic while maintaining its uptime. With Karpenter, the company can implement several optimizations using a single tool. For example, Karpenter gracefully handles the necessary Kubernetes node disruptions for security updates and pod scheduling without interrupting system throughput and stability. Using several Karpenter NodePools, Unitary can provide different offerings to its customers according to their different latency requirements. Moreover, the company is developing subsecond-latency solutions so that it can classify harmful images before they go live. Thanks to the speed at which Karpenter can provision new Kubernetes nodes, Unitary now has the responsiveness required to achieve these results.

This flexibility improves productivity; what might require several teams in another company can be done by Unitary with a handful of people. Because Unitary doesn’t need to continuously manage clusters, its small team can focus on more important work, such as enhancing the user experience and developing new features. “Even running at peak traffic, using over 1,000 nodes to process more than 20 million videos daily, our entire team spent 5 days abroad without needing to manage or interact with the clusters,” says Camp. The system automatically replaces unhealthy nodes, rescheduling pods to verify that no text, image, or video is left unmoderated. Also, Unitary has reduced container boot times on its ML inference workloads by up to 80 percent. That means that large containers can be spun up quickly to react to changing patterns of customer traffic.

Using Karpenter, Unitary unlocked the benefits of intelligent selection for Amazon EC2 instances across zones. The company uses a blend of Amazon EC2 Spot Instances, which let users take advantage of unused Amazon EC2 capacity on AWS, and Amazon EC2 On-Demand Instances, which let users pay for compute capacity by the hour or second with no long-term commitments. The consolidation feature of Karpenter, in addition to the tool’s speed, has empowered Unitary to quickly redistribute workloads and adapt to the dynamic availability of more cost-effective resources. In fact, Unitary estimates that using Spot Instances for GPU inference infrastructure has reduced costs by 50–70 percent over 18 months.

As Unitary scaled its infrastructure to meet growing demand, the need for robust observability became increasingly critical. The number of nodes in each Kubernetes cluster can fluctuate within minutes from 30 to over 1,000. That variability means handling a growing volume of time series data and scraping more samples with Prometheus, an open-source monitoring and alerting solution. After trying multiple observability tools, Unitary decided to use Amazon Managed Service for Prometheus—a service that offers highly available, secure, and managed monitoring for containerized systems. Thus, the company was able to collect the level of metrics it required without having to manage the scaling of Prometheus resources.

On AWS, Unitary helps platforms with over 100 million combined users combat toxic content while minimizing the impact on human moderators. As the company continues to grow, Amazon EKS and Karpenter will continue to be a key part of its infrastructure. “We did not have to pay a premium to achieve the ML functionality that we want on Amazon EKS—we needed to pay for only the underlying compute,” says Camp. “By keeping these costs low, we can continue to innovate at a more agile pace and transfer these savings to our customers.”

Unitary continually trains its models to keep pace with its customers’ changing needs. It also plans to invent solutions on AWS to continue overcoming the challenges of traditional content moderation and make the internet safer. “As a startup, we wanted to rapidly scale our product, so we needed to build a fast, flexible, and highly scalable platform,” says Sasha Haco, CEO of Unitary. “On AWS, we did just that."