Observe.AI Cuts Costs by Over 50% with Machine Learning on AWS

Observe.AI optimizes the customer experience through an artificial intelligence (AI)-powered workforce platform. Employing a large language model (LLM) designed for contact centers, Observe.AI enhances contact center agent performance and extracts insights from customer interactions using conversation intelligence. Each month, the platform processes millions of conversations and generates hundreds of inferences per conversation.

As machine learning (ML) adoption continues to grow across industries, testing the performance of customers’ ML services under varying data loads has become increasingly crucial for Observe.AI. Aashraya Sachdeva, staff engineer in machine learning at Observe.AI, says, "While onboarding new customers, we were assessing our ML system's capability to handle a tenfold increase in data load, corresponding to the tenfold rise in conversations processed daily. Our ML engineers and scientists faced challenges in accurately predicting this capability when transitioning models from research to production."

The company sought to deploy a larger ML model in production for enhanced accuracy. Simultaneously, there was a careful effort to manage latency and control costs associated with the implementation. Achieving an optimal return on investment through fine-tuning its infrastructure was key, and the business wanted a solution compatible with its existing Amazon Web Services (AWS) environment.

"We sought a more straightforward method to identify the optimal infrastructure, assess our readiness for increased load, and determine the associated costs for serving code to customers. We also wanted precise insights into the developer time required for implementation," Aashraya explains.

To address its challenge of predicting ML load sizes, Observe.AI created and open-sourced the One Load Audit Framework (OLAF). Integrated with Amazon SageMaker, a service that builds, trains, and deploys ML models for any use case, OLAF identifies bottlenecks and performance issues in ML services, offering latency and throughput measurements under both static and dynamic data loads. The framework also seamlessly incorporates ML performance testing into the software development lifecycle, facilitating accurate provisioning and cost savings.

Aashraya explains, "OLAF provides our ML engineers and scientists with a plug-and-play model. They simply input their AWS credentials and the Amazon SageMaker endpoint, and the tool conducts load testing, providing latency numbers and expected errors for a particular model or instance."

Following the initial build, Observe.AI integrated Amazon SageMaker features into OLAF, including multi-container deployment and batch inferencing. "We wanted to understand how these incremental features affect scalability in terms of cost," adds Aashraya. Next, the company incorporated Amazon Simple Queue Service (Amazon SQS), a fully managed message queuing service for microservices, distributed systems, and serverless applications. By downloading Amazon SQS load traces, OLAF users can observe the rate at which ML messages enter the system to predict data load size. Aashraya notes, "This feature assists us in easily testing queue-based array processing systems, which are becoming more prevalent."

Finally, Observe.AI integrated Amazon Simple Notification Service (Amazon SNS), a fully managed service for application-to-application and application-to-person messaging that helps OLAF users replicate specific patterns within Amazon SNS.