Sprinklr Reduces Machine Learning Inference Costs on AWS Inferentia

Sprinklr, founded in 2009, is an American software company with employees all over the world. The company is an early adopter of new AWS services, and its mission is to help organizations worldwide make their customers happier. It offers over 31 different software products across 4 robust product suites and has developed an advanced proprietary artificial intelligence engine for enterprises to analyze publicly available data and engage with customers across 30 digital and social channels. With Sprinklr, businesses can collaborate across teams internally and digital channels externally to create a better customer experience.

Sprinklr is always looking to improve its customer experience while lowering compute costs and optimizing efficiency. “Our goal is to always make use of the latest technology so that we can have greater cost savings,” says Jamal Mazhar, vice president of infrastructure and DevOps at Sprinklr. Sprinklr hoped to reduce latency while lowering its ML inference costs and looked to innovations from AWS to meet those challenges. “When we learned about AWS Inferentia, it was a natural process for us to take that into consideration for our cost initiative drives,” says Yogin Patel, senior director of product engineering, artificial intelligence at Sprinklr. With the goals of reducing compute costs and improving customer satisfaction, Sprinklr began testing Amazon EC2 Inf1 Instances in July 2021.

Sprinklr’s Unified-CXM platform uses ML algorithms on unstructured data sourced from many different channels to deliver insights to its customers. For example, the company’s natural language processing and computer vision ML models analyze different data formats sourced from social media posts, blog posts, video content, and other content available on public domains across more than 30 channels. Sprinklr is able to derive customer sentiment and intent from this content to deliver product insights to its customers. Currently, the company runs about 10 billion predictions per day across its more than 500 models. Sprinklr divides its workloads into two groups—latency optimized and throughput optimized. Latency refers to how long it takes for an inference to reach its destination, and throughput refers to the number of packets that are processed within a specific time period. “If latency goes down by 20 percent in even one model, that compounds to very large cost savings,” says Patel.

AWS Inferentia features a large amount of on-chip memory, which can be used for caching large models instead of storing them off chip. This can have a significant impact in reducing inference latency because the processing cores of AWS Inferentia, called NeuronCores, have high-speed access to models that are stored in on-chip memory and are not limited by the off-chip memory bandwidth. NeuronCores also provide high-performance inference in the cloud at significantly lower costs, and they make it easy for developers to integrate ML into their business applications.

When Sprinklr began migrating models to Amazon EC2 Inf1 Instances and running benchmark tests, the company saw latency reduce by more than 30 percent on the latency-optimized workloads. “We’re always interested in testing new AWS services, experimenting with workloads, and benchmarking new instances,” says Patel. Seeing the significantly reduced latency that AWS Inferentia was able to deliver in tests, Sprinklr decided to migrate all of its latency-optimized workloads to Amazon EC2 Inf1 Instances. “The goal is always to have lower latency, which means a better customer experience,” Mazhar says. “Using Amazon EC2 Inf1 Instances, we are able to achieve that.”

After migrating about 20 models to Amazon EC2 Inf1 Instances, Sprinklr started migrating its computer vision and text models for improved efficiency and cost savings. The team is now able to deploy a model using Amazon EC2 Inf1 Instances in under 2 weeks. As the company migrated to AWS Inferentia, it found the process simple with the ample resources and support available. “We have been able to quickly get in touch with the right teams,” says Mazhar. “The support from AWS helps us boost our customer satisfaction and staff productivity.”

As Sprinklr continues to migrate models to AWS Inferentia, it will add more voice-related models, including automatic speech recognition and intent recognition, to help businesses engage with their customers further. Sprinklr expects that deploying these models on AWS Inferentia will give its customers the performance and low latency they need at significantly lower costs.