Upping Performance by 40% and Optimizing Cost Using Amazon OpenSearch Service with Dream11

Based in Mumbai, India, Dream11 brings Indian sports fans closer to the sports they love by providing world-class fan engagement and user experience through fantasy sports. The Dream11 app lets users create their own teams and play fantasy cricket, soccer, kabaddi, basketball, hockey, volleyball, handball, rugby, futsal, American football, and baseball. To select players for their fantasy teams, users need to have unfettered access to up-to-date statistics, especially before and after real-world matches. As a result, Dream11 experiences heavy spikes in workloads, and these would strain CPU utilization unevenly as the system took in terabytes of data. Some CPUs reached 95 percent utilization, while others hovered around 6 percent.

Dream11 had been running its search and analytics capabilities on Elasticsearch. It managed its own instances of Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload. Manual scaling of clusters contributed to high operational overhead, and the company’s need to add instances to accommodate frequent peaks in use resulted in high compute costs.

In 2021, Elasticsearch announced that it was phasing out its open-source distribution. Dream11 could no longer update its clusters, and it started to experience lagging speed and other performance issues. The company faced two options: migrate to a commercial license or switch to a forked version of Elasticsearch. Dream11 chose to implement OpenSearch, which remained an open-source and community-driven project. “We did our own research and found that Amazon OpenSearch Service was quite flexible, and its pricing was similar to the cost of managing our own OpenSearch machines on AWS,” says Mehul Batra, software engineer at Dream11.

For the migration, Dream11 received support and guidance from AWS experts who helped optimize cluster configuration, performance, and security based on operational metrics and best practices. For example, Dream11 implemented Amazon EC2 instances powered by AWS Graviton Processors, designed by AWS to deliver the best price performance for cloud workloads running in Amazon EC2. Using AWS Graviton processor–based instances, Dream11 improved cluster performance by 40 percent while using 30 percent fewer nodes. The company optimized compute costs significantly.

Dream11 engineers no longer need to add up to 100 instances to handle peak demand. Instead, they scale vertically, using the more powerful processors to distribute CPU utilization more uniformly among fewer machines. “We realized that we were getting better performance using the AWS Graviton processor–based machines specific to OpenSearch, which are built for the search engines,” Batra says. “Our CPUs are more stable.”

Even at a peak of 40,000 concurrent queries, players no longer experience time-outs or long delays. Previously, Dream11 analysts, data scientists, or product teams sometimes had to wait up to 10 seconds for data to refresh. Now, every query returns results within 150 milliseconds of latency, which has made it about 98 percent quicker for internal users to see updated statistics.

Using the managed scaling built into Amazon OpenSearch Service, Dream11 significantly cut its operational overhead. Engineers no longer spend many hours manually scaling up at peak and scaling down after matches. With their time savings, Dream11 engineers focus on improving search and analytics capabilities for internal users by building customized visualizations and reports to unlock valuable insights. Additionally, engineers use predefined features of Amazon OpenSearch Service with reusable drag-and-drop components to manage the cluster’s lifecycle policy, rollover policy, alias creation, and alerting, which used to require custom scripts.

It also built fine-grained access control into dashboards while keeping the functionality simple for internal users as they track key performance indicators. Dream11 established a tenant-based architecture so that different persona users—data analysts or scientists, for example—access only the dashboards for which they have permission. Users maintain the single sign-on that they had with the previous system, simply entering their emails to log on or query data without the need for a username and password. “That was something that was missing in our old cluster,” Batra says. “We really wanted to use it in OpenSearch so that users wouldn’t interfere with each other’s privacy.”