Upping Performance by 40% and Optimizing Cost Using Amazon OpenSearch Service with Dream11

As an extremely popular global fantasy sports platform, Dream11 provides players with near-real-time updates and recommendations based on the statistics of real athletes. The management of its Elasticsearch clusters had become costly and complex, and engineers spent hours scaling manually to accommodate fluctuating data volumes. When Elasticsearch announced the sunsetting of its open-source distribution, Dream11 researched other options to power the search and analytics engine that is critical to its business.

Dream11 chose to adopt a solution managed by Amazon Web Services (AWS) so that it could run and scale clusters without having to worry about managing, monitoring, and maintaining the infrastructure. It migrated to Amazon OpenSearch Service, an open-source, distributed search and analytics suite derived from Elasticsearch. Working alongside AWS experts to fine-tune its compute infrastructure, Dream11 significantly optimized costs, boosted performance by 40 percent, and built an enhanced experience for its 200 million users.

Based in Mumbai, India, Dream11 brings Indian sports fans closer to the sports they love by providing world-class fan engagement and user experience through fantasy sports. The Dream11 app lets users create their own teams and play fantasy cricket, soccer, kabaddi, basketball, hockey, volleyball, handball, rugby, futsal, American football, and baseball. To select players for their fantasy teams, users need to have unfettered access to up-to-date statistics, especially before and after real-world matches. As a result, Dream11 experiences heavy spikes in workloads, and these would strain CPU utilization unevenly as the system took in terabytes of data. Some CPUs reached 95 percent utilization, while others hovered around 6 percent.Dream11 had been running its search and analytics capabilities on Elasticsearch. It managed its own instances of Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload. Manual scaling of clusters contributed to high operational overhead, and the company’s need to add instances to accommodate frequent peaks in use resulted in high compute costs.In 2021, Elasticsearch announced that it was phasing out its open-source distribution. Dream11 could no longer update its clusters, and it started to experience lagging speed and other performance issues. The company faced two options: migrate to a commercial license or switch to a forked version of Elasticsearch. Dream11 chose to implement OpenSearch, which remained an open-source and community-driven project. “We did our own research and found that Amazon OpenSearch Service was quite flexible, and its pricing was similar to the cost of managing our own OpenSearch machines on AWS,” says Mehul Batra, software engineer at Dream11.

For the migration, Dream11 received support and guidance from AWS experts who helped optimize cluster configuration, performance, and security based on operational metrics and best practices. For example, Dream11 implemented Amazon EC2 instances powered by AWS Graviton Processors, designed by AWS to deliver the best price performance for cloud workloads running in Amazon EC2. Using AWS Graviton processor–based instances, Dream11 improved cluster performance by 40 percent while using 30 percent fewer nodes. The company optimized compute costs significantly.

Dream11 engineers no longer need to add up to 100 instances to handle peak demand. Instead, they scale vertically, using the more powerful processors to distribute CPU utilization more uniformly among fewer machines. “We realized that we were getting better performance using the AWS Graviton processor–based machines specific to OpenSearch, which are built for the search engines,” Batra says. “Our CPUs are more stable.”

Even at a peak of 40,000 concurrent queries, players no longer experience time-outs or long delays. Previously, Dream11 analysts, data scientists, or product teams sometimes had to wait up to 10 seconds for data to refresh. Now, every query returns results within 150 milliseconds of latency, which has made it about 98 percent quicker for internal users to see updated statistics.

To further enhance the internal user experience and generate analytics to drive business outcomes, Dream11 plans to implement additional features of OpenSearch. For example, OpenSearch’s Machine Learning Commons library provides algorithms to train models and predict trends in data. Dream11 will also use machine learning for anomaly detection, a feature of OpenSearch that automatically detects anomalies as the system ingests data and then uses the alerting functionality to send notifications in near real time. “We chose Amazon OpenSearch Service for the perks of OpenSearch with the robustness of automatic updates, managed scaling, and AWS community support,” Batra says. “Now, our data analysts and scientists happily live with their data and dashboards.”

The flagship brand of leading Indian sports technology company Dream Sports, Dream11 is a fantasy sports platform with more than 200 million users playing 11 sports on the app.

Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.