Boosting query performance 30% using Amazon OpenSearch Service with Druva

Druva has hundreds of customers who comply with the Federal Risk and Authorization Management Program (FedRAMP), which requires organizations to monitor their systems continuously and store audit logs for 90 days. Because AWS built its core infrastructure to satisfy security requirements for the highest-sensitivity organizations, Druva had built its compute infrastructure on Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload.

For 8 years, Druva had been using a third-party logging solution for centralized log management with Elasticsearch as its primary data store, requiring an engineer to spend up to 2 days per month on patching, maintenance, or complex upgrades. Furthermore, the system struggled to process 300 GB of log data across multiple accounts daily, resulting in indexing bottlenecks during peak periods. The system averaged 3–5 minutes of indexing latency, well short of the company’s 60-second target. “Many times, we would be an hour or two behind on log ingestion, or we would have to restore from a backup if an upgrade was unsuccessful,” says Tony Hendrick, Druva’s manager of US cloud operations.

When its legacy logging solution stopped supporting Elasticsearch, Druva decided to self-manage a system built on OpenSearch. “It was doable, but we ran into the same issues as before,” says Mansi Chikarmane, cloud operations engineer at Druva. “A lot of the documentation pointed toward Amazon OpenSearch Service, so we decided to give it a try.”

Druva recognized that its small team would benefit from the use of Amazon OpenSearch Service and fully migrated from its self-managed system in just 1 month. “We knew that the use of an AWS-managed service would resolve the maintenance overhead and provide seamless integration,” says Chikarmane. “It just simplified our architecture because we already have all our infrastructure on AWS.” Druva worked alongside AWS Enterprise Support, which helps to accelerate innovation, strengthen security, and streamline cloud operations with AI-powered insights and AWS experts. Together, they considered important technical decisions that could exceed performance requirements, reduce costs, and improve security posture.

Moreover, they worked through a blue/green deployment strategy for existing applications as Druva simultaneously built its new Druva Data Security Cloud, a FedRAMP-certified hybrid offering. “Because the application fell under that FedRAMP certification maintained by AWS, it eased the entire development process,” says Hendrick. Druva automated storage policies through Index State Management (ISM) in Amazon OpenSearch Service, migrating logs to UltraWarm storage for Amazon OpenSearch Service after 30 days and deleting them after the 90-day FedRAMP mandate.

After discussing with its AWS technical account manager the potential for improved compute price performance, Druva migrated to AWS Graviton processors, a family of processors designed to deliver the best price performance for cloud workloads running in Amazon EC2. To optimize capacity needs, the AWS Enterprise Support team recommended that Druva use different instance types for master and data nodes. Master nodes run on Amazon EC2 C6g Instances, ideal for running compute-intensive workloads, while indexing nodes use general-purpose Amazon EC2 M6g Instances. Additionally, Druva implemented a cost-effective tiered storage approach for its substantial monthly dataset in Amazon Simple Storage Service (Amazon S3), an object storage service.

After revamping its architecture, Druva experienced 30 percent better query performance and a 35 percent reduction in indexing latency. It cut mean time to resolution in half and, using anomaly detection in Amazon OpenSearch Service, decreased mean time to detection by 80 percent. “We also had reduced support overhead,” says Chikarmane. “We've virtually eliminated version compatibility outages; that's an issue we simply don't face anymore.” Additionally, Druva has reduced maintenance time from days to 30 minutes.

Upon migrating to Graviton, Druva cut 10 percent off its compute costs. After 14 days, Druva elected to purchase Amazon EC2 Reserved Instances and saved an additional 49 percent in compute costs. Additionally, the company has cut storage costs by about 25 percent despite a 15 percent increase in overall indexed storage space.

Using Amazon OpenSearch Service, Druva has enhanced observability by simplifying the ability to add alerts and automating monitoring. “That’s led to us detecting things before the customer even reaches out,” says Hendrick. “Our mean time to detect has significantly improved using Amazon OpenSearch Service, contributing to a healthier and more secure environment overall.”