How Devo’s security analytics solution uses Amazon EBS to optimize storage cost and performance

Devo is a cloud-native, multi-tenant centralized log management platform used by some of the world’s largest organizations to ingest, secure and enable real-time querying and analytics of their data. When managing data for major brand-name retail, financial services, and public sector organizations as Devo does, there’s no room for error when it comes to the availability. Devo delivers speed, efficiency, and protection for their customers by providing rapid query capabilities and real-time alerting, while maintaining hot data with varying retention periods.

In this post, we describe how Devo uses two Amazon Elastic Block Store (EBS) volume types — Throughput Optimized HDD (st1) and General Purpose SSD (gp3) — to achieve cost effectiveness and performance optimization while providing customers with outstanding, consistent service. For Devo, high-performance block storage is critical to scale fast and most importantly, to protect against failures. With EBS, Devo found powerful features that made it easier to automatically and reliably store persistent data, while optimizing cost investments in cloud storage.

Amazon EBS

Amazon EBS provides seven volume types: Provisioned IOPS SSD (io2 Block Express, io2, and io1), General Purpose SSD (gp3 and gp2), Throughput Optimized HDD (st1) and Cold HDD (sc1). These volume types differ in performance characteristics and price, allowing you to tailor your storage performance and cost to the needs of your applications. For example, Throughput Optimized HDD (st1) volumes provide low-cost magnetic storage that defines performance with dominant throughput measured in MiB/s. A st1 volume can burst up to a maximum of 500 MiB/s per TiB with a baseline throughput performance of 40 MiB/s at a lower cost compared to the SSD-based EBS volume types.

In an environment where intense read and write operations support a wide variety of transactional workloads, the SSD-backed Amazon EBS General Purpose volumes (gp3) provide superior input/output operations per second (IOPS) performance at an affordable price point – up to 20% lower price per GB than the existing gp2 volumes. The gp3 volume type provides a baseline performance of 3000 IOPS and 125 MiB/s regardless of volume size. Customers who need higher performance can scale up to 16,000 IOPS and 1,000 MiB/s for an additional fee.

How Devo uses Amazon EBS

The Devo analytics cluster uses Amazon EC2 instances for its data nodes, and this cluster relies on local NVMe drives to write current data as quickly as possible. The data is replicated to the backup EBS gp3 volumes for higher durability. For longer-term storage, the data is then written to EBS st1 volumes, which deliver the high throughput needed for hot searchability and provide an additional layer of persistence. Read requests are distributed across multiple st1 volumes to spread the throughput load. This is an ideal use case for st1 volumes which work best for throughput-intensive and big data workloads with large data sets, large I/O block sizes, and sequential I/O patterns.

During periods of high demand, Devo leverages the Elastic Volumes feature in Amazon EBS to dynamically increase performance with no downtime. Devo uses CloudWatch events to monitor st1 burst credits. If the st1 volume credit limit is reached, Devo modifies the volume type to gp3, keeping volume size consistent, to obtain higher IOPS that can meet peak loads. Once the peak declines and credits are available again, Devo switches back to st1 volumes.

The benefits of working with Amazon EBS

Devo meets its customers’ high expectations using a combination of current-generation instance types with EBS gp3 and st1 volumes for optimized compute, memory, and storage. With this strategy, Devo can quickly scale its platform capacity, tune performance, and change the type of live volumes with zero interruption to the workload.

By using gp3 volumes in peak hours, and st1 volumes in non-peak hours, Devo gets the best disk and cost performance that can scale and adapt to the changing business needs of its customers. This storage solution provides the optimal cost structure, allowing Devo to burst to 500 IOPS at st1 price levels, while leveraging gp3 for higher IOPS requirements. With EBS, Devo has found a way to improve margins and competitive flexibility.

Thanks for reading this post! If you have any comments or questions, leave them in the comments section.