Posted On: Dec 3, 2019
Amazon Elasticsearch Service now offers UltraWarm, a performance-optimized warm storage tier. UltraWarm lets you store and interactively analyze your data using Elasticsearch and Kibana while reducing your cost per GB by up to 90% over existing Amazon Elasticsearch Service hot storage options. With UltraWarm, Amazon Elasticsearch Service now supports hot-warm domain configurations. Hot storage is used for indexing and providing the fastest access to data. UltraWarm complements hot storage with less expensive, more durable storage for older data that you access less frequently, all while maintaining the same interactive analysis experience.
Real-time analysis of machine-generated data is essential in identifying and resolving operational and security issues for modern applications. With hot storage, Amazon Elasticsearch Service can ingest large amounts of log data and analyze it in real time, but storing months or years of data can be cost-prohibitive at scale. Some users of Elasticsearch use storage-dense D2 instances for warm storage, but that solution struggles to achieve high utilization and requires replica shards for durability.
UltraWarm amplifies the benefits of traditional hot-warm configurations. It uses a combination of Amazon S3 and optimized compute nodes powered by the AWS Nitro System to provide a hot-like experience for aggregations and visualizations. S3 provides low-cost storage with 99.999999999% (11 9's) durability, removing the need for replicas. S3 also abstracts away the notion of overhead, so each UltraWarm node can use 100% of its available storage for primary data. To improve performance, these nodes use granular caching across all layers of the stack, adaptive prefetching, and query processing optimizations to provide similar, or in many cases superior performance to traditional warm nodes which rely on high density local storage.
The preview of UltraWarm is available on Elasticsearch version 6.8 in 3 regions: US East (N. Virginia, Ohio), US West (Oregon, N). You can use the AWS Console, CLI, or APIs to configure an Amazon Elasticsearch Service domain with UltraWarm. To learn more, please visit our documentation.