Posted On: Mar 24, 2021
Amazon Elasticsearch Service now supports automated memory management of Elasticsearch clusters with the new Auto-Tune feature. Auto-Tune is an adaptive resource management system that automatically adjusts Elasticsearch internal settings to handle dynamic workloads, optimizing cluster resources to improve efficiency and performance. With Auto-Tune, you can achieve performance boost in ingestion throughput for log analytics workloads, and reduced tail latencies for search queries.
Elasticsearch runs in a Java virtual machine (JVM), and tuning the memory settings is critical to support large ingest volumes and search workloads. Auto-Tune’s adaptive system identifies the resource bottlenecks affecting the performance of incoming workloads and takes appropriate remediation actions across the cluster to support the required scale and performance. This is achieved by resizing and re-allocating memory and Elasticsearch resources within the existing nodes in the cluster, while ensuring maximum cluster performance, efficiency and availability. For example, tuning heap sizes, bulk queue sizes, and garbage collection settings bolsters ingestion throughput; changing cache sizes improves tail latency for searches; and adjusting read and bulk queue sizes can lead to fewer rejections and higher availability for search workloads.
To use Auto-Tune, customers simply enable the feature from the console or CLI and it will dynamically start adapting to the workloads. Auto-Tune for queues and caches does not require any process restart and is enabled by default on new domains. Auto-Tune will publish tuning events to Amazon CloudWatch and Amazon EventBridge when relevant memory tunings are available or applied to the cluster setting. These events will also be visible under the ‘Notifications’ view in the Amazon Elasticsearch Service console. Customers can visualize the real-time effects of the changes on cluster performance and efficiency in dashboards. Auto-Tune leverages Performance Analyzer that is available in Open Distro for Elasticsearch, an Apache 2.0-licensed distribution of Elasticsearch. Auto-Tune is available on all domains running Elasticsearch 6.7 and above. To learn more, see the documentation.
Auto-Tune for Amazon Elasticsearch Service is now available in 24 regions globally: US East (N. Virginia, Ohio), US West (Oregon, N. California), AWS GovCloud (US-Gov-East, US-Gov-West), Canada (Central), South America (Sao Paulo), EU (Ireland, London, Frankfurt, Paris, Stockholm, Milan), Asia Pacific (Singapore, Sydney, Tokyo, Seoul, Mumbai, Hong Kong), Middle East (Bahrain), China (Beijing – operated by Sinnet, Ningxia – operated by NWCD) and Africa (Cape Town). Please refer to the AWS Regional Services List for more information about Amazon Elasticsearch Service availability.