How do I troubleshoot high JVM memory pressure on my Elasticsearch cluster?

Last updated: 2020-03-09

My Amazon Elasticsearch Service (Amazon ES) cluster has high JVM memory pressure. How do I resolve this?

Resolution

If JVM memory pressure stays high for more than a few minutes, you'll encounter a ClusterBlockException, JVM OutOfMemoryError, or other cluster performance issues.

  • If JVM memory pressure is consistently above 75%, then your Amazon ES cluster likely doesn't have enough resources to support the amount of data or the query load. Amazon ES then triggers a different garbage collection algorithm to free up memory.
  • If JVM memory pressure is around 95%, then Amazon ES kills processes to allocate memory. If a critical process is killed, then one or more cluster nodes can fail.

To prevent high JVM memory pressure, follow these best practices:

  • Confirm that shards are evenly distributed between nodes.
  • Choose the correct number of shards. For more information about indexing and shard count, see Get Started with Amazon Elasticsearch Service: How Many Shards Do I Need?
  • Avoid sending a large number of requests at one time.
  • Avoid aggregating on text fields.
  • If you must aggregate on text fields, change the mapping type to keyword. If the JVM memory pressure gets too high, then use the following API operations to clear the field data cache: POST /index_name/_cache/clear (index-level cache) and POST /_cache/clear (cluster-level cache).
    Note: Clearing the cache can disrupt queries that are in progress.

For more information about how to troubleshoot high JVM memory pressure, see Why did my Elasticsearch node crash?