How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?

3 minute read
1

My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this.

Short description

By default, OpenSearch Service uses 50% of an instance's RAM for JVM heaps up to 32 GiB in size. The JVM memory pressure specifies the percentage of the Java heap in a cluster node. The following guidelines indicate what the JVM memory pressure percentages mean:

  • If JVM memory pressure reaches 75%, then OpenSearch Service initiates the Concurrent Mark Sweep (CMS) garbage collector for x86 Amazon Elastic Compute Cloud (Amazon EC2) instance types. ARM-based Graviton Amazon EC2 instance types use the Garbage-First (G1) garbage collector that uses additional short pauses and heap defragmentation. The garbage collection is a CPU-intensive process. If memory usage continues to grow, then you might encounter ClusterBlockException, JVM OutOfMemoryError, or other cluster performance issues. For more information, see Recovering from a continuous heavy processing load.
  • If JVM memory pressure exceeds 92% for 30 minutes, then OpenSearch Service blocks all write operations.
  • If JVM memory pressure reaches 100%, then OpenSearch Service JVM is configured to exit and eventually restarts on OutOfMemory (OOM).

The following reasons can cause high JVM memory pressure:

  • Spikes in the numbers of requests to the cluster.
  • Aggregations, wildcards, and selecting wide time ranges in the queries.
  • Unbalanced shard allocations across nodes or too many shards in a cluster.
  • Field data or index mapping explosions.
  • Instance types that can't handle incoming loads.

Resolution

Reduce traffic to the cluster to resolve high JVM memory pressure issues. To reduce traffic to the cluster, follow these best practices:

For more information about how to troubleshoot high JVM memory pressure, see Why did my OpenSearch Service node crash?

Related information

Troubleshooting Amazon OpenSearch Service

How can I scale up or scale out an Amazon OpenSearch Service domain?

Get started with Amazon Elasticsearch Service: How many shards do I need?

AWS OFFICIAL
AWS OFFICIALUpdated a year ago