How do I resolve the 429 error "es_rejected_execution_exception" in Amazon Elasticsearch Service?
Last updated: 2020-10-16
When I write data to my Amazon Elasticsearch Service (Amazon ES) cluster, the requests are rejected. I get a ThreadpoolBulkRejected error message that looks like this:
error":"elastic: Error 429 (Too Many Requests): rejected execution of org.elasticsearch.transport.TransportService$7@b25fff4 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@768d4a66[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 820898]] [type=es_rejected_execution_exception]"
Note: Different versions of Amazon ES use different thread pools to process calls to the _index API. Elasticsearch versions 1.5 and 2.3 use the index thread pool. Elasticsearch versions 5.x, 6.0, and 6.2 use the bulk thread pool. Elasticsearch versions 6.3 and later use the write thread pool.
The es_rejected_execution_exception[bulk] is a bulk queue error. It occurs when the number of requests to the Elasticsearch cluster exceeds the bulk queue size (threadpool.bulk.queue_size). The bulk queue on each node can hold between 50 and 200 requests, depending on which Elasticsearch version you are using. When the queue is full, new requests are rejected.
The es_rejected_execution_exception[write] is a ThreadpoolWriteQueue error. The es_rejected_execution_exception[write] occurs when the number of requests to the Elasticsearch cluster exceeds threadpool.write.queue_size, the _index queue size. The ThreadpoolWriteQueue on each node can hold between 50 and 200 requests, depending on which Elasticsearch version you are using. When the queue is full, new requests are rejected.
Note: The bulk queue size can't be increased in most Amazon ES versions. The queue exists for a reason—it limits requests to a manageable amount. For more information, see Threadpool section in the Elasticsearch documentation.
Use one of the following methods to resolve es_rejected_execution_exception errors:
- Add more nodes: Each node has a write queue, so adding more nodes can give you more queuing capacity. To add nodes, see Configuring Amazon ES domains.
Note: Make sure that you have enough active indexing shards to distribute to the new nodes. Otherwise, the addition of nodes doesn't improve your queueing capacity. An "active indexing shard" is a shard that received at least one indexing request in the last five minutes.
- Switch to a larger instance type: The number of threads in the thread pool on each node for write requests is equal to the number of available processors. Switch to an instance with more virtual CPUs (vCPUs) to get more threads to process _index requests. For more information, see Choosing instance types and testing.
- Improve indexing performance: When documents are indexed faster, the write queue is less likely to reach capacity. For more information about performance tuning, see How can I improve indexing performance on my Elasticsearch cluster?