How do I resolve the 429 error "es_rejected_execution_exception" in Amazon Elasticsearch Service?
Last updated: 2019-07-16
When I write data to my Amazon Elasticsearch Service (Amazon ES) cluster, the requests are rejected. I get a ThreadpoolBulkRejected error message like this:
error":"elastic: Error 429 (Too Many Requests): rejected execution of org.elasticsearch.transport.TransportService$7@b25fff4 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@768d4a66[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 820898]] [type=es_rejected_execution_exception]"
The es_rejected_execution_exception[bulk] is a bulk queue error. It occurs when the number of requests to the Elasticsearch cluster exceeds the bulk queue size (threadpool.bulk.queue_size). The bulk queue on each node can hold between 50 and 200 requests, depending on which Elasticsearch version you are using. When the queue is full, new requests are rejected.
Note: It's not possible to increase the bulk queue size in most Amazon ES versions. The queue exists for a reason—it limits requests to a manageable number. For more information, see Threadpool Section in the Elasticsearch documentation.
Use one of the following methods to resolve es_rejected_execution_exception errors:
- Add more nodes: Each node has a bulk queue, so adding more nodes gives you more queuing capacity. To add nodes, see Configuring Amazon ES Domains (Console). Note: Adding more data nodes won't help if you don't have enough actively indexing shards to distribute to the new nodes. An "actively indexing shard" is a shard that has received at least one indexing request in the last 5 minutes.
- Switch to a larger instance type: The number of threads in the thread pool on each node for bulk requests is equal to the number of available processors. Switch to an instance with more virtual CPUs (vCPUs) to get more threads to process bulk requests. For more information, see Choosing Instance Types and Testing.
- Improve indexing performance: When documents are indexed faster, the bulk queue is less likely to reach capacity. For more information about performance tuning, see Indexing Performance Tips in the Elasticsearch documentation.