Why is my Amazon Elasticsearch Service (Amazon ES) domain stuck in the "Processing" state?

Last updated: 2020-03-10

My Amazon Elasticsearch Service (Amazon ES) domain is stuck in the "Processing" state after I made a configuration change. How do I resolve this? 

Short Description

Your Amazon ES domain might be stuck in the "Processing" state for several reasons:

It's a best practice to check the number of shards and shard assignments before trying to troubleshoot other possible reasons.

For more information about shard reallocation and updating its required parameters, see Index-level shard allocation filtering on the Elastic website.

Resolution

Too many shards and not enough nodes

Amazon ES migrates shards during snapshot recoveries, domain configuration changes, replication level changes, node failures, and node startup. When your cluster has an unbalanced number of shards (in relation to available nodes), you might experience other issues. A shard remains in an "Unassigned" state if there aren't enough nodes to evenly distribute the number of shards. This can result in a slow migration or your cluster can get stuck in the "Processing" state.

One solution is to reduce the number of shards in your cluster. For more information about designating the appropriate number of shards, see Choosing the Number of Shards.

To determine whether shards are migrating, type the following command syntax:

$ curl -XGET "ES_Endpoint/_cat/recovery?v&active_only"

Note: Wait a few minutes before running the command again. If there's a change in the output, then the shards are still being migrated.

Then, check the number of shards allocated to each node and the amount of disk space in use:

$ curl -XGET ES_Endpoint/_cat/allocation?v

Note: This command syntax also indicates whether the cluster node failed due to high disk usage or JVM memory pressure.

Unassigned shards

To check the number of shards and indices in your cluster, enter the following command syntaxes:

$ curl -XGET ES_Endpoint/_cat/indices
$ curl -XGET ES_Endpoint/_cat/shards

After you identify the unassigned shards, minimize the number of unnecessary shards in your domain. For more information about shard calculations, see Get Started with Amazon Elasticsearch Service: How Many Shards Do I Need?

Reduce the number shards in the cluster

To reduce the number of shards in your cluster or any overhead, remove all old or outdated indices:

$ curl -XDELETE ES_Endpoint/oldindex1,oldindex2

If you have an index rotation configured, each rotation results in a newly created set of shards. By default, Amazon ES creates five primary shards per index and one replica shard for every primary shard. This means that index rotation can eventually lead to an overloaded master node. To avoid an overloaded master node, consider changing your index rotation strategy or using an index template. These best practices can help to control future shard growth.

Note: You can specify shard count only when you create a new index or re-index your existing data. Before indexing your document, choose the number of shards.

To specify the number of shards in a new index, run the following command:

$ curl -XPUT ES_Endpoint/index-name -H 'Content-Type: application/ json' -d'
{
      "settings": {
            "index": {
                    "number_of_shards": 3,
                    "number_of_replicas": 1                   
       }        
}'

Note: Amazon ES can't change the number of primary shards in an existing index. For more information about indexing data in Amazon ES, see Introduction to Indexing.

To restructure an existing index, perform the following tasks:

1.    Create and define a new index template. The following syntax specifies the number of new shards created in the cluster:

$ curl -XPUT ES_Endpoint/_template/template_1 -H 'Content-Type: application/json' -d'
{
      "index_patterns": ["*"],
      "settings": {
            "number_of_shards": 3,
            "number_of_replicas": 1     
      }
}'

2.    Re-index by calling the _reindex API operation (Amazon ES versions 5.1 and later). The following syntax moves your data into a new index, specifying the new number of shards in your index template:

$ curl -XPOST ES_Endpoint/_reindex -H 'Content-Type: application/json' -d '
{
      "source": {   
            "index": "old_index"
      },
      "dest": {
            "index": "new_index"
      }   
}'

Important: If your access policy includes AWS Identity and Access Management (IAM) users or roles, you must sign HTTP requests to the Amazon Elasticsearch Service APIs.

3.    Verify that the new and old indices have the same number of documents:

$ curl -XGET ES_Endpoint/_cat/indices/old_index,new_index?v

4.    When the new and old indices show the same number of documents, you can then delete the old index:

$ curl -XDELETE ES_Endpoint/old_index

5.    Re-run your configuration update. For more information about configuration changes, see About Configuration Changes.