How do I troubleshoot low storage space in my Amazon Elasticsearch Service domain?

Last updated: 2021-03-25

My Amazon Elasticsearch Service (Amazon ES) domain is out of storage space, and I received an error. How do I resolve this?

Short description

Your Amazon ES domain has certain storage space requirements for processing workloads. The ClusterBlockException error can occur because your Amazon ES domain ran out of storage space. For example, when you allocate storage to an Amazon ES cluster node, up to 20% of that space (20 GB) is reserved space. Also, operating systems such as Linux reserve 5% of the file system to support any critical processes performed by the root user. For more information about Amazon ES storage space requirements, see Calculating storage requirements.

To resolve low storage space issues, try the following:

  • Increase the size of the domain's Amazon Elastic Block Store (Amazon EBS) volumes. You can also add more cluster nodes to the Amazon ES domain.
  • Update your cluster sharding strategy so that shards are evenly distributed across all nodes.
  • Reduce the amount of data stored in your domain. You can delete unnecessary files, optimize old indices, or reduce the domain's replica count. Be sure to take a manual snapshot first.
    Note: Reducing the domain's replica count can reduce fault tolerance. It's a best practice to configure at least one replica for each index.
  • Use the Amazon CloudWatch FreeStorageSpace metric to monitor the amount of available storage in your cluster. To receive notifications whenever storage space is low, configure your CloudWatch alarms.

Depending on your data storage source, you might need to incorporate additional troubleshooting techniques.

Resolution

Updating your storage configuration settings

Important: Before you update your configuration settings, verify the configuration changes that could trigger a blue/green deployment. Also, make sure that your dedicated master node type is the recommended node type.

Use the cat allocation command to check how much storage space is available for each node in your cluster:

curl ES_Endpoint/_cat/allocation?v

To update your Amazon ES domain configuration settings, perform the following steps:

1.    Sign in to your AWS Management Console.

2.    Open the Amazon ES console.

3.    Choose the Amazon ES domain that you want to add storage to.

4.    Choose Edit Domain.

5.    Update your domain settings.

Note: If you're using Amazon EBS volume for storage, update your Storage configuration settings. Or you can update the number of data nodes.

6.    Choose Submit.

If your domain uses Amazon EBS volumes for data storage

If the domain uses Amazon EBS volumes for storage, consider the following approaches:

If your domain uses Amazon Elastic Compute Cloud (Amazon EC2) I3 instances for data storage

If you're using Amazon EC2 I3 instances for data storage, consider the following approaches:

  • Add more nodes to your Elasticsearch cluster.
  • Scale up the instance type to obtain more storage space.
  • Delete any unwanted indices from your Elasticsearch cluster.