How do I troubleshoot low storage space in my Amazon OpenSearch Service domain?

Last updated: 2021-07-30

My Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) domain is out of storage space, and I received an error. How do I resolve this?

Short description

Your OpenSearch Service domain has certain storage space requirements for processing workloads. The ClusterBlockException error can occur because your OpenSearch Service domain ran out of storage space. For example, when you allocate storage to a cluster node, up to 20% of that space (20 GB) is reserved space. Also, operating systems such as Linux reserve 5% of the file system to support any critical processes performed by the root user. For more information about Amazon OpenSearch Service storage space requirements, see Calculating storage requirements.

To resolve low storage space issues, try the following:

  • Increase the size of the domain's Amazon Elastic Block Store (Amazon EBS) volumes. You can also add more cluster nodes to the OpenSearch Service domain.
  • Update your cluster sharding strategy so that shards are evenly distributed across all nodes.
  • Reduce the amount of data stored in your domain. You can delete unnecessary files, optimize old indices, or reduce the domain's replica count. Be sure to take a manual snapshot first.
    Note: Reducing the domain's replica count can reduce fault tolerance. It's a best practice to configure at least one replica for each index.
  • Use the Amazon CloudWatch FreeStorageSpace metric to monitor the amount of available storage in your cluster. To receive notifications whenever storage space is low, configure your CloudWatch alarms.

Depending on your data storage source, you might need to incorporate additional troubleshooting techniques.

Resolution

Updating your storage configuration settings

Important: Before you update your configuration settings, verify the configuration changes that could trigger a blue/green deployment. Also, make sure that your dedicated master node type is the recommended node type.

Use the cat allocation command to check how much storage space is available for each node in your cluster:

curl ES_Endpoint/_cat/allocation?v

To update your OpenSearch Service domain configuration settings, perform the following steps:

1.    Sign in to your AWS Management Console.

2.    Open the OpenSearch Service console.

3.    Choose the OpenSearch Service domain that you want to add storage to.

4.    Choose Edit Domain.

5.    Update your domain settings.

Note: If you're using Amazon EBS volume for storage, update your Storage configuration settings. Or you can update the number of data nodes.

6.    Choose Submit.

If your domain uses Amazon EBS volumes for data storage

If the domain uses Amazon EBS volumes for storage, consider the following approaches:

If your domain uses Amazon Elastic Compute Cloud (Amazon EC2) I3 instances for data storage

If you're using Amazon EC2 I3 instances for data storage, consider the following approaches:

  • Add more nodes to your OpenSearch Service cluster.
  • Scale up the instance type to obtain more storage space.
  • Delete any unwanted indices from your OpenSearch Service cluster.