How do I troubleshoot low storage space in my Amazon OpenSearch Service domain?

Last updated: 2021-07-30

Your OpenSearch Service domain has certain storage space requirements for processing workloads. The ClusterBlockException error can occur because your OpenSearch Service domain ran out of storage space. For example, when you allocate storage to a cluster node, up to 20% of that space (20 GB) is reserved space. Also, operating systems such as Linux reserve 5% of the file system to support any critical processes performed by the root user. For more information about Amazon OpenSearch Service storage space requirements, see Calculating storage requirements.

To resolve low storage space issues, try the following:

• Increase the size of the domain's Amazon Elastic Block Store (Amazon EBS) volumes. You can also add more cluster nodes to the OpenSearch Service domain.
• Update your cluster sharding strategy so that shards are evenly distributed across all nodes.
• Reduce the amount of data stored in your domain. You can delete unnecessary files, optimize old indices, or reduce the domain's replica count. Be sure to take a manual snapshot first.
  Note: Reducing the domain's replica count can reduce fault tolerance. It's a best practice to configure at least one replica for each index.
• Use the Amazon CloudWatch FreeStorageSpace metric to monitor the amount of available storage in your cluster. To receive notifications whenever storage space is low, configure your CloudWatch alarms.

Depending on your data storage source, you might need to incorporate additional troubleshooting techniques.

Important: Before you update your configuration settings, verify the configuration changes that could trigger a blue/green deployment. Also, make sure that your dedicated master node type is the recommended node type.

Use the cat allocation command to check how much storage space is available for each node in your cluster:

curl ES_Endpoint/_cat/allocation?v

To update your OpenSearch Service domain configuration settings, perform the following steps:

1.    Sign in to your AWS Management Console.

2.    Open the OpenSearch Service console.

3.    Choose the OpenSearch Service domain that you want to add storage to.

4.    Choose Edit Domain.

5.    Update your domain settings.

Note: If you're using Amazon EBS volume for storage, update your Storage configuration settings. Or you can update the number of data nodes.

6.    Choose Submit.

If the domain uses Amazon EBS volumes for storage, consider the following approaches:

• Increase the size of the EBS volumes. The maximum volume size depends on the node's Amazon Elastic Compute Cloud (Amazon EC2) instance type or Elasticsearch version type. For example, 512 GiB is the maximum volume size for Elasticsearch version 1.5.
• If you can't increase the size of the EBS volumes, add additional nodes or scale up your domain and choose a new EBS limit. For more information about EBS volumes, see EBS volume size limits.
• Create a backup of unwanted indices to your Amazon Simple Storage Service (Amazon S3) bucket. Then, delete the indices from your OpenSearch Service cluster to free up disk space. Or, you can use Curator to rotate data in OpenSearch Service. You can also use Index State Management to rotate indices in OpenSearch Service.
  Note: You must have a manual snapshot repository set up to create any backups.

If you're using Amazon EC2 I3 instances for data storage, consider the following approaches:

• Add more nodes to your OpenSearch Service cluster.
• Scale up the instance type to obtain more storage space.
• Delete any unwanted indices from your OpenSearch Service cluster.