How do I troubleshoot an UltraWarm storage migration issue in Amazon OpenSearch Service?

Last updated: 2021-08-06

I'm unable to migrate my Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) index from hot to UltraWarm storage. How do I resolve this issue?

Resolution

When you migrate an index to UltraWarm storage in OpenSearch Service, you might receive the following error message:

{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[USmz1Ze][x.x.x.x:9300][indices:admin/ultrawarm/migration/warm]" } ], "type": "illegal_argument_exception", "reason": "Rejecting migration request for index [index_name] since there isn't enough disk space on hot node [qwertpoiy123lke4woj34n] to perform force merge" }, "status": 400 }

Before an UltraWarm storage migration, OpenSearch Service validates the amount of available hot disk space. OpenSearch Service automatically runs a force merge operation during the disk space validation. When low disk space is detected, the force merge operation request is blocked. The request is blocked because the force merge operation processes one shard at a time. Additionally, each shard requires three times the amount of occupied hot disk space and an additional 20 GB of free storage space per node.

To resolve any low disk space issues before an index migration, perform the following:

1.    Check to see which nodes have shards that exist in the index:

GET _cat/shards/INDEXNAME?v

2.    Check the available disk space per node:

GET /_cat/allocation?v

The nodes must have enough free storage space to fulfill the disk space requirement for a migration (three times the shard size plus 20 GB). For more information about calculating the on-size disk of an index, see Calculating storage requirements.

3.    (Optional) If you don't have enough disk space, then delete old or unused indices. This deletion frees up disk space on the node so that a force merge operation can succeed.

4.    (Optional) If you can't delete indices from the node, then increase the EBS volume. For more information, see How do I add storage space to an Amazon OpenSearch Service domain?

Additional tips:


Did this article help?


Do you need billing or technical support?