How do I troubleshoot an ultraWarm storage migration issue in Amazon Elasticsearch Service?

Last updated: 2021-01-05

I'm unable to migrate my Amazon Elasticsearch Service (Amazon ES) index from hot to ultraWarm storage. How do I resolve this issue?

Resolution

When you migrate an index to ultraWarm storage in Amazon ES, you might receive the following error message:

{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[USmz1Ze][x.x.x.x:9300][indices:admin/ultrawarm/migration/warm]" } ], "type": "illegal_argument_exception", "reason": "Rejecting migration request for index [index_name] since there isn't enough disk space on hot node [qwertpoiy123lke4woj34n] to perform force merge" }, "status": 400 }

Before an UltraWarm storage migration, Amazon ES validates the amount of available hot disk space. Amazon ES automatically runs a force merge operation during the disk space validation. When low disk space is detected, the force merge operation request is blocked. The request is blocked because the force merge operation processes one shard at a time. Additionally, each shard requires twice the amount of hot disk space currently being occupied and an additional 20 GB of free storage space per node.

To resolve any low disk space issues before an index migration, perform the following:

1.    Check to see which nodes have shards that exist in the index:

GET _cat/shards/INDEXNAME?v

2.    Check the available disk space per node:

GET /_cat/allocation?v

The nodes must have enough free storage space to fulfill the disk space requirement for a migration (twice the shard size plus 20 GB). For more information about calculating the on-size disk of an index, see Calculating storage requirements.

3.    (Optional) If you don't have enough disk space, delete old or unused indices. This deletion frees up disk space on the node so that a force merge operation can succeed.

4.    (Optional) If you can't delete indices from the node, increase the EBS volume. For more information, see How do I add storage space to an Amazon Elasticsearch Service (Amazon ES) domain?

Additional tips:


Did this article help?


Do you need billing or technical support?