How do I troubleshoot an UltraWarm storage migration issue in Amazon OpenSearch Service?

2 minute read
0

I'm unable to migrate my Amazon OpenSearch Service index from hot to UltraWarm storage. How do I resolve this issue?

Resolution

When you migrate an index to UltraWarm storage in OpenSearch Service, you might receive the following error message:

{
     "error": {
          "root_cause": [{
               "type": "remote_transport_exception",
               "reason": "[USmz1Ze][x.x.x.x:9300][indices:admin/ultrawarm/migration/warm]"
          }],
          "type": "illegal_argument_exception",
          "reason": "Rejecting migration request for index [index_name] since there isn't enough disk space on hot node [qwertpoiy123lke4woj34n] to perform force merge"
     },
     "status": 400
}

Before an UltraWarm storage migration, OpenSearch Service validates the amount of available hot disk space. OpenSearch Service automatically runs a force merge operation during the disk space validation. When low disk space is detected, then the force merge operation request is blocked. The request is blocked because the force merge operation processes one shard at a time. Additionally, each shard requires three times the amount of occupied hot disk space and an additional 20 GB of free storage space per node.

To resolve any low disk space issues before an index migration, perform the following:

1.    Check to see which nodes have shards that exist in the index:

GET _cat/shards/INDEXNAME?v

2.    Check the available disk space per node:

GET /_cat/allocation?v

The nodes must have enough free storage space to fulfill the disk space requirement for a migration (three times the shard size plus 20 GB). For more information about calculating the on-size disk of an index, see Calculating storage requirements.

3.    (Optional) If you don't have enough disk space, then delete old or unused indices. This deletion frees up disk space on the node so that a force merge operation can succeed.

4.    (Optional) If you can't delete indices from the node, then increase the Amazon Elastic Block Store (Amazon EBS) volume. For more information, see How do I troubleshoot low storage space in my OpenSearch Service domain?

Additional tips:


AWS OFFICIAL
AWS OFFICIALUpdated 3 years ago