How do I troubleshoot an ultraWarm storage migration issue in Amazon Elasticsearch Service?
Last updated: 2021-01-05
I'm unable to migrate my Amazon Elasticsearch Service (Amazon ES) index from hot to ultraWarm storage. How do I resolve this issue?
Resolution
When you migrate an index to ultraWarm storage in Amazon ES, you might receive the following error message:
{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[USmz1Ze][x.x.x.x:9300][indices:admin/ultrawarm/migration/warm]" } ], "type": "illegal_argument_exception", "reason": "Rejecting migration request for index [index_name] since there isn't enough disk space on hot node [qwertpoiy123lke4woj34n] to perform force merge" }, "status": 400 }
Before an UltraWarm storage migration, Amazon ES validates the amount of available hot disk space. Amazon ES automatically runs a force merge operation during the disk space validation. When low disk space is detected, the force merge operation request is blocked. The request is blocked because the force merge operation processes one shard at a time. Additionally, each shard requires twice the amount of hot disk space currently being occupied and an additional 20 GB of free storage space per node.
To resolve any low disk space issues before an index migration, perform the following:
1. Check to see which nodes have shards that exist in the index:
GET _cat/shards/INDEXNAME?v
2. Check the available disk space per node:
GET /_cat/allocation?v
The nodes must have enough free storage space to fulfill the disk space requirement for a migration (twice the shard size plus 20 GB). For more information about calculating the on-size disk of an index, see Calculating storage requirements.
3. (Optional) If you don't have enough disk space, delete old or unused indices. This deletion frees up disk space on the node so that a force merge operation can succeed.
4. (Optional) If you can't delete indices from the node, increase the EBS volume. For more information, see How do I add storage space to an Amazon Elasticsearch Service (Amazon ES) domain?
Additional tips:
- Delete old or unused documents from your index. Too many documents can increase shard sizes, which slow down a migration. Note: Deleting a document doesn't immediately free up disk space. For more information, see Why is the Deleted Documents metric so high in my Amazon Elasticsearch Service cluster?
- Monitor disk utilization metrics (such as FreeStorageSpace) in Amazon CloudWatch. For more information, see ClusterBlockException.
Did this article help?
Do you need billing or technical support?