How do I use Index State Management (ISM) to manage low storage space in Amazon OpenSearch Service?

Last updated: 2021-09-16

My Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) cluster is running low on storage space. How do I use Index State Management (ISM) in OpenSearch Service to manage storage space issues?

Short description

Index State Management (ISM) allows you to automate routine tasks and then apply them to indices and index patterns in OpenSearch Service. With ISM, you can define custom management policies that help you maintain issues such as low disk space. For example, you can use a rollover operation and an ISM policy to automate deletion of old indices based on conditions such as index size. The rollover operation rolls over a target to a new index when an existing index meets the defined condition.

To create an ISM policy for an index pattern using an operation like rollover, perform the following steps:

1.    Set up your rollover index.

2.    Create an ISM policy.

3.    Attach the policy to an index.

4.    Add the template.

After you attach your policy to an index, your index begins to initialize and then transitions into different states until the rollover operation completes. For more information about the rollover operation, see Rollover on the OpenDistro for Elasticsearch website.

Resolution

Set up your rollover index

Create an index and alias where the index format matches the index pattern:

^.*-\d+$.

Important: Make sure to correctly configure your rollover alias. Otherwise, you receive an error message.

In the following example, “test-index-000001” is created and populated with several documents. Because this example uses a rollover index, the index format must match the pattern.

PUT test-index-000001/_doc/1
{
    "user" : "testuser",
    "post_date" : "2020-05-08T14:12:12",
    "message" : "ISM testing"
}

A rollover index requires an alias that points towards the latest index. This means that you must create an alias ("test-index") using the following query:

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "test-index-000001", "alias" : "test-index" } }
    ]
}

Create an ISM policy

In OpenSearch Dashboards, choose the Index Management tab, and then create an ISM policy for your rollover operation.

For example:

{
    "policy": {
        "policy_id": "Roll_over_policy",
        "description": "A test policy. DO NOT USE FOR PRODUCTION!",
        "schema_version": 1,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "rollover": {
                            "min_size": "10mb"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm"
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 2
                        }
                    }
                ],
                "transitions": []
            }
        ]
    }
}

In this ISM policy, there are two defined states: "hot" and "warm." By default, your index is in the "hot" state. The index transitions into the "warm" state as soon as the size of the index reaches 10 MB and a new rollover index is created. In the "warm" state, you can perform various actions on the index such as changing the replica count to two, or performing a force_merge operation.

Attaching the policy to an index

To attach your ISM policy to an index, perform the following steps:

1.    Open OpenSearch Dashboards from the OpenSearch Service console.

2.    Choose the Index Management tab.

3.    Select the index that you want to attach your ISM policy to (for example: "test-index-000001").

4.    Choose Apply policy.

5.    (Optional) If your policy specifies any actions that require an alias, provide the alias, and then choose Apply. Your index appears under the list of Managed Indices.

Adding the template

Attach the policy to a specific index such as "test-index-000002," which was created as an outcome of the ISM policy. This attachment makes sure that the indices also rollover when the required condition (such as index size) is met.

You can use an index template like this:

PUT _template/template_1
{
  "index_patterns": [
    "test-index*"
  ],
  "settings": {
    "index": {
      "opendistro": {
        "index_state_management": {
          "policy_id": "Roll_over_policy",
          "rollover_alias": "test-index"
        }
      }
    }
  }
}

In this example, the JSON policy ID is "Roll_over_policy". This JSON template makes sure that when the "test-index-000002" index is created, the rollover policy is attached:

GET test-index-000002/_settings
{
  "test-index-000002" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "policy_id" : "Roll_over_policy",
            "rollover_alias" : "test-index"
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "test-index-000002",
        "creation_date" : "1589031424388",
        "number_of_replicas" : "2",
        "uuid" : "DUHeUdTnSveeYsH9aIkt-Q",
        "version" : {
          "created" : "7010199"
        }
      }
    }
  }
}

Note: This index also populates under the Managed Indices section in OpenSearch Dashboard's Index Management tab.

ISM policy states

When an ISM policy is attached to an index, the index enters an "Initializing" state. From the "Initializing" state, the index moves into the "Default" state, which is defined in the policy. This "Initializing" operation, and every subsequent operation, can take 30 to 48 minutes. ISM uses this time to perform policy actions, and then checks for any conditions and transitions the index into different states. A random jitter of 0-60% is also added to make sure that there aren't surges of activity coming from all indices at the same time.

Note: For a rollover operation, an index is "complete" after the index rolls over, transitions into a "warm" state, and the replica count is updated.

If you're using an ISM policy and the index isn't properly migrating, check the status of the ISM.

To check the status of the migration for particular index, use the following syntax:

GET _ultrawarm/migration/<put_index_name_here>/_status

To get a summary migration of all indices, use the following syntax:

GET _ultrawarm/migration/_status?

Did this article help?


Do you need billing or technical support?