How do I use Index State Management (ISM) to manage low storage space in Amazon Elasticsearch Service?

Last updated: 2020-12-23

My Amazon Elasticsearch Service (Amazon ES) cluster is running low on storage space. How do I use Index State Management (ISM) in Amazon ES to manage storage space issues?

Short description

Index State Management (ISM) allows you to automate routine tasks and then apply them to indices and index patterns in Amazon ES. With ISM, you can define custom management policies that help you maintain issues such as low disk space. For example, you can use a rollover operation and an ISM policy to automate deletion of old indices based on conditions like index size. The rollover operation rolls over a target to a new index when an existing index meets the defined condition.

To create an ISM policy for an index pattern using an operation like rollover, perform the following steps:

1.    Set up your rollover index.

2.    Create an ISM policy.

3.    Attach the policy to an index.

4.    Add the template.

After you attach your policy to an index, your index begins to initialize, and then transitions into different states until the rollover operation completes. For more information about the rollover operation, see Rollover on the OpenDistro for Elasticsearch website.

Resolution

Set up your rollover index

Create an index and alias where the index format matches the index pattern:

^.*-\d+$.

Important: Make sure to correctly configure your rollover alias. Otherwise, you receive an error message.

In the following example, “test-index-000001” is created and populated with several documents. Because this example uses a rollover index, the index format must match the pattern.

PUT test-index-000001/_doc/1
{
    "user" : "testuser",
    "post_date" : "2020-05-08T14:12:12",
    "message" : "ISM testing"
}

A rollover index requires an alias that points towards the latest index. This means you must create an alias ("test-index") using the following query:

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "test-index-000001", "alias" : "test-index" } }
    ]
}

Create an ISM policy

In the Index Management tab in the Kibana service, create an ISM policy for your rollover operation like this:

{
    "policy": {
        "policy_id": "Roll_over_policy",
        "description": "A test policy. DO NOT USE FOR PRODUCTION!",
        "schema_version": 1,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "rollover": {
                            "min_size": "10mb"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm"
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 2
                        }
                    }
                ],
                "transitions": []
            }
        ]
    }
}

In this ISM policy, there are two defined states: "hot" and "warm." By default, your index is in the "hot" state. The index transitions into the "warm" state as soon as it reaches 10 MB and a new rollover index is created. In the "warm" state, you can perform various actions on the index such as changing the replica count to two, or performing a force_merge operation.

Attach the policy to an index

To attach your ISM policy to an index, perform the following steps:

1.    Open Kibana from the Amazon ES console.

2.    Choose the Index Management tab.

3.    Select the index that you want to attach your ISM policy to (for example: "test-index-000001").

4.    Choose Apply policy.

5.    (Optional) If your policy specifies any actions that require an alias, provide the alias, and then choose Apply. Your index appears under the list of Managed Indices.

Add the template

Attach the policy to a specific index such as "test-index-000002," which was created as an outcome of the ISM policy. This attachment makes sure that the indices will also rollover when the required condition (such as index size) is met.

You can use an index template like this:

PUT _template/template_1
{
  "index_patterns": [
    "test-index*"
  ],
  "settings": {
    "index": {
      "opendistro": {
        "index_state_management": {
          "policy_id": "Roll_over_policy",
          "rollover_alias": "test-index"
        }
      }
    }
  }
}

In this example, the JSON policy ID is "Roll_over_policy". This JSON template makes sure that when the "test-index-000002" index is created, the rollover policy is attached:

GET test-index-000002/_settings
{
  "test-index-000002" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "policy_id" : "Roll_over_policy",
            "rollover_alias" : "test-index"
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "test-index-000002",
        "creation_date" : "1589031424388",
        "number_of_replicas" : "2",
        "uuid" : "DUHeUdTnSveeYsH9aIkt-Q",
        "version" : {
          "created" : "7010199"
        }
      }
    }
  }
}

Note: This index also populates under the Managed Indices section in Kibana's Index Management tab.

ISM policy states

When an ISM policy is attached to an index, the index enters an "Initializing" state. From the "Initializing" state, the index moves into the "Default" state, which is defined in the policy. This "Initializing" operation, and every subsequent operation, can take 30 to 48 minutes. ISM uses this time to perform policy actions, then checks for any conditions and transitions the index into different states. A random jitter of 0-60% is also added to ensure that there aren't surges of activity coming from all indices at the same time.

Note: For a rollover operation, an index is "complete" after the index rolls over, transitions into a "warm" state, and the replica count is updated.


Did this article help?


Do you need billing or technical support?