How do I use ISM to manage low storage space in Amazon OpenSearch Service?

7 minute read
0

My Amazon OpenSearch Service cluster is running low on storage space.

Short description

Index State Management (ISM) allows you to automate routine tasks and then apply them to indices and index patterns in OpenSearch Service. With ISM, you can define custom management policies that help you maintain issues such as low disk space. For example, you can use a rollover operation and an ISM policy to automate deletion of old indices based on conditions such as index size. The rollover operation rolls over a target to a new index when an existing index meets the defined condition.

To create an ISM policy for an index pattern using an operation like rollover, perform the following steps:

1.    Set up your rollover index.

2.    Create an ISM policy.

3.    Attach the policy to an index.

4.    Add the template.

After you attach your policy to an index, your index begins to initialize and then transitions into different states until the rollover operation completes. For more information about the rollover operation, see rollover on the Open Distro for OpenSearch website.

Resolution

Set up your rollover index

Create an index and alias where the index format matches the index pattern:

^.*-\d+$

Important: Make sure to correctly configure your rollover alias. Otherwise, you receive an error message.

In the following example, "test-index-000001" is created and populated with several documents. Because this example uses a rollover index, the index format must match the pattern.

PUT test-index-000001/_doc/1
{
  "user": "testuser",
  "post_date": "2020-05-08T14:12:12",
  "message": "ISM testing"
}

A rollover index requires an alias that points towards the latest index. This means that you must create an alias ("test-index") using the following query:

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "test-index-000001",
        "alias": "test-index"
      }
    }
  ]
}

Note: If a rollover operation is included in the ISM policy, you must include rollover alias. For more information, see Why does the rollover index action in my ISM policy keep failing in Amazon OpenSearch Service?

Create an ISM policy

In OpenSearch Dashboards, choose the Index Management tab, and then create an ISM policy for your rollover operation.

For example:

Rollover to warm state

{
  "policy": {
    "policy_id": "Roll_over_policy",
    "description": "A test policy. DO NOT USE FOR PRODUCTION!",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm"
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 2
            }
          }
        ],
        "transitions": []
      }
    ]
  }
}

In this ISM policy, there are two defined states: "hot" and "warm." By default, your index is in the "hot" state. The index transitions into the "warm" state as soon as the size of the index reaches 10 MB and a new rollover index is created. In the "warm" state, you can perform various actions on the index such as changing the replica count to two or performing a force_merge operation.

Rollover to delete after few days

{
  "policy": {
    "policy_id": "Roll_over_policy",
    "description": "A test policy. DO NOT USE FOR PRODUCTION!",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ]
  }
}

In this ISM policy, there are two defined states: "hot" and "delete" By default, index is in the "hot" state. After the index reaches 10 MB, a new rollover index is created. Then, after 30 days the index transitions to the "delete" state and the index is deleted.

Attaching the policy to an index

To attach your ISM policy to an index, perform the following steps:

1.    Open OpenSearch Dashboards from the OpenSearch Service console. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    Choose the Index Management tab.

3.    Select the index that you want to attach your ISM policy to (for example: "test-index-000001").

4.    Choose Apply policy.

5.    (Optional) If your policy specifies any actions that require an alias, provide the alias, and then choose Apply. Your index appears under the list of Policy Managed Indices.

Updating the policy of an existing index

Note : Any update made in the existing policy doesn't apply to existing indices automatically, it requires a reapplication of the same policy to the indices.

To reapply your ISM policy to any existing index, perform the following steps:

1.    Open OpenSearch Dashboards from the OpenSearch Service console.

2.    Choose the Index Management tab.

3.    From the Policy Managed Indices section, choose Change Policy.

4.    Choose the indices that you want to apply change to (for example: "test-index-000001").

5.    Choose the current state of the indices.

6.    From the Choose New Policy section, choose update policy name.

7.    (Optional) If you want to switch the indices to another state after the policy's updated, choose Switch indices to the following state after the policy takes effect. Then, choose the state from dropdown list.

Adding the template

Attach the policy to a specific index such as "test-index-000002," which was created as an outcome of the ISM policy. With this attachment, the indices also rollover after the required condition (such as index size) is met.

You can create and use an ISM template like this one:

PUT _plugins/_ism/policies/test_policy
{
  "policy": {
    "description": "A test policy. DO NOT USE FOR PRODUCTION!",
    "last_updated_time": 1642027350875,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm"
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 2
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": [
        "test*"
      ],
      "priority": 100
    }
  }
}

In this example, the explain index API verifies that the "test_policy" template that you created is attached to the newly created index:

GET _plugins/_ism/explain/test-index-000002
{
  "test-index-000002": {
    "index.plugins.index_state_management.policy_id": "test_policy",
    "index.opendistro.index_state_management.policy_id": "test_policy",
    "index": "test-index-000002",
    "index_uuid": "CZrQ-RzRS8SmiWIuyqFmVg",
    "policy_id": "test_policy",
    "enabled": true
  },
  "total_managed_indices": 1
}

Note: This index also populates under the Managed Indices section in the OpenSearch Dashboards Index Management tab.

ISM policy states

When an ISM policy is attached to an index, the index enters an "Initializing" state. From the "Initializing" state, the index moves into the "Default" state, which is defined in the policy. This "Initializing" operation, and every subsequent operation, can take 30 to 48 minutes. ISM uses this time to perform policy actions, and then checks for any conditions and transitions the index into different states. A random jitter of 0-60% is also added to make sure that there aren't surges of activity coming from all indices at the same time.

Note: For a rollover operation, an index is "complete" after the index rolls over, transitions into a "warm" state, and the replica count is updated.

If you're using an ISM policy and the index isn't properly migrating, then check the status of the ISM.

To check the status of the migration for particular index, use the following syntax:

GET _ultrawarm/migration/<put_index_name_here>/_status

To get a summary migration of all indices, use the following syntax:

GET _ultrawarm/migration/_status?

Related information

Example policies

AWS OFFICIAL
AWS OFFICIALUpdated a year ago