General Availability of UltraWarm for Amazon Elasticsearch Service

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Today, we are happy to announce the general availability of UltraWarm for Amazon Elasticsearch Service.

This new low-cost storage tier provides fast, interactive analytics on up to three petabytes of log data at one-tenth of the cost of the current Amazon Elasticsearch Service storage tier.

UltraWarm, complements the existing Amazon Elasticsearch Service hot storage tier by providing less expensive storage for older and less-frequently accessed data while still ensuring that snappy, interactive experience that Amazon Elasticsearch Service customers have come to expect. Amazon Elasticsearch Service stores data in Amazon S3 while using custom, highly-optimized nodes, purpose-built on the AWS Nitro System, to cache, pre-fetch, and query that data.

There are many use cases for the Amazon Elasticsearch Service, from building a search system for your website, storing, and analyzing data from application or infrastructure logs. We think this new storage tier will work particularly well for customers that have large volumes of log data.

Amazon Elasticsearch Service is a popular service for log analytics because of its ability to ingest high volumes of log data and analyze it interactively. As more developers build applications using microservices and containers, there is an explosive growth of log data. Storing and analyzing months or even years worth of data is cost-prohibitive at scale, and this has led customers to use multiple analytics tools, or delete valuable data, missing out on essential insights that the longer-term data could yield.

AWS built UltraWarm to solve this problem and ensures that developers, DevOps engineers, and InfoSec experts can analyze recent and longer-term operational data without needing to spend days restoring data from archives to an active searchable state in an Amazon Elasticsearch Service cluster.

So let’s take a look at how you use this new storage tier by creating a new domain in the AWS Management Console.

Firstly, I go to the Amazon Elasticsearch Service console and click on the button to Create a new domain. This then takes me through a workflow to set up a new cluster, for the most part setting up a new domain with UltraWarm is identical to setting up a regular domain, I will point out the couple of things you will need to do differently.

On Step 1 of the workflow, I click on the radio button to create a Production deployment type and click Next.

I continue to fill out the configuration in Step 2. Then, near the end, I check the box to Enable UltraWarm data nodes and select the Instance type I want to use. I go with the default ultrawarm1.medium.elasticsearch and then ask for 3 of them, there is a requirement to have at least 2 nodes.

Everything else about the setup is identical to a regular Amazon Elasticsearch Service setup. After having set up the cluster, I then go to the dashboard and select my newly created domain. The dashboard confirms that my newly created domain has 3 UltraWarm data nodes, each with 1516 (GiB) free storage space.

As well as using UltraWarm on a new domain, you can also enable it for existing domains using the AWS Management Console, CLI, or SDK.

Once you have UltraWarm nodes setup, you can migrate an index from hot to warm by using the following request.

POST _ultrawarm/migration/my-index/_warm

You can then check the status of the migration by using the following request.

GET _ultrawarm/migration/my-index/_status
{
  "migration_status": {
    "index": "my-index",
    "state": "RUNNING_SHARD_RELOCATION",
    "migration_type": "HOT_TO_WARM",
    "shard_level_status": {
      "running": 0,
      "total": 5,
      "pending": 3,
      "failed": 0,
      "succeeded": 2
    }
  }
}

UltraWarm is available today on Amazon Elasticsearch Service version 6.8 and above in 22 regions globally.

Happy Searching