In-place version upgrades for Amazon Elasticsearch Service

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Today, Amazon Elasticsearch Service (Amazon ES) announces support for in-place Elasticsearch upgrades for domains that are running version 5.1 or later. This new feature lets you move to the latest release in the same major version (for example, 5.3 to 5.6) or from the latest release in a major version to the latest release in the next major version (for example, 5.6 to 6.3) with a few clicks.  It also preserves the same domain endpoint URL so your services that interface with the domain do not have to change configurations to access the new version.

When compared with the manual effort previously required for domain upgrades, the new feature provides you with a new way to simplify your operations. Prior to this feature, you had to create manual snapshots on the earlier version, create a new domain for the targeted version, restore the manual index snapshot to the new domain, and finally, adjust your application configuration to point to the new domain.

The example in this post demonstrates how to migrate from a 5.1 Elasticsearch domain to a 6.3 Elasticsearch domain.

What to expect

The upgrade process has been automated so that a carefully planned sequence of events helps you move to the target version successfully. During the upgrade, you can expect additional demands to be placed on your domain.

The process entails the following:

• Checking for issues that can block an upgrade
• Creating a snapshot of the Elasticsearch cluster to address a rollback in case of an unsuccessful upgrade
• Relocating shards under the new Elasticsearch version

Starting an upgrade from 5.1 to 5.6

To perform in-place version upgrades for an active domain, sign in to the AWS Management Console and choose Elasticsearch Service. On the Amazon ES console, choose your domain name in the list to open its dashboard. Then choose Upgrade domain.

You are presented with the targeted upgrade version and the options to either upgrade the domain now or check the upgrade eligibility. If you choose Check upgrade eligibility as shown following, the service runs the checks for upgrading the domain to the targeted version.

After you submit the check for upgrade eligibility, you can find the results by choosing the Upgrade history tab.

After validating that the pre-upgrade checks are successful, you can then begin the upgrade to version 5.6. Choose Upgrade domain again to upgrade the domain. Choose Upgrade, and submit your request.

The upgrade process is irreversible, and you can’t pause or cancel it. During an upgrade, you can’t make configuration changes to the domain. In fact, the console disables all buttons that can be used to invoke configuration changes on the cluster. The domain is available for read and write actions on your indexes so that you can still interact with your domain for your services.

After you submit the upgrade request, one final prompt presents you with the chance to cancel the operation. Choose Continue to proceed. You are directed back to the Overview pane, where you can see the domain status appearing as Upgrade processing.

In the same Overview pane, you can get additional detail about the upgrade in the Upgrade status field.  The combination of these two fields provides a clear picture of the processing for an upgrade.

Depending on the size of your domain, this process can take anywhere from several minutes to several hours.

Reviewing the new 5.6 version

When the domain has completed the upgrade process from 5.1 to 5.6, you will notice that the Elasticsearch version number has changed from 5.1 to 5.6. At this time, you can run your validation tests on the domain to assert performance gains or other features that you intended to access with the upgrade.

Upgrading from a minor version to a major version

In the previous sections, you reviewed the process for performing a minor version upgrade. You took an existing 5.1 version domain and upgraded it to version 5.6. Major versions introduce new features and performance enhancements along with bug fixes typically addressed in minor versions.

When new features are introduced, they can introduce breaking changes. For example, in version 5.x, you can have multiple mapping types for an index. In 6.x, multiple mapping types are not supported. Make sure that your code or operational activities take these things into account. Amazon ES performs a series of steps to help address these types of issues. However, your client code still needs to address the changes that could cause it to fail against a new Elasticsearch version.

To upgrade from version 5.6 to version 6.3, perform the same steps you did for the minor version upgrade from version 5.1 to version 5.6. Choose Upgrade domain again to upgrade the domain to version 6.3.

As you saw in the prior upgrade procedure, your domain status is Upgrade processing. The 5.6 version appears as the upgrade source.

Again, depending on the size of your domain, this process can take anywhere from several minutes to several hours. You can still query your data during the process.

Reviewing the entire upgrade process

After the upgrade has finalized, your domain is now at version 6.3. Compared to versions 5.x of the Elasticsearch product, version 6.x provides enhancements such as optimizations for sparsely populated fields and faster queries with sorted indexes.

Navigate to the Upgrade history tab to review the entire sequence of events for the upgrade from version 5.1 to version 6.3.

As shown in the Upgrade history pane, you can see the progression and the checks that were performed during the upgrade process. Your domain is now at version 6.3.

Conclusion

The new Amazon ES in-place version upgrade feature provides a painless way to upgrade your Elasticsearch domains to the latest minor and major versions of Elasticsearch, from version 5.1 forward. This feature releases your team from many of the operational burdens of performing manual version upgrades. The feature also makes your domain management more seamless.

You can incorporate these steps into your scripts and automation processes by using the new features for Elasticsearch version upgrades in the AWS CLI and AWS SDK. Moving your domains to the latest version not only keeps your development team happy, but it also keeps your customers happy by offering consumers of your services increased performance and features.

About the Author

Kevin Fallis is an AWS solutions architect specializing in search and other distributed parallel processing technologies.  His passion at AWS is to help customers leverage the correct mix of AWS services to achieve success for their business goals.  His after work activities include family, music, DIY projects, and coaching sports.