Get Started with Amazon Elasticsearch Service: Use Dedicated Master Instances to Improve Cluster Stability

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Welcome to this introductory series on Elasticsearch and Amazon Elasticsearch Service (Amazon ES). In this and future articles, we provide the basic information that you need to get started with Elasticsearch on AWS.

What dedicated master instances are and why you need them
Dedicated master instances perform a single function in an Elasticsearch cluster: orchestration. In every Elasticsearch cluster, one instance is the cluster’s master instance. When an Elasticsearch cluster contains only data instances, one of them is elected the cluster master.

The problem is that when traffic gets heavy, the cluster master can get overloaded and drop out of communication. If this happens, your cluster will not respond to write requests until it elects a new master, at which point the cycle might repeat itself. You can alleviate this issue by deploying dedicated master instances—that is, instances that do not respond to API calls and have no other duties. Instead, they just orchestrate.

Instances in an Amazon ES cluster have roles
Elasticsearch is a distributed solution. When you create an Amazon ES cluster, you specify the number and type of instances you want to deploy. Then Amazon ES does the work of spinning up those instances, configuring them, deploying Elasticsearch, and making sure that they can communicate with one another. Elasticsearch uses the instances to store data, respond to read and write requests, hold cluster state, process background tasks, and more.

Instances in an Amazon ES cluster have one of two roles:

Master instances do not directly receive any traffic that you send to Amazon ES. When you send Elasticsearch API calls, the data instances coordinate and respond to the calls. The data instances are able to respond based on the stored cluster state that they receive from the master instance. The master distributes this information to the data instances when you make an API call that changes something in the cluster’s state. For example, if you index a document that has a new field, the data instance coordinates with the master instance to update the schema for that index across the cluster.

Every cluster has a single master instance
First, let’s define some terms:

• Quorum: Amazon ES sets the quorum as half of the eligible instances, rounded down, plus one. If you have three eligible instances, quorum is two. If you have two eligible instances, quorum is also two.
• Eligible: An instance is eligible to be elected the master if it has node.master set to true. When you create a domain with no dedicated master instances, all data instances are master-eligible (node.master=true and node.data=true).

Every cluster elects a single instance to be that cluster’s master instance through a voting process among all the instances that are eligible. To elect a master successfully, a quorum of eligible instances must vote. If the cluster does not have a quorum of eligible instances, it cannot elect a master.

The following figure illustrates an Amazon ES domain with no dedicated master instances. The cluster contains three data instances, all of which are master-eligible. The top-left instance is the master instance for this cluster.

Using dedicated master instances to specify master-only instances
When you create your Amazon ES domain, you can specify dedicated master instances along with data instances. When you do that, the dedicated master instances get only the master instance role (node.master=true), and the data instances get only the data instance role (node.data=true). Data instances serve Elasticsearch API traffic. Master instances perform the master function.

The following diagram shows an Amazon ES domain with dedicated master instances.This domain has three dedicated master instances. The cluster’s master instance is elected from the eligible instances—the three instances that are tagged with the master role. No data instance in this cluster will be elected master.

Creating dedicated master instances
When you create your domain or update your domain’s configuration, you can choose to deploy dedicated master instances. In the console wizard, choose Enable dedicated master.This reveals a set of choices that allow you to control the instance type and instance count for dedicated master instances.Sizing dedicated master instances
When you choose to deploy dedicated master instances, you can choose an instance type that’s different from the instance type of your data instances. Recall that dedicated master instances do different work and usually do much less work than data instances. You will usually select a smaller instance type than you use for your data instances.

When you use dedicated master instances, choose three as the instance count. When you choose three instances as dedicated master instances, it requires two instances to elect a cluster master. This ensures that if your cluster splits based on network issues, with some instances out of communication with the cluster, the orphaned instances cannot create a second cluster master (three does not divide evenly into more than one group with two master-eligible instances).

The following table gives some starting recommendations for dedicated master instance type, based on the number of data instances in the cluster.

* Data instance counts above 20 are available by requesting a limit increase.

The table provides a good baseline. You should expect to adjust master instance size and type for your workload. As with all scale questions, you need to monitor performance and ensure that you have enough CPU and Java virtual machine (JVM) heap for your dedicated masters.

Here are a few factors that drive the selection of dedicated master instances:

• Total number of indexes: The cluster state includes the mappings for all indexes in the cluster. The more indexes, the more mappings, the more you need processing and network power on the cluster master to deliver the state to instances in the cluster.
• Total number of shards: This number is driven partly by the number of indexes, partly by the primary and replica shard count for those indexes. The shard descriptions also take up space in the cluster state and drive resource usage accordingly.
• Rate of change: When you add a new index, or add a new field to the index, you change the cluster state. This change to the cluster state must be distributed to the cluster. The more of these changes you make, the more traffic they generate and the more work the master needs to do.

Conclusion
We recommend using dedicated master instances for all production workloads. The added stability and protection against split-brain are an obvious benefit when Amazon ES is backing a mission-critical application. Dedicated masters usually don’t make sense for development workloads like proof-of-concepts. In most of these cases, the cost of running masters outweighs the marginal benefit of keeping the cluster up and running, especially since creating a new Amazon ES domain is so simple.

About the Author

Jon Handler (@_searchgeek) is an AWS solutions architect specializing in search technologies. He works with our customers to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using AWS.