September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

In this blog post, we show how you can secure your Amazon Elasticsearch Service (Amazon ES) domain with authentication and authorization based on Microsoft Active Directory (AD). You do so by using an Nginx reverse proxy, running custom authorization code. Amazon ES doesn’t have any built-in support for integration with AD/LDAP for access control. This blog post provides a simple, yet powerful solution to this problem.

Amazon Elasticsearch Service (Amazon ES) is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. This service offers built-in integrations with Kibana, Logstash, and AWS services including Amazon Kinesis Firehose, AWS Lambda, and Amazon CloudWatch, so that you can build solutions quickly.

With Amazon ES, you can use AWS Identity and Access Management (IAM) to have fine-grained access control to your data in the cluster. In a previous post, you learned how to control access to your Amazon ES domain by applying resource-based permissions, identity-based permissions, and Signature Version 4 signing in your client applications. After reading this post, you should be able to deploy an Amazon Elasticsearch domain integrated with your corporate AD (running on-premises or in AWS). You also should be able to use AD users and groups for fine-grained access control to your cluster and to the built-in Kibana dashboard. Using this approach, your corporate users can access Amazon ES indexes and Kibana dashboards using Single Sign-On (SSO). You can also use this approach to control which user, group, or service account gets access to what. And you can do so without creating IAM user credentials for every single BI or analytics user in your organization.

Solution overview

The following diagram shows our solution.Here’s how the solution works at a high level:

The client application or applications and any browser-based applications (for example, Kibana App) access Amazon ES through the Nginx reverse proxy’s IP address or DNS name. The clients accessing the proxy endpoint need to supply an AD user name and password (personal account or service account) in the HTTP or HTTPS request’s basic authorization header. Providing an invalid user name or password returns a 401 not authorized error. Providing a valid user name or password but with insufficient access to the Amazon ES domain returns a 403 forbidden error.

The Amazon EC2 instance that the Nginx proxy uses resides in a public subnet. This EC2 instance does all the heavy lifting. This work includes receiving HTTP and HTTPS requests from clients, authenticating with the AD, checking IAM role-based authorization, and forwarding the request to Amazon ES. The solution in this post isn’t a high availability design, and it should only be used as a reference and learning example. Consider it as a starting idea and build upon it. For more ideas, see the possible enhancements section at the end of this post.

Active Directory resides either in a corporate data center or in AWS. The Nginx proxy EC2 instance communicates with the AD using LDAPS. The AWS-managed Active Directory service now supports LDAPS. So if you’re using AWS-managed AD or planning to host AD in AWS, you should consider turning on LDAPS support. For more information, read the post How to enable LDAPS for your AWS Microsoft Active Directory on the AWS Security Blog.

Your Amazon ES domain resides in your AWS account.  You have two options to choose for the endpoint access—either a public endpoint or a VPC endpoint. In this post, we use the public endpoint option. However, this solution can also work with a VPC endpoint for Amazon ES with some design changes.

Let’s now look at how this works on the inside. The Nginx reverse proxy has a custom authentication and authorization code that runs as a local daemon. Every HTTP and HTTPS request, hitting the proxy, is first routed to the auth daemon running locally on the proxy instance (port 8888). This routing uses the subrequest feature of Nginx proxy. The auth daemon verifies the user name and password supplied in the request header by making an LDAP bind call to the AD and then using the LDAP search call.

If the credentials supplied in the request header are invalid, the subrequest fails and the proxy returns an HTTP 401 not authorized error. If the AD credentials are valid, the auth daemon retrieves a list of all AD groups for that user. It only searches the AD groups that begin with a specified prefix (for example, ‘AWS‘). For each AD group found, the auth daemon looks up the matching IAM role (that is, the IAM role with the exact same name as the AD group). The auth daemon then evaluates the IAM role’s policies to determine if that IAM role is permitted to perform the specified action. The action might be an HTTP method call from the request header, for example GET, POST, PUT, DELETE, or HEAD. The action is performed on the specified resource, the URL from the request header. This evaluation uses the IAM policy simulator API.

The auth daemon iterates through all the matching IAM roles for the AD groups of the user, until a successful result is found (that is, an ALLOWED result). If the auth daemon finds an IAM role that has permissions to perform the specified action on the specified resource, then the auth daemon returns HTTP 200 for the subrequest. It also forwards the original request to the Amazon ES domain. Otherwise, the auth daemon returns HTTP 403 forbidden. The Amazon ES domain responds to the request, forwarded by the Nginx proxy. The access control policy on the Amazon ES domain is configured to allow all requests from the public IP of the Nginx proxy, using an IP-based access policy. So the Amazon ES domain trusts the proxy and the proxy authenticates and authorizes the incoming requests.

Let’s look at an example to understand this.

Assume that your organization has Bob and Alice, two AD users who need access to Amazon ES with different permission levels. Bob needs to have administrator permissions to perform all actions on all resources in the Amazon ES domain. Alice needs to have only read-only permission to the orders index.

You create two AD groups and add each user to the appropriate AD group. Bob is added to the AD group called AWSESAdmin and Alice is added to the AD group called AWSESUser. You also create two IAM roles in your AWS account, with the same names as the AD groups. Each IAM role has a different policy attached to it, giving different access permissions on the Amazon ES domain.

The example IAM policy for the AWSESAdmin IAM role is here:

{
    "Version": "2012-10-17",
    "Statement": [
      {
      "Action": ["es:*"],
      "Resource": 
		"arn:aws:es:ap-southeast-2:111122223333:domain/mydomain/*",
      "Effect": "Allow"
      }
    ]
}

This example IAM policy for the AWSESUser IAM role is here:

{
    "Version": "2012-10-17",
    "Statement": [
      {
      "Action": ["es:ESHttpGet"],
      "Resource": 
		"arn:aws:es:ap-southeast-2:111122223333:domain/mydomain/orders/*",
      "Effect": "Allow"
      }
    ]
}

When Bob and Alice access Amazon ES, through the proxy, their credentials are validated. Their AD group membership determines what access is allowed. Bob has the AWSESAdmin role, so the AWSESAdmin IAM role is evaluated for full access. Alice has the AWSESUser role, so the AWSESUser IAM role is evaluated for read-only (HTTP GET) access to the orders index.

In this model, AD and IAM are used together to control access to the Amazon ES domain. The key security features of this solution are these:

• Corporate AD is the source of authentication. You don’t need to create an IAM user for every user and service in your organization that needs to access Amazon ES. You only need to create a set of AD groups (for example, AWSESAdmin, AWSESDataLoader, AWSESReadOnly, and so on) and put users in the appropriate groups.
• IAM is the source of authorization. This feature means that you can create fine-grained access policies on the Amazon ES resources. You create a limited set of IAM roles, each of which has one-to-one mapping with a parallel item in one of your AD groups. Thus, you can achieve complete control over your Amazon ES domain. At the same time, you can give users and services easy access.

Deploying the solution

Let’s now walk through the steps for deploying this solution. You need to have certain prerequisites in place before deploying the solution:

• You should have an existing VPC with a public subnet. The Nginx proxy is deployed in this public subnet with an Elastic IP address.
• You should have an AD server accessible from the public subnet. This server can either be an AD in your data center, an AD running on an Amazon EC2 instance, or an AWS-managed AD.
• You should have a set of service account credentials that the proxy can use to bind with the AD and make LDAP queries. You need the fully qualified domain name of this service account and its password.

When you have all these in place, you can follow these steps to deploy and run the solution.

Step 1: Gather the inputs required to create the solution

Complete this table by populating the input parameter values that you need to launch the AWS CloudFormation template.

Step 2: Prepare your AD for user and group management

Create the following three user groups in your AD and add users to these groups, as needed.

Important: Remember to change the prefix to suit your needs and it must match the GroupPrefix input parameter.

• AWSESAdmin: This group should contain all the Admin users that need full access to the Amazon ES domain.
• AWSESDataLoader: This group should contain the users and services that need read-write access to the Amazon ES domain.
• AWSESUser: This group should contain the users and services that need read-only access to the Amazon ES domain.

Step 3: Download and launch the CloudFormation template

Get the CloudFormation template for this solution. Launch the template in your AWS account with the input parameters you have gathered. The template creates the following resources:

• Amazon ES domain with IP-based access policy
• Single Nginx proxy with an Elastic IP Address
• Three IAM roles with different policies to access the Amazon ES domain

The template gives the following outputs:

1. Amazon Resource Name (ARN) of the Amazon ES domain.
2. Private IP URLs for the Amazon ES domain through the proxy. You use these to access the Amazon ES domain.
3. Private IP URLs for the in-built Kibana dashboard through the proxy. You use these for accessing the Kibana dashboard.

Step 4: Test the solution

After you launch the CloudFormation template and the resources are created, you can test the solution. You do so by going to the private IP URL of the proxy from your web browser or by making a REST API call (for example, with Postman or SOAPUI tools). We suggest that you use the HTTP port for your initial testing, because you get SSL certificate warning errors on the HTTPS port. This result occurs because the Nginx proxy server in this solution uses a self-signed certificate.

If you use your web browser, you are prompted to enter your AD credentials. Make sure that your AD user belongs to one of the AD user groups you created in step 2 above. If you use a tool or client application to make a REST API call, you should provide the AD user credentials as basic authorization in the HTTP request header.

Possible enhancements

You can make a couple of enhancements to this solution to suit your needs. As mentioned earlier, this blog post gives you only an idea that you can use and build upon. If you intend to use this approach for production purposes, you should consider the following at the least:

1. The Nginx proxy can be the single point of failure. You should use the AWS services Auto Scaling and Elastic Load Balancing to achieve high availability and better performance.
2. The Nginx proxy uses a self-signed SSL certificate. You should use a trusted CA certificate for better security posture. Doing so also prevents SSL warnings on your web browsers and client applications.
3. As mentioned in the AWS documentation, the rest.action.multi.allow_explicit_index advanced option has access control implications that you should be aware of.  Setting this value to false (parameter ESDomainAllowExplicitIndex in the supplied template) prevents users from bypassing restrictions. However, it breaks Kibana. We suggest that you define your IAM policies taking this into consideration. Rather than mixing broad allows and focused denies, the safest approach is to follow the principle of least privilege. In this approach, you grant only the permissions that are required to perform a task.
4. The complete code and configuration files for this blog post are on GitHub here. You can download and extend the code for your particular use case.

Conclusion

Using the solution in this blog post, you can add fine-grained authentication and authorization to your Amazon ES domain. The combination of LDAP/AD and IAM gives you a simple, relatively secure way to achieve your requirements. If you have questions or comments about this blog post, please leave a message in the comments below.

About the Author

Ali Asgar Juzer is a consultant at Amazon Web Services. He works with our customers to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using AWS.