Get started with Amazon Elasticsearch Service: an easy way to send AWS SigV4 signed requests

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Welcome to this introductory series on Elasticsearch and Amazon Elasticsearch Service (Amazon ES). In this and future blog posts, we provide the basic information that you need to get started with Elasticsearch on AWS.

Introduction

If you are using an IAM policy or policies that specify a user or role Amazon Resource Name (ARN) to control access to your Amazon Elasticsearch Service domain, you must use AWS Signature Version 4 (AWS SigV4) signing on all requests to your domain. For a deeper dive on securing your domain, see also this blog post.

Looking at the AWS documentation on signing requests reveals a process that seems quite complex. Although it looks daunting, it’s fairly easy to write a small piece of code that uses three open-source libraries to sign and send requests to Amazon ES. We use Boto3, the Python Requests library, and the Requests AWS4Auth library.

Code snippet

The following code snippet takes the HTTP method, the URL to call, the service to call, the AWS Region, and an optional request body. This code uses Boto3’s Session class to retrieve credentials—for more information on how Boto3 determines the credentials, see the Boto3 docs. The AWS4Auth class signs the request with the credentials. Finally, the Requests library sends the request to Amazon ES.

Note: Boto3 is AWS’s Python SDK. The Requests and Requests_aws4auth libraries were developed by a third party, not AWS. AWS is not responsible for the functioning or suitability of external content.

import boto3
import requests
from requests_aws4auth import AWS4Auth

def send_signed(method, url, service='es', region='us-west-2', body=None):
    credentials = boto3.Session().get_credentials()
    auth = AWS4Auth(credentials.access_key, credentials.secret_key, 
                  region, service, session_token=credentials.token)

    fn = getattr(requests, method)
    if body and not body.endswith("\n"):
        body += "\n"
    fn(url, auth=auth, data=body, 
       headers={"Content-Type":"application/json"})

For use in your own code, you should add error handling and result parsing, and should return at least the HTTP status code.

Notice that if the body doesn’t end with a newline character, the code adds one. Doing this is necessary in calls that require a final newline—for example, calls to the _bulk API action.

Notice also that the code adds a content header to all requests. As of Elasticsearch 6.0’s strict type checking, this header is required. Although it is not required for earlier versions of Elasticsearch, it’s compatible with them.

Usage

As a simple example, the following code sends a single document to Elasticsearch.

    url = 'http://<Amazon ES Endpoint>/blogs/blog'
    doc = '''{
                 "author": "Jon",
                 "title": "signing is easy!"
             }'''
    send_signed('post', url, body=doc)

Be sure to substitute the actual endpoint of your domain. If you run this code on an Amazon EC2 instance that has a role assigned to it with a policy that allows es:HttpPost, IAM authenticates the POST based on the request’s signature, and Elasticsearch indexes the document.

Conclusion

You might worry that sending AWS SigV4 signed requests is complicated. Don’t let it stop you! As you can see, implementing request signing is simple.

About the Author

Jon Handler (@_searchgeek) is an AWS solutions architect specializing in search technologies. He works with our customers to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using AWS.