Amazon Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use and real-time APIs along with the availability, scalability, and security required by production workloads. Amazon Elasticsearch Service is ideally suited for log analytics, application monitoring, interactive search, and other analytics use cases. But collecting, converting, mapping, and loading data from data sources to your Elasticsearch index can be difficult and time-consuming. You have to convert the raw data into a structured data format such as JSON or CSV, clean up the data, and map it to target data fields. You also have to batch and buffer the data for efficient loading so that the data is available immediately for querying without overloading your cluster’s compute and networking resources. To easily accomplish all of this, you can use Amazon Elasticsearch Service’s built-in integrations with Amazon Kinesis Firehose, LogstashAmazon CloudWatch Logs, or AWS IoT depending on your specific use case requirements.

You can configure Amazon Kinesis Firehose to prepare your streaming data before it is loaded to Amazon Elasticsearch Service. With this feature, you can easily convert raw streaming data from your data sources into the formats required by your Elasticsearch index, without having to build your own data processing pipelines.

To use this feature, simply select an AWS Lambda function from the Amazon Kinesis Firehose delivery stream configuration tab in the AWS Management Console. Amazon Kinesis Firehose will automatically apply the AWS Lambda function to every input data record and load the transformed data to your Amazon Elasticsearch index.

Amazon Kinesis Firehose provides pre-built Lambda blueprints for converting common data sources such as Apache logs and system logs to JSON and CSV formats. You can use these pre-built blueprints without any change, customize them further, or write your own custom functions. You can also configure Amazon Kinesis Firehose to automatically retry failed jobs and back up the raw streaming data. Learn more »

Amazon Elasticsearch Service supports integration with Logstash, an open-source data processing tool that collects data from sources, transforms it, and then loads it to Elasticsearch. You can easily deploy Logstash on Amazon EC2, and set up your Amazon Elasticsearch domain as the backend store for all logs coming through your Logstash implementation. Logstash supports a library of pre-built filters to easily perform common transformations such as parsing unstructured log data into structured data through pattern-matching; renaming, removing, replacing, and modifying fields in your data records; and aggregating metrics. Learn more »

Amazon CloudWatch Logs lets you monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can configure a CloudWatch Logs log group to stream data to your Amazon Elasticsearch Service domain in near real-time through a CloudWatch Logs subscription. This integration is convenient if you are already using CloudWatch Logs to collect log data, and would like to share that data with your Amazon Elasticsearch Service users. Learn more »

AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. With AWS IoT, you can capture data from connected devices such as consumer appliances, embedded sensors, and TV set-top boxes, and configure AWS IoT from the AWS Management console to load the data directly to Amazon Elasticsearch Service. This enables you to provide your customers near real-time access to IoT data and metrics. Learn more »

Choosing the right ingestion mechanism depends on the requirements of your specific use case such as data latency and the type of data. For large data volumes, we recommend Amazon Kinesis Firehose, which is fully managed, automatically scales to match the throughput of your data, and requires no ongoing administration. It can also transform, compress, and batch the data before loading it to Amazon Elasticsearch Service domain. Often, the choice also comes down to services you are already using. For example, if you are already collecting application logs using Amazon CloudWatch Logs, you can simply load that data into your Amazon Elasticsearch Service domain without much additional effort.

Get started with our free tier, which provides free usage of up to 750 hours per month of a single-AZ t2.micro.elasticsearch or t2.small.easticsearch instance and 10 GB per month of optional Amazon EBS storage.

Get Started for Free