Integrations are a key way to reduce operational costs, especially when they are built in to your data systems. Through key data integrations between Amazon OpenSearch Service and other AWS services, you’re able to leverage the key benefits of OpenSearch Service on your data, wherever it is stored, oftentimes reducing operational costs and data management complexity. Additionally, data ingestion allows for efficient data conversion, clean up, mapping, and loading, so that the data is available immediately after querying without overloading your cluster’s compute and networking systems.
With Amazon OpenSearch Service, you can easily accomplish all of this, by leveraging Amazon OpenSearch Ingestion, and the integrations with various AWS services including Amazon Kinesis Data Firehose, Logstash, Amazon CloudWatch, or AWS IoT. And with zero-ETL integrations including with Amazon S3 and DynamoDB, you can accomplish even more. These integrations and connections provide you the flexibility to select the integration and ingestion tool that meets your use case requirements.
Data ingestion using Amazon OpenSearch Ingestion
Amazon OpenSearch Ingestion is a capability of Amazon OpenSearch Service for collecting, transforming, and routing data to Amazon OpenSearch Service domains. OpenSearch Ingestion drives cost efficiencies and helps ensure data quality. OpenSearch Ingestion Service runs OpenSearch Data Prepper pipelines as a serverless process that automatically scales with processing needs of your ingest workloads. Learn more »
Amazon OpenSearch Service zero-ETL integration with Amazon S3
The Amazon OpenSearch Service zero-ETL integration with Amazon S3 is a new way of connecting to one’s data, allowing customers to use the rich interactive capabilities of OpenSearch Service directly on infrequently queried data stored in their Amazon S3 data lake with zero-ETL. The zero-ETL connection with S3 removes the need for customers to build or manage complex ETL pipelines, allowing them to perform queries critical to derive insights and visualize security and observability data, while reducing costs and the operational complexity of duplicating data or managing multiple analytics tools. Learn more >>
Zero-ETL integration with DynamoDB
The Amazon OpenSearch Service zero-ETL integration with Amazon DynamoDB allows you to use advanced search capabilities such as full-text and vector search on your operational data in Amazon DynamoDB. This zero-ETL integration uses Amazon OpenSearch Ingestion to seamlessly synchronize your data from Amazon DynamoDB to Amazon OpenSearch Service without having to write any custom code. This zero-ETL integration reduces the operational burden and cost involved in keeping the data in sync between these two stores, allowing you to focus on your applications. Learn More>>
Data ingestion using Amazon Kinesis Data Firehose
With Amazon Kinesis Firehose, you can easily convert raw streaming data from your data sources into the formats required by your Elasticsearch or OpenSearch index and load it to Amazon OpenSearch Service, without having to build your own data processing pipelines.
To use this feature, simply select an AWS Lambda function from the Amazon Kinesis Firehose delivery stream configuration tab in the AWS Management Console. Amazon Kinesis Firehose will automatically apply the AWS Lambda function to every input data record and load the transformed data to your Amazon OpenSearch Service index.
Amazon Kinesis Firehose provides pre-built Lambda blueprints that can be used without any change or customized for converting common data sources such as Apache logs and system logs to JSON and CSV formats. You can also configure Amazon Kinesis Firehose to automatically retry failed jobs and back up the raw streaming data. Learn more »
Data ingestion using Logstash
Amazon OpenSearch Service supports integration with Logstash, an open-source data processing tool that collects data from sources, transforms it, and then loads it to Elasticsearch or OpenSearch. You can easily deploy Logstash on Amazon EC2 and set up your Amazon OpenSearch Service domain as the backend store for all logs coming through your Logstash implementation. Logstash supports a library of pre-built filters to easily perform common transformations such as parsing unstructured log data into structured data through pattern-matching; renaming, removing, replacing, and modifying fields in your data records; and aggregating metrics. Learn more »
Data ingestion using Amazon CloudWatch Logs
Amazon CloudWatch Logs lets you monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can configure a CloudWatch Logs log group to stream data to your Amazon OpenSearch Service domain in near real-time through a CloudWatch Logs subscription. This integration is convenient if you are already using CloudWatch Logs to collect log data, and would like to share that data with your Amazon OpenSearch Service users. Learn more »
Data ingestion using AWS IoT
AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. With AWS IoT, you can capture data from connected devices such as consumer appliances, embedded sensors, and TV set-top boxes. Using the AWS Management console you can configure AWS IoT to load the data directly to Amazon OpenSearch Service, enabling you to provide your customers near real-time access to IoT data and metrics. Learn more »
How to choose the right ingestion mechanism
Choosing the right ingestion mechanism depends on your use case requirements such as data latency and data type. For large data volumes, we recommend using Amazon Kinesis Data Firehose, which is fully managed, automatically scales to match the throughput of your data, and requires no ongoing administration. It can also transform, compress, and batch the data before loading it to Amazon OpenSearch Service domain. Often, the choice also comes down to the services you are already using. For example, if you are already collecting application logs using Amazon CloudWatch Logs, you can simply load that data into your Amazon OpenSearch Service domain without much additional effort.
OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.