Amazon OpenSearch Ingestion

Ingest, transform and route data at scale to Amazon OpenSearch Domains and Serverless collections

Why Amazon OpenSearch Service Ingestion?

Amazon OpenSearch Ingestion is a feature of Amazon OpenSearch Service that allows you to ingest, filter, transform, enrich, and route data to an Amazon OpenSearch domain or Serverless collection. Amazon OpenSearch Ingestion is capable of ingesting data from a wide variety of sources and has a rich ecosystem of built-in processors to take care of your most complex data transformation needs. Amazon OpenSearch Ingestion is serverless in nature and will scale automatically to meet the requirements of your most demanding workloads, helping you focus on your business logic while abstracting away the complexity of managing complex data pipelines for your observability and security use cases.

Benefits of Amazon OpenSearch Service

Realize storage cost reductions by deduplicating, sampling, and routing noisy data to lower cost storage.
Enforce data quality by transforming, filtering, and enriching data with built-in processors and by adopting schemas to accelerate observability and reduce security investigation times.
Protect sensitive data by redacting and obfuscating sensitive information before it gets to a destination.
Route data using conditional logic to maintain compliance with data residency laws.

Key features

AWS is a leading contributor of the OpenSearch project, which many customers use. You’ll get all of the new innovations for OpenSearch Data Prepper within this managed service. Beyond those features, which the community drives and contributes to, Amazon OpenSearch Ingestion Service also brings these capabilities:

  • AWS-managed software installation and patching
  • AWS monitors and repairs the service, 24x7
  • AWS upgrades versions
  • Zero downtime for updates and upgrades
  • Availability SLA: 99.9%
  • Serverless, with automatic scaling for ingestion workloads

Customers and partners

CyberArk customer review

“At CyberArk EPM (Endpoint Privilege Manager), a cloud-based multi-tenant system, we manage millions of endpoints and collect high-traffic data events using AWS OpenSearch. By leveraging Amazon OpenSearch Ingestion, we replaced our previous self-managed Logstash pipeline with an AWS-managed one, which eliminated the burden of managing our own infrastructure and provided us with a more scalable, cost-effective, reliable, and secure architecture for our data ingestion. This decision was made with the added advantage of CyberArk EPM achieving FedRAMP High In-Process status, while Amazon OpenSearch Ingestion already being FedRAMP compliant, allowing us to keep high level of security in our offering."

Ori Doolman, Senior Software Architect - CyberArk EPM

CyberArk logo

Calyptia customer review

“At Calyptia we’ve been working with data ingestion for 12+ years as the creators and maintainers of the Cloud Native Computing Foundation project, Fluentd and Fluent Bit. With the latest versions of these projects we are excited for users to gain more control in their first mile with the combination of the Fluent projects and OpenSearch Ingestion Service. With the ingestion service users can continue to scale agents and processing without having to worry about managing and maintaining infrastructure.”

Anurag Gupta, Co-founder - Calyptia

Calyptia logo

Confluent customer review

“We are thrilled to partner with the Amazon OpenSearch team as they build their OpenSearch Ingestion service, which will provide a native integration with Apache Kafka and Confluent. This integration will help our joint customers access real-time data via Apache Kafka inside OpenSearch so they can rethink customer experiences, build real-time backend operations, or launch new products and services. As the leading contributor to Apache Kafka, Confluent has 10X’ed Kafka by building a complete and cloud-native data streaming platform that allows you to move data from wherever it is created to where businesses can take action in the multi-SaasS world we all live in. This allows OpenSearch users to benefit from the 100's of data sources that Confluent is integrated with. We are excited to see what our joint customers build as they set data in motion with Confluent and OpenSearch.”

Paul Mac Farland, VP of Partner & Innovation Ecosystem - Confluent

confluent logo