Posted On: Aug 31, 2023
Amazon OpenSearch Ingestion now allows you to ingest streaming data from Amazon Managed Streaming for Apache Kafka (MSK), enabling you to seamlessly index the data from Amazon MSK in Amazon OpenSearch Service managed domains or serverless collections without the need for any third-party data connectors. With this integration, you can now use Amazon OpenSearch Ingestion to perform near- real-time aggregations, sampling and anomaly detection on data ingested from Amazon MSK, helping you to build efficient data pipelines to power your complex observability use cases.
Amazon OpenSearch Ingestion pipelines can consume data from one or more topics in an Amazon MSK cluster and transform the data before writing it to Amazon OpenSearch Service or Amazon S3. While reading data from Amazon MSK via Amazon OpenSearch Ingestion, you can configure the number of consumers per topic and tune different fetch parameters for high and low priority data. Furthermore, you can also optionally use AWS Glue Schema Registry to specify your data schema to dynamically read data at ingest time. Also as part of this launch, Amazon OpenSearch Ingestion now supports Data Prepper 2.4.0, which introduces new features like S3 batch processing, filtering in sinks, Avro & Parquet codecs for S3 sinks and improvements to anomaly detection. You can check out the complete list of features in this blog post.
This feature is available in all the AWS commercial regions where Amazon OpenSearch Ingestion is currently available.
To learn more, see the Amazon OpenSearch Ingestion webpage and the Amazon OpenSearch Service Developer Guide.