Posted On: Nov 17, 2023
Amazon OpenSearch Ingestion now offers persistent buffering when ingesting streaming data from push-based sources, offering data durability and simplifying data ingestion architecture. Customers can now use persistent buffering to ingest data from http sources like FluentD and FluentBit and OpenTelemetry collectors without the need to set up a standalone buffer. Persistent buffering in Amazon OpenSearch Ingestion is serverless and scales elastically to meet the throughput needs of even the most demanding workloads.
Customers can turn on persistent buffering for their existing pipelines or create new pipelines with persistent buffering turned on. Amazon OpenSearch Ingestion will then ensure that all the ingested data is persisted to a disk-based buffer that is replicated across multiple AZs for durability before being routed to a destination. The disk storage of persistent buffering prevents it from being overwhelmed when data builds up due to destinations being unreachable ensuring that back pressure is not applied on the source. Along with the end-to-end acknowledgement that is already offered for pull-based sources, Amazon OpenSearch Ingestion now offers complete durability of data for all its sources.
This feature is available in all the AWS commercial regions where Amazon OpenSearch Ingestion is currently available.
To learn more, see the Amazon OpenSearch Ingestion webpage and the Amazon OpenSearch Service Developer Guide.