Posted On: Nov 29, 2022
Amazon Redshift now supports real-time streaming ingestion for Amazon Kinesis Data Streams (KDS) and Amazon Managed Streaming for Apache Kafka (MSK). Amazon Redshift streaming ingestion eliminates the need to stage streaming data in Amazon S3 before ingesting it into Amazon Redshift, enabling customers to achieve low latency, measured in seconds, while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Data engineers, data analysts, and big data developers are evolving their analytics from batch to real-time, adopting streaming engines like Amazon KDS and Amazon MSK, to implement near real-time responsive logic and analytics on streaming application data. Currently, customers who want to ingest real-time data from services like Amazon KDS and Amazon MSK, into Amazon Redshift must first stage the data in Amazon S3 and use the COPY command, which achieves latency in the minutes. With the new streaming ingestion capability in Amazon Redshift, you can use SQL (Structured Query Language) within Redshift to provide the ability to connect to and directly ingest data from multiple Amazon KDS streams or multiple Amazon MSK topics simultaneously. Amazon Redshift streaming ingestion simplifies data pipelines by letting you create materialized views on top of streams directly. The materialized views can also include SQL transforms as part of your ELT (Extract Load Transform) pipeline.
Once the materialized views are defined, streaming data is automatically and continuously ingested from the KDS stream or MSK topic into the Amazon Redshift streaming materialized view when the Auto Refresh feature is enabled. You can also choose to manually refresh the streaming materialized view when direct control over ingest scheduling is desired. This approach allows you to perform downstream processing and transformations of streaming data using existing Amazon Redshift tools and SQL that you are familiar with, at no additional cost.
Amazon Redshift streaming ingestion is now generally available in all Amazon Web Services (AWS) regions where Amazon Redshift is available. To get started with Amazon Redshift streaming ingestion, provision an Amazon Redshift cluster in the CURRENT track. For more information on how to get started with Amazon Redshift streaming ingestion, see the Amazon Redshift documentation page.