Integrating Kinesis Data Streams (KDS) with other AWS services and third-party applications provides businesses with powerful tools for processing, analyzing, and gaining insights from real-time data, which can help them to make better decisions, improve their operations, and gain a competitive advantage. KDS provides 40+ integrations with AWS services and third-party tools allowing you to enable streaming data processing and analytics.
On the producer side, KDS integrates directly with AWS and third-party services. On the consumer side, KDS integrates through Kinesis Data Firehose or directly. Below are some of the commonly used integrations:
Amazon Kinesis Data Streams Producers
AWS Integrations
AWS Amplify
Amazon QLDB
Amazon Quantum Ledger Database is a ledger database that combines the strong transactional consistency of a relational database with the scale of modern event-driven system, the verifiable data integrity common to blockchain, and the flexibility of a document database.
Writing QLDB stream records to KDS.
Amazon Pinpoint
Amazon Pinpoint is a global communications service, powering end customer messaging through channels that includes SMS, push, email and voice.
Streaming Amazon Pinpoint events to KDS.
Amazon Aurora
Amazon Aurora is a MySQL- and PostgreSQL-compatible relational database built for the cloud that combines the performance and availability of traditional enterprise databases with the simplicity and cost effectiveness of open-source databases.
Pushing Amazon Aurora DB activities to KDS in real time
Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS) is a collection of managed services that makes it simple to set up, operate, and scale databases in the cloud.
Streaming Amazon RDS changes to KDS.
Amazon DynamoDB
Amazon DynamoDB is a key-value and document database that delivers single-digit-millisecond performance at any scale.
Using Kinesis Data Streams to capture changes to DynamoDB.
Amazon CloudFront
Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.
Delivering CloudFront Real-time to KDS.
Amazon Connect
Amazon Connect is an omnichannel cloud contact center. You can set up a contact center in a few steps, add agents who are located anywhere, and start engaging with your customers.
Exporting contact records and agent events from Amazon Connect and perform real-time analysis through KDS
Amazon CloudWatch
Amazon CloudWatch collects and visualizes real-time logs, metrics, and event data in automated dashboards to streamline your infrastructure and application maintenance.
Real-time processing of log data with KDS
AWS IOT Core
AWS IoT Core is a managed cloud service that enables connected devices to securely interact with cloud applications and other devices.
Writing to KDS from AWS IOT
Amazon EventBridge
Amazon EventBridge Pipes helps you create point-to-point integrations between event producers and consumers with optional transform, filter and enrich steps. With EventBridge Pipes, you can deliver events to over 14 AWS services. Find other EventBridge integrations here.
From EventBridge to KDS
AWS Database Migration Service
AWS Database Migration Service (AWS DMS) is a cloud service that makes it possible to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores.
Using Amazon Kinesis Data Streams as a target
Kinesis Agent
Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Data Streams.
Writing to KDS using the Kinesis agent.
Kinesis Producer Library (KPL)
The Kinesis Producer Library simplifies producer application development, allowing developers to achieve high write throughput to a Kinesis data stream.
Developing producers using the Amazon Kinesis Producer Library.
Third-party Integrations
Apache Flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
Apache Flink connector for KDS.
Fluentd
Fluentd is an open source data collector for unified logging layer.
Stream processing with Kinesis.
Debezium
Debezium is an open source distributed platform for change data capture.
Streaming MySQL Data Changes to Amazon Kinesis.
Oracle GoldenGate
Oracle GoldenGate is a software product that allows you to replicate, filter, and transform data from one database to another database.
Data replication to Kinesis Data Stream using Oracle GolderGate
Kafka
Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems.
Kinesis kafka connector
Adobe Experience Platform
Adobe Experience Platform enables organizations to centralize and standardize customer data from any system. It then applies data science and machine learning to dramatically improve the design and delivery of rich, personalized experiences.
From Adobe to Kinesis
Striim
Striim is a complete, end-to-end, in-memory platform for collecting, filtering, transforming, enriching, aggregating, analyzing, and delivering data in real time.
Kinesis Writer for Striim
Amazon Kinesis Data Streams Consumers
Through Kinesis Data Firehose
Kinesis Data Firehose
With Kinesis Data Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.
Kinesis Data Firehose can send data records to various destinations, including Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, and any HTTP endpoint that is owned by you or any of your third-party service providers. The following are the supported destinations:
- Amazon OpenSearch
- Amazon Redshift
- Amazon S3
- Coralogix
- Datadog
- Dynatrace
- Honeycomb
- Logic Monitor
- Logz.io
- MongoDB Cloud
- New Relic
- Splunk
- Sumo Logic
Destination settings for above integrations through Kinesis Data Firehose
Amazon Kinesis Data Streams Consumers
Direct AWS Integrations
Amazon Managed Service for Apache Flink
Amazon Managed Service for Apache Flink is the easiest way to transform and analyze streaming data in real time using Apache Flink.
Amazon Managed Service for Apache Flink Resources
Amazon Redshift
Amazon Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and machine learning to deliver the best price performance at any scale.
Streaming Ingestion from KDS
AWS Lambda
AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.
Using AWS Lambda with Amazon Kinesis
Amazon EventBridge
Amazon EventBridge Pipes helps you create point-to-point integrations between event producers and consumers with optional transform, filter and enrich steps. With EventBridge Pipes, you can deliver events to over 14 AWS services. Find other EventBridge integrations here.
From EventBridge to KDS
Amazon Simple Queue Service (SQS)
Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
Use KDS with SQS
Amazon Simple Notification Service (SNS)
Amazon Simple Notification Service (SNS) sends notifications two ways, A2A and A2P. A2A provides high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications. A connection between KDS and SNS can be established using EventBridge Pipes.
AWS Glue
AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.
Streaming ETL jobs in AWS Glue
Direct Third-party Integrations
Kafka Confluent Platform
Confluent Platform is built on top of Kafka and provides additional features and functionality that help enterprises build and manage real-time data pipelines and streaming applications.
Amazon Kinesis Source Connector for Confluent Platform
Talend
Talend is a data integration and management software that allows users to collect, transform, and connect data from various sources in a scalable and efficient manner.
Connect talend to an Amazon Kinesis stream
Databricks
Databricks is a cloud-based platform that provides a collaborative environment for data engineering, data science, and machine learning.
The Kinesis connector for Structured Streaming
Apache Spark
Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
Spark Streaming + Kinesis Integration
Apache Flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
Amazon Kinesis Data Streams Connector
Adobe Experience Platform
Adobe Experience Platform enables organizations to centralize and standardize customer data from any system. It then applies data science and machine learning to dramatically improve the design and delivery of rich, personalized experiences.
From KDS to Adobe
Kinesumer
A Go client implementing a client-side distributed consumer group client for Amazon Kinesis.
Github Link