The Streaming Data Solution for Amazon Kinesis provides AWS CloudFormation templates where data flows through producers, streaming storage, consumers, and destinations. To support multiple use cases and business needs, the Solution offers four CloudFormation template. Similar to the Streaming Data Solution for Amazon MSK, the templates are configured to apply best practices to monitor functionality using dashboards and alarms, and to secure data.
Benefits
AWS Solution overview
-
Option 1
-
Option 2
-
Option 3
-
Option 4
-
Option 1
-
AWS CloudFormation template using Amazon API Gateway and AWS Lambda
Click to enlargeAWS CloudFormation template using Amazon API Gateway and AWS Lambda
Streaming Data Solution for Amazon Kinesis architecture
Option 1 – AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda
This AWS CloudFormation template deploys a reference architecture that includes the following:
- An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.
- An Amazon Cognito user pool is used to control who can invoke REST API methods.
- Kinesis Data Streams to store the incoming streaming data.
- An AWS Lambda function processes the records from the data stream.
- Errors and failed records that occur during AWS Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS). The queue stores metadata for failed batch records and Lambda errors, allowing customers to retrieve these records and determine the next steps to resolve them.
Streaming Data Solution for Amazon Kinesis
Estimated cost Source code Option 1: AWS CloudFormation template using API Gateway, Kinesis Data Streams, and AWS LambdaVersion 1.7.2
Release date: 01/2023
Author: AWSEstimated deployment time: 5-10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
-
Option 2
-
AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
Click to enlargeAWS CloudFormation template using AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
Streaming Data Solution for Amazon Kinesis architecture
Option 2 – AWS CloudFormation template using AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
This AWS CloudFormation template deploys a reference architecture that includes the following:
- An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.
- Amazon Kinesis Data Streams stores the incoming data stream.
- Kinesis Data Analytics Studio processes the incoming records and saves the processed data in an Amazon Simple Storage Service (Amazon S3) bucket.
- An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.
Streaming Data Solution for Amazon Kinesis
Version 1.7.2
Release date: 01/2023
Author: AWSEstimated deployment time: 5-10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
-
Option 3
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
Click to enlargeAWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
Streaming Data Solution for Amazon Kinesis architecture
Option 3 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
This option deploys a reference architecture that includes the following:
- Amazon Kinesis Data Streams stores the incoming streaming data.
- Amazon Kinesis Data Firehose buffers the data before delivering the output to an Amazon S3 bucket. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration.
- An Amazon CloudWatch dashboard monitors the data ingestion and buffering. CloudWatch alarms are set on essential metrics for Kinesis Data Firehose.
Streaming Data Solution for Amazon Kinesis
Version 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 5-10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
- Amazon Kinesis Data Streams stores the incoming streaming data.
-
Option 4
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
Click to enlargeAWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
Streaming Data Solution for Amazon Kinesis architecture
Option 4 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
This option deploys a reference architecture similar to Option 2 and includes the following:
- An Amazon Elastic Compute Cloud (Amazon EC2) instance that uses the Amazon Kinesis Producer Library (KPL) to generate data.
- Kinesis Data Streams stores the incoming streaming data.
- Kinesis Data Analytics processes the incoming records and asynchronously invokes an external endpoint.
- The demo application invokes an AWS Lambda function.
- The external API can be any integration supported by Amazon API Gateway (for example, an Amazon SageMaker endpoint).
- An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.
Streaming Data Solution for Amazon Kinesis
Version 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 5-10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
- An Amazon Elastic Compute Cloud (Amazon EC2) instance that uses the Amazon Kinesis Producer Library (KPL) to generate data.
Related content
This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Kinesis Data Analytics (KDA) with Apache Flink to asynchronously invoke underlying services or databases.
Browse our library of AWS Solutions to get answers to common architectural problems.
Find AWS Partners to help you get started.
Find prescriptive architectural diagrams, sample code, and technical content for common use cases.