Important: Streaming Data Solution for Amazon Kinesis will retire on January 18, 2025. After that time, all existing deployments will continue to work and existing customers will retain full control of their environments and data; however, the Solution will no longer be supported or maintained.
Overview
The Streaming Data Solution for Amazon Kinesis comes with four deployment options and their accompanying AWS CloudFormation templates that are configured to apply best practices for streaming data, including data monitoring through dashboards and alarms, as well as data security.
Streaming data must be durably captured by massively scalable storage that is capable of handling high data volume from data producers. A producer can consist of thousands of data sources, each continuously generating streaming data. These sources typically submit records simultaneously in small sizes (kilobytes).
This AWS Solution helps you manage, secure, and store a wide variety of streaming data, such as log files generated by your customers using mobile or web applications. It also supports streaming data from ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers.
Benefits
Technical details
You can automatically deploy this architecture using the implementation guide and the four accompanying AWS CloudFormation templates.
-
Option 1
-
Option 2
-
Option 3
-
Option 4
-
Option 1
-
AWS CloudFormation template using Amazon API Gateway and AWS Lambda
Click to enlargeAWS CloudFormation template using Amazon API Gateway and AWS Lambda
Step 1
An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.Step 2
An Amazon Cognito user pool is used to control who can invoke REST API methods.Step 3
Kinesis Data Streams stores the incoming streaming data.Step 4
An AWS Lambda function processes the records from the data stream.Step 5
Errors and failed records that occur during Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS).
The queue stores metadata for failed batch records and Lambda errors, allowing customers to retrieve these records and determine the next steps to resolve them.
Step 1
An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.Step 2
An Amazon Cognito user pool is used to control who can invoke REST API methods.Step 3
Kinesis Data Streams stores the incoming streaming data.Step 4
An AWS Lambda function processes the records from the data stream.Step 5
Errors and failed records that occur during Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS).
The queue stores metadata for failed batch records and Lambda errors, allowing customers to retrieve these records and determine the next steps to resolve them.
About this deploymentPublish Date- Publish Date
Deployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 2
-
AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon CloudWatch
Click to enlargeAWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon CloudWatch
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 2
Kinesis Data Streams stores the incoming data stream.Step 3
Amazon Managed Service for Apache Flink Studio processes the incoming records and saves the processed data in an Amazon Simple Storage Service (Amazon S3) bucket.Step 4
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 2
Kinesis Data Streams stores the incoming data stream.Step 3
Amazon Managed Service for Apache Flink Studio processes the incoming records and saves the processed data in an Amazon Simple Storage Service (Amazon S3) bucket.Step 4
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.About this deploymentPublish Date- Publish Date
Deployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 3
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Data Firehose, and Amazon S3
Click to enlargeAWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Data Firehose, and Amazon S3
Step 1
Kinesis Data Streams stores the incoming streaming data.Step 2
Amazon Data Firehose buffers the data before delivering the output to an S3 bucket. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration.Step 3
A CloudWatch dashboard monitors the data ingestion and buffering. CloudWatch alarms are set on essential metrics for Data Firehose.Step 1
Kinesis Data Streams stores the incoming streaming data.Step 2
Amazon Data Firehose buffers the data before delivering the output to an S3 bucket. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration.Step 3
A CloudWatch dashboard monitors the data ingestion and buffering. CloudWatch alarms are set on essential metrics for Data Firehose.About this deploymentPublish Date- Publish Date
Deployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 4
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon API Gateway
Click to enlargeAWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon API Gateway
Step 1
An EC2 instance uses KPL to generate data.Step 2
Kinesis Data Streams stores the incoming streaming data.Step 3
Managed Service for Apache Flink processes the incoming records and asynchronously invokes an external endpoint.Step 4
The demo application invokes a Lambda function.Step 5
The external API can be any integration supported by API Gateway (for example, an Amazon SageMaker endpoint).Step 6
A CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.Step 1
An EC2 instance uses KPL to generate data.Step 2
Kinesis Data Streams stores the incoming streaming data.Step 3
Managed Service for Apache Flink processes the incoming records and asynchronously invokes an external endpoint.Step 4
The demo application invokes a Lambda function.Step 5
The external API can be any integration supported by API Gateway (for example, an Amazon SageMaker endpoint).Step 6
A CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.About this deploymentPublish Date- Publish Date
Deployment optionsReady to get started?Deploy this solution by launching it in your AWS Console
Related content
This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Managed Service for Apache Flink to asynchronously invoke underlying services or databases.
This is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.
In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.