Streaming Data Solution for Amazon Kinesis

A massively scalable and durable real-time data streaming service that allows you to continuously capture data for processing

Overview

The Streaming Data Solution for Amazon Kinesis provides AWS CloudFormation templates where data flows through producers, streaming storage, consumers, and destinations. To support multiple use cases and business needs, the Solution offers four CloudFormation template. Similar to the Streaming Data Solution for Amazon MSK, the templates are configured to apply best practices to monitor functionality using dashboards and alarms and to secure data.

Streaming data must be durably captured by massively scalable storage that is capable of handling high data volume from data producers. A producer can be thousands of data sources, each generating streaming data continuously and which, typically, submit records simultaneously and in small sizes (kilobytes).

Streaming data includes a wide variety of data such as log files generated by customers using mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers.

Benefits

Automated configuration
Automatically configure the AWS services necessary to easily capture, store, process, and deliver streaming data.
Four template options
Choose from four different AWS CloudFormation template options. Test new service combinations for your production environment and improve existing applications.
Real-time use cases
Capture high-volume application logs, analyze clickstream data, continuously deliver to a data lake, and more.
Customizable source code
Customize the Solution's boilerplate code, and then use the monitoring capabilities to quickly transition from testing to production.

Technical details

The diagrams below present the four AWS CloudFormation templates that you can automatically deploy using the solution's implementation guide.
  • Option 1
  • AWS CloudFormation template using Amazon API Gateway and AWS Lambda
    Option 1 – AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda
    AWS Architecture Blog
    Real-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache Flink

    This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Managed Service for Apache Flink to asynchronously invoke underlying services or databases.

    Read the full blog post 
    Training
    Introduction to Amazon Kinesis Analytics

    This is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.

    Enroll now 
    Training
    Data Analytics Fundamentals

    In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.

    Enroll now 
    About this deployment
    Version
    1.8.1
    Released
    10/2023
    Author
    AWS
    Est. deployment time
    5-10 mins
    Estimated cost
    Download implementation guide  Source code  CloudFormation template  Subscribe to RSS feed 
    Deployment options
    Ready to get started?
    Deploy this solution by launching it in your AWS Console
    Did this AWS Solution help you?
    Provide feedback
  • Option 2
  • AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon CloudWatch 

    Option 2 – AWS CloudFormation template using AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon CloudWatch
    AWS Architecture Blog
    Real-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache Flink

    This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Managed Service for Apache Flink to asynchronously invoke underlying services or databases.

    Read the full blog post 
    Training
    Introduction to Amazon Kinesis Analytics

    This is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.

    Enroll now 
    Training
    Data Analytics Fundamentals

    In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.

    Enroll now 
    About this deployment
    Version
    1.8.1
    Released
    10/2023
    Author
    AWS
    Est. deployment time
    5-10 mins
    Estimated cost
    Download implementation guide  Source code  CloudFormation template  Subscribe to RSS feed 
    Deployment options
    Ready to get started?
    Deploy this solution by launching it in your AWS Console
    Did this AWS Solution help you?
    Provide feedback
  • Option 3
  • AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
    Option 3 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
    AWS Architecture Blog
    Real-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache Flink

    This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Managed Service for Apache Flink to asynchronously invoke underlying services or databases.

    Read the full blog post 
    Training
    Introduction to Amazon Kinesis Analytics

    This is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.

    Enroll now 
    Training
    Data Analytics Fundamentals

    In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.

    Enroll now 
    About this deployment
    Version
    1.8.1
    Released
    10/2023
    Author
    AWS
    Est. deployment time
    5-10 mins
    Estimated cost
    Download implementation guide  Source code  CloudFormation template  Subscribe to RSS feed 
    Deployment options
    Ready to get started?
    Deploy this solution by launching it in your AWS Console
    Did this AWS Solution help you?
    Provide feedback
  • Option 4
  • AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon API Gateway
    Option 4 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon API Gateway
    AWS Architecture Blog
    Real-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache Flink

    This post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Managed Service for Apache Flink to asynchronously invoke underlying services or databases.

    Read the full blog post 
    Training
    Introduction to Amazon Kinesis Analytics

    This is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.

    Enroll now 
    Training
    Data Analytics Fundamentals

    In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.

    Enroll now 
    About this deployment
    Version
    1.8.1
    Released
    10/2023
    Author
    AWS
    Est. deployment time
    5-10 mins
    Estimated cost
    Download implementation guide  Source code  CloudFormation template  Subscribe to RSS feed 
    Deployment options
    Ready to get started?
    Deploy this solution by launching it in your AWS Console
    Did this AWS Solution help you?
    Provide feedback