What does this AWS Solutions Implementation do?

The AWS Streaming Data Solution for Amazon Kinesis automatically configures the AWS services necessary to easily capture, store, process, and deliver streaming data. This solution helps you solve for real-time streaming use cases like capturing high volume application logs, analyzing clickstream data, continuously delivering to a data lake, and more. To provide flexibility to adapt to different use cases and business needs, this solution offers two AWS CloudFormation templates. Developers can use this solution to test new service combinations as the basis for their production environment, or to improve an existing application they have already implemented.

This solution shortens your development time by days by removing or reducing the need for you to: model and provision resources using AWS CloudFormation; set up Amazon CloudWatch alarms, dashboards, and logging; and manually implement streaming data best practices in AWS. This solution is data and logic agnostic, enabling you to start with boilerplate code and quickly start customizing. After deployment, you can use the solution’s monitoring capabilities to easily transition to production.

The AWS Streaming Data Solution for Amazon Kinesis uses templates where data flows through producers, streaming storage, consumers, and destinations. Producers continuously generate data and send it to streaming storage where it is durably captured and made available for processing by a data consumer. Data consumers process the data and then send it to a destination.

Currently, this solution provides two templates, but more are in development. The AWS CloudFormation templates use Amazon Kinesis Data Streams for streaming storage. One template uses Amazon API Gateway for ingestion and AWS Lambda as a consumer. The second template uses direct ingestion to Kinesis Data Streams from Amazon Elastic Compute Cloud (Amazon EC2), Amazon Kinesis Data Analytics as a consumer, and Amazon Simple Storage Service (Amazon S3) as a storage destination. Each template enables you to combine different components and extend them to meet your needs.

AWS Solutions Implementation overview

This solution automatically configures the core AWS services necessary to easily capture, store, process, and deliver streaming data. Two options are available, each with customizable demo code. Choose the template that meets your use case and business requirements, then use the implementation guide to deploy.

  • Option 1
  • Option 2

About the AWS CloudFormation templates

For the services you select, the following is provided in each Amazon CloudFormation template created by the AWS Streaming Data Solution for Amazon Kinesis: recommended default configurations for created resources; Amazon CloudWatch logs, metrics, alarms, and dashboards; AWS Identity and Access Management (IAM) roles and managed policies; Amazon CloudTrail setup; and security best practices including encryption using AWS Key Management Service.

Features

Automated configurations

Automatically configure the AWS services necessary to easily capture, store, process, and deliver streaming data.

Flexibility

To provide flexibility to adapt to different use cases and business needs, this solution offers two AWS CloudFormation templates. Developers can use this solution to test new service combinations as the basis for their production environment, or to improve an existing application they have already implemented.

Real-time streaming use cases

Capture high volume application logs, analyze clickstream data, continuously deliver to a data lake, and more.

Data and logic agnostic

Quickly customize the solution’s boilerplate code. After deployment, easily transition to production with the solution’s monitoring capabilities using Amazon CloudWatch.
Build icon
Deploy a Solution yourself

Browse our library of AWS Solutions Implementations to get answers to common architectural problems.

Learn more 
Find an APN partner
Find an APN Partner

Find AWS certified consulting and technology partners to help you get started.

Learn more 
Explore icon
Explore Solutions Consulting Offers

Browse our portfolio of Consulting Offers to get AWS-vetted help with solution deployment.

Learn more