Overview
The Streaming Data Solution for Amazon MSK provides AWS CloudFormation templates where data flows through producers, streaming storage, consumers, and destinations. To support multiple use cases and business needs, this solution offers four AWS CloudFormation templates. Similar to the Streaming Data Solution for Amazon Kinesis, the templates are configured to apply best practices to monitor functionality using dashboards and alarms, and to secure data.
Streaming data must be durably captured by massively scalable storage that is capable of handling high data volume from data producers. A producer can be thousands of data sources, each generating streaming data continuously and which, typically, submit records simultaneously and in small sizes (kilobytes).
Streaming data includes a wide variety of data such as log files generated by customers using mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers.
Benefits
Technical details
-
Option 1
-
Option 2
-
Option 3
-
Option 4
-
Option 1
-
AWS CloudFormation template using Amazon Managed Streaming for Apache Kafka (Amazon MSK)
AWS CloudFormation template using Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Option 1 – AWS CloudFormation template using Amazon MSK
Step 1
This AWS CloudFormation template deploys an Amazon MSK cluster.
Option 1 – AWS CloudFormation template using Amazon MSK
Step 1
This AWS CloudFormation template deploys an Amazon MSK cluster.
Related content
AWS Architecture BlogAmazon MSK Backup for Archival, Replay, or AnalyticsRead the blogThis post covers patterns and solutions that can be used to backup MSK topics to S3, which enables customers to reduce long-term data retention settings in MSK. Some customers store long term-data in MSK for data analytics and machine learning workloads. We share a pattern to simplify this architecture by offloading topics data in S3 and use S3 for analytics/ML.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
TrainingAmazon MSK LabsEnroll nowThis site hosts information and hands-on Labs pertaining to Amazon MSK. These labs can either be run on personal or corporate AWS accounts or accounts provisioned by AWS Account teams for events that use Event Engine.
About this deploymentVersion1.7.6Released05/2023AuthorAWSEst. deployment time25-30 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 2
-
AWS CloudFormation template using Amazon MSK and AWS Lambda
AWS CloudFormation template using Amazon MSK and AWS Lambda
Option 2 – AWS CloudFormation template using Amazon MSK and AWS Lambda
Step 1
This AWS CloudFormation template deploys a Lambda function that processes records in an Apache Kafka topic. The default function is a Node.js application that logs the received messages, but it can be customized to meet your business needs.Option 2 – AWS CloudFormation template using Amazon MSK and AWS Lambda
Step 1
This AWS CloudFormation template deploys a Lambda function that processes records in an Apache Kafka topic. The default function is a Node.js application that logs the received messages, but it can be customized to meet your business needs.Related content
AWS Architecture BlogAmazon MSK Backup for Archival, Replay, or AnalyticsRead the blogThis post covers patterns and solutions that can be used to backup MSK topics to S3, which enables customers to reduce long-term data retention settings in MSK. Some customers store long term-data in MSK for data analytics and machine learning workloads. We share a pattern to simplify this architecture by offloading topics data in S3 and use S3 for analytics/ML.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
TrainingAmazon MSK LabsEnroll nowThis site hosts information and hands-on Labs pertaining to Amazon MSK. These labs can either be run on personal or corporate AWS accounts or accounts provisioned by AWS Account teams for events that use Event Engine.
About this deploymentVersion1.7.6Released05/2023AuthorAWSEst. deployment time25-30 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 3
-
AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
Option 3 – AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
Step 1
An AWS Lambda function that processes records in an Apache Kafka topic.Step 2
An Amazon Kinesis Data Firehose delivery stream that buffers data before delivering it to the destination.Step 3
An Amazon Simple Storage Service (Amazon S3) bucket that stores all original events from the Amazon MSK cluster.Option 3 – AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
Step 1
An AWS Lambda function that processes records in an Apache Kafka topic.Step 2
An Amazon Kinesis Data Firehose delivery stream that buffers data before delivering it to the destination.Step 3
An Amazon Simple Storage Service (Amazon S3) bucket that stores all original events from the Amazon MSK cluster.Related content
AWS Architecture BlogAmazon MSK Backup for Archival, Replay, or AnalyticsRead the blogThis post covers patterns and solutions that can be used to backup MSK topics to S3, which enables customers to reduce long-term data retention settings in MSK. Some customers store long term-data in MSK for data analytics and machine learning workloads. We share a pattern to simplify this architecture by offloading topics data in S3 and use S3 for analytics/ML.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
TrainingAmazon MSK LabsEnroll nowThis site hosts information and hands-on Labs pertaining to Amazon MSK. These labs can either be run on personal or corporate AWS accounts or accounts provisioned by AWS Account teams for events that use Event Engine.
About this deploymentVersion1.7.6Released05/2023AuthorAWSEst. deployment time25-30 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 4
-
AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
Option 4 – AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
Step 1
A Kinesis Data Analytics Studio notebook that reads events from an existing topic in an Amazon MSK cluster.Step 2
An Amazon Simple Storage Service (Amazon S3) bucket that stores the output.Option 4 – AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
Step 1
A Kinesis Data Analytics Studio notebook that reads events from an existing topic in an Amazon MSK cluster.Step 2
An Amazon Simple Storage Service (Amazon S3) bucket that stores the output.Related content
AWS Architecture BlogAmazon MSK Backup for Archival, Replay, or AnalyticsRead the blogThis post covers patterns and solutions that can be used to backup MSK topics to S3, which enables customers to reduce long-term data retention settings in MSK. Some customers store long term-data in MSK for data analytics and machine learning workloads. We share a pattern to simplify this architecture by offloading topics data in S3 and use S3 for analytics/ML.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
TrainingAmazon MSK LabsEnroll nowThis site hosts information and hands-on Labs pertaining to Amazon MSK. These labs can either be run on personal or corporate AWS accounts or accounts provisioned by AWS Account teams for events that use Event Engine.
About this deploymentVersion1.7.6Released05/2023AuthorAWSEst. deployment time25-30 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console
