What does this AWS Solution do?
The Streaming Data Solution for Amazon MSK provides AWS CloudFormation templates where data flows through producers, streaming storage, consumers, and destinations. To support multiple use cases and business needs, this solution offers four AWS CloudFormation templates. Similar to the Streaming Data Solution for Amazon Kinesis, the templates are configured to apply best practices to monitor functionality using dashboards and alarms, and to secure data.
Benefits
AWS Solution overview
-
Option 1
-
Option 2
-
Option 3
-
Option 4
-
Option 1
-
AWS CloudFormation template using Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Click to enlarge
Streaming Data Solution for Amazon MSK architecture
Option 1 – AWS CloudFormation template using Amazon MSK
This AWS CloudFormation template deploys the following architecture:
- An Amazon MSK cluster.
- An Amazon EC2 instance that contains the Apache Kafka client libraries required to communicate with the Amazon MSK cluster. This client machine is located on the same VPC as the cluster, and it can be accessed via AWS Systems Manager Session Manager.
Streaming Data Solution for Amazon MSK
Estimated cost Source code Option 1: AWS CloudFormation template using Amazon MSKVersion 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 25-30 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
-
Option 2
-
AWS CloudFormation template using Amazon MSK and AWS Lambda
Click to enlarge
Streaming Data Solution for Amazon MSK architecture
Option 2 – AWS CloudFormation template using Amazon MSK and AWS Lambda
This AWS CloudFormation template deploys a reference architecture that includes:
- A Lambda function that processes records in an Apache Kafka topic. The default function is a Node.js application that logs the received messages, but it can be customized to meet your business needs.
Streaming Data Solution for Amazon MSK
Estimated cost Source code Option 2: AWS CloudFormation template using Amazon MSK and AWS LambdaVersion 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 5 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
- A Lambda function that processes records in an Apache Kafka topic. The default function is a Node.js application that logs the received messages, but it can be customized to meet your business needs.
-
Option 3
-
AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
Click to enlarge
Streaming Data Solution for Amazon MSK architecture
Option 3 – AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data Firehose
This AWS CloudFormation template deploys a reference architecture that includes the following:
- An AWS Lambda function that processes records in an Apache Kafka topic.
- An Amazon Kinesis Data Firehose delivery stream that buffers data before delivering it to the destination.
- An Amazon Simple Storage Service (Amazon S3) bucket that stores all original events from the Amazon MSK cluster.
Streaming Data Solution for Amazon MSK
Estimated cost Source code Option 3: AWS CloudFormation template using Amazon MSK, AWS Lambda, and Amazon Kinesis Data FirehoseVersion 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
- An AWS Lambda function that processes records in an Apache Kafka topic.
-
Option 4
-
AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
Click to enlarge
Streaming Data Solution for Amazon MSK architecture
Option 4 – AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3
This AWS CloudFormation template deploys a reference architecture that includes the following:
- A Kinesis Data Analytics Studio notebook that reads events from an existing topic in an Amazon MSK cluster.
- An Amazon Simple Storage Service (Amazon S3) bucket that stores the output.
Streaming Data Solution for Amazon MSK
Estimated cost Source code Option 4: AWS CloudFormation template using Amazon MSK, Amazon Kinesis Data Analytics, and Amazon S3Version 1.7.2
Release date: 01/2023
Author: AWS
Estimated deployment time: 10 min
Use the button below to subscribe to solution updates.Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
- A Kinesis Data Analytics Studio notebook that reads events from an existing topic in an Amazon MSK cluster.
Related content
Browse our library of AWS Solutions to get answers to common architectural problems.
Find AWS Partners to help you get started.
Find prescriptive architectural diagrams, sample code, and technical content for common use cases.