Q: What is a streaming data producer?

A: A producer can be thousands of data sources, each generating streaming data continuously and which, typically, submit records simultaneously and in small sizes (kilobytes). Streaming data includes a wide variety of data such as log files generated by customers using mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers. Such data producers generate data using various ingestion patterns to ensure durable and reliable data capture in real-time.

Q: What is a streaming data consumer?

A: Streaming data storage allows the processing of data in real-time by data consumers. Consumers process data sequentially and incrementally, either on a record-by-record basis, or over sliding time windows. The data is used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Information derived from such analyses gives companies visibility into many business and customer activities such as service usage (for metering/billing), server activity, website clicks, and geo-location of devices, people, and physical goods. Emerging situations can be addressed promptly.

Q: What is a streaming data destination?

A: Streaming data is continuously processed by a data consumer. Afterwards, the processed data is sent to one or more destinations. These destinations can serve as a final storage destination or can be used to chain together streaming data use cases.

Q: What is Amazon MSK?

A: For more information about Amazon MSK, refer to the Amazon MSK FAQs.


Q: Where can I find the source code for this solution?

A: You can visit our GitHub repository to download the templates and scripts for this solution and to share your customizations with others.

Q: Does this solution use the AWS CDK?

A: Yes, this solution’s AWS CloudFormation templates are built using the AWS Cloud Development Kit (AWS CDK). The AWS CDK is an open source software development framework to model and provision your cloud application resources using familiar programming languages. The AWS CDK speeds up your development process by providing you the expressive power of programming languages for defining infrastructure.

Q: How do I customize the solution?

A: This solution exposes the essential configurations for each AWS resource that is set up during deployment. After creation, you can update additional configurations directly in the AWS CloudFormation template or through the AWS CDK.

Q: Where can I find learning resources for Amazon MSK?

A: The Amazon MSK Labs are a learning resource that take you through getting started,
a use case example of ingesting and analyzing real-time clickstream data, and best
practices for migrating your self-managed Apache Kafka cluster to Amazon MSK.
They also showcase how to leverage advanced Amazon MSK features (such Cruise
Control, TLS mutual authentication, and open monitoring), which can be applied to
clusters created using the solution.

Q: Can I deploy this solution in any AWS Region?

A: This solution uses Amazon Managed Streaming for Apache Kafka (Amazon MSK), which is not currently available in all AWS Regions. Therefore, you must launch this solution in an AWS Region where this service is available. For the most current service availability by Region, refer to the AWS Regional Services List.

Training and Certification

AWS Training and Certification builds your competence, confidence, and credibility through practical cloud skills that help you innovate and build your future.  Learn more »

Data Analytics Fundamentals

In this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.

Enroll now »

AWS Certified Solutions Architect – Associate

This exam validates your ability to effectively demonstrate knowledge of how to architect and deploy secure and robust applications on AWS technologies.

Schedule your exam »

Amazon MSK Labs

This site hosts information and hands-on Labs pertaining to Amazon MSK. These labs can either be run on personal or corporate AWS accounts or accounts provisioned by AWS Account teams for events that use Event Engine.

Launch the workshop »

Partner resources

The AWS Partner Network (APN) is focused on helping partners build successful AWS-based businesses to drive superb solutions and customer experiences. APN Partners are focused on customer success, helping you take full advantage of all the business benefits that AWS has to offer. With their deep expertise on AWS, APN Partners are uniquely positioned to help your company at any stage of your Cloud Adoption Journey and to help you solve some of your most complex problems.

Need more resources to get started with AWS?

Visit the Getting Started Resource Center to find tutorials, projects and videos to get started with AWS.

Learn more »