Resources & FAQ

Q: What is a streaming data producer?

A: A producer can be thousands of data sources, each generating streaming data continuously and which, typically, submit records simultaneously and in small sizes (kilobytes). Streaming data includes a wide variety of data such as log files generated by customers using mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers. Such data producers generate data using various ingestion patterns to ensure durable and reliable data capture in real-time.

Q: What is streaming data storage?

A: Streaming data must be durably captured by massively scalable storage that is capable of handling high data volume from data producers. One example of streaming storage is Amazon Kinesis Data Streams, a massively scalable and durable real-time data streaming service that enables you to continuously capture data for processing.  

Q: What is a streaming data consumer?

A: Streaming data storage enables the processing of data in real-time by data consumers. Consumers process data sequentially and incrementally, either on a record-by-record basis, or over sliding time windows. The data is used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Information derived from such analyses gives companies visibility into many business and customer activities such as service usage (for metering/billing), server activity, website clicks, and geo-location of devices, people, and physical goods. Emerging situations can be addressed promptly.

Q: What is a streaming data destination?

A: Streaming data is continuously processed by a data consumer. Afterwards, the processed data is sent to one or more destinations. These destinations can serve as a final storage destination or can be used to chain together streaming data use cases.

Q: Does this solution come with sample datasets and code?

A: Yes, a sample data generator is included as part of the template using direct ingestion and Amazon Kinesis Data Analytics. You can also use the Amazon Kinesis Data Generator to create additional sample datasets.

Q: Where can I find the source code for this solution?

A: You can visit our GitHub repository to download the templates and scripts for this solution and to share your customizations with others.

Q: Does this solution use the AWS CDK?

A: Yes, this solution’s AWS CloudFormation templates are built using the AWS Cloud Development Kit (AWS CDK). The AWS CDK is an open source software development framework to model and provision your cloud application resources using familiar programming languages. The AWS CDK speeds up your development process by providing you the expressive power of programming languages for defining infrastructure.

Q: How do I customize the solution?

A: This solution exposes the essential configurations for each AWS resource that is set up during deployment. After creation, you can update additional configurations directly in the AWS CloudFormation template or through the AWS CDK.

Q: Can I deploy this solution in any AWS Region?

A: This solution uses Amazon Kinesis Data Analytics, which is not currently available in all AWS Regions. Therefore, you must launch this solution in an AWS Region where this service is available. For the most current service availability by Region, refer to the AWS Service Region Table.