Amazon Kinesis Data Analytics
Amazon Kinesis Data Analytics is the easiest way to transform and analyze streaming data in real time with Apache Flink. Apache Flink is an open source framework and engine for processing data streams. Amazon Kinesis Data Analytics reduces the complexity of building, managing, and integrating Apache Flink applications with other AWS services.
Amazon Kinesis Data Analytics takes care of everything required to run streaming applications continuously, and scales automatically to match the volume and throughput of your incoming data. With Amazon Kinesis Data Analytics, there are no servers to manage, no minimum fee or setup cost, and you only pay for the resources your streaming applications consume.
Powerful real-time processing
Amazon Kinesis Data Analytics provides built-in functions to filter, aggregate, and transform streaming data for advanced analytics. It processes streaming data with sub-second latencies, enabling you to analyze and respond to incoming data and events in real time.
No servers to manage
Amazon Kinesis Data Analytics is serverless; there are no servers to manage. It runs your streaming applications without requiring you to provision or manage any infrastructure. Amazon Kinesis Data Analytics automatically scales the infrastructure up and down as required to process incoming data.
Pay only for what you use
With Amazon Kinesis Data Analytics, you only pay for the processing resources that your streaming applications use. There are no minimum fees or upfront commitments.
Easy to use
Amazon Kinesis Data Analytics enables you to easily and quickly build queries and sophisticated streaming applications in three simple steps: setup your streaming data sources, write your queries or streaming applications, and setup your destination for processed data.
Build sophisticated streaming applications with Apache Flink
Amazon Kinesis Data Analytics includes open source libraries and runtimes based on Apache Flink that enable you to build an application in hours instead of months using your favorite IDE. The extensible libraries include specialized APIs for different use cases, including stateful stream processing, streaming ETL, and real-time analytics. You can use the libraries to integrate with AWS services like Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon Elasticsearch Service, Amazon S3, Amazon DynamoDB, and more.
Use your favorite language
Amazon Kinesis Data Analytics supports building applications in Java, Scala, and Python. You can use these languages to easily build applications that perform joins, aggregations over time windows, filters, and more. You can extend the open source libraries and include custom libraries from the language of your choice. You can also use SQL to interactively analyze streaming data using the AWS console and send the results to other AWS services.
Build streaming applications with Apache Beam
Amazon Kinesis Data Analytics supports running streaming applications built through Apache Beam’s Java SDK in a serverless Apache Flink environment. Apache Beam is an open-source, unified model for defining streaming and batch data processing applications that can be executed across multiple execution engines. You can easily build Apache Beam streaming applications in Java and run them on Amazon Kinesis Data Analytics and other execution engines.
How it works
You can develop streaming extract-transform-load (ETL) applications with Amazon Kinesis Data Analytics built-in operators to transform, aggregate, and filter streaming data. You can easily deliver your data in seconds to Amazon Kinesis Data Streams, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Elasticsearch Service, Amazon S3, custom integrations, and more using built-in connectors.
Watch how John Deere extracts IoT sensor measurements from agricultural equipment, transforms the data into useful customer information in real time, and loads the transformed data into a data lake.
You can interactively query streaming data using standard SQL, build Apache Flink applications using Java, Python and Scala, and build Apache Beam applications using Java to analyze data streams.
Stateful event processing
You can develop applications that process events from one or more data streams and trigger conditional processing and external actions. You can identify patterns like anomaly detection in your data streams using standard SQL and Apache Flink libraries for complex event processing.
Check out how Zynga processes game events triggered by player actions.
Get started with Amazon Kinesis Data Analytics
Instantly get access to the AWS Free Tier.
Build your streaming application from the Amazon Kinesis Data Analytics console.