Process Amazon Kinesis Aggregated Data with AWS Lambda

Ian Meyers is a Solutions Architecture Senior Manager with AWS

Last year, we introduced the Amazon Kinesis Producer Library (KPL) to simplify the development of applications that need to send data to Amazon Kinesis Streams. Many customers use aggregation, which allows you to send multiple records to a single Amazon Kinesis Streams record.  Although the KPL made it easy to aggregate events, it left you with the problem of unpacking this information when you consumed the record from the stream.

We are excited to announce that you can now process this data easily using Amazon Kinesis de-aggregation modules. These modules support Java, Python, and Node.js and allow you to extract user records from a KPL-aggregated stream to AWS Lambda or your multi-lang KCL application.

We’ve also launched version 1.2.0 of the Lambda-based Amazon Kinesis Streams to Amazon Kinesis Firehose Forwarder, which automatically extracts KPL user records from your stream before forwarding for archiving in Amazon S3 or loading to Amazon Redshift.

If you have questions or suggestions, please leave a comment below.

——————————

Related

Implementing Efficient and Reliable Producers with the Amazon Kinesis Producer Library

Looking to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.