Posted On: Aug 13, 2020

AWS Lambda now supports Amazon Managed Streaming for Apache Kafka (Amazon MSK) as an event source, giving customers more choices to build serverless applications with streaming data. Customers can build Apache Kafka consumer applications with Lambda functions without needing to worry about infrastructure management. Amazon MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.

Lambda makes it easy to process data streams from Amazon Kinesis Data Streams or Amazon DynamoDB Streams. Now, it's also easy to read from Apache Kafka topics and process batches of records, one batch per partition at a time. The Lambda function is invoked when the batch size is maximized, or the payload exceeds 6MB. Customers can scale concurrency for their applications by increasing the number of partitions within a topic, with a caveat that using multiple partitions may affect ordering of messages.

To get started, select the Amazon MSK topic as the event source for your Lambda function through the AWS Management Console, AWS CLI, AWS SAM, or AWS SDK for Lambda. Amazon MSK as a Lambda event source is available in all AWS Regions where both AWS Lambda and Amazon MSK are available, with the exception of the AWS China Regions and the AWS GovCloud (US) Regions. This feature requires no additional charge to use. You pay for the Lambda invocations triggered by an Apache Kafka topic. To learn more about building an Apache Kafka consumer application with Lambda, read the Lambda Developer Guide.