AWS Lambda now supports self-managed Apache Kafka as an event source
AWS Lambda now allows customers to build applications that can be triggered by messages in an Apache Kafka cluster hosted on any infrastructure. Customers can keep their existing Apache Kafka cluster as-is, and quickly and easily build Kafka consumer applications with Lambda without needing to worry about provisioning or managing servers.
Today, Lambda supports Amazon Managed Streaming for Kafka (Amazon MSK) as an event source. Now, in addition to Amazon MSK, customers can also build applications triggered from Apache Kafka clusters that they manage, as long as Lambda can access the cluster via a public IP address or an Amazon VPC endpoint. Messages are processed in batches of up to 10,000 records, which may contain messages from one or more partitions. Messages within a partition are always processed in order. Lambda is invoked when the batch size is reached, or the payload size reaches 6MB. Lambda supports consuming messages from unauthenticated and authenticated clusters.
Customers can get started by selecting Apache Kafka as event source for their Lambda function through the AWS Management Console, AWS CLI or AWS SDK for Lambda. This functionality is available in the following AWS Regions: Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), EU (Frankfurt), EU (Ireland), EU (London), EU (Paris), EU (Stockholm), Middle East (Bahrain), South America (Sao Paulo), US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon). This feature requires no additional charge to use. To learn more about building an Apache Kafka consumer application using Lambda, read the Lambda Developer Guide and the AWS Compute Blog.