Posted On: Nov 9, 2023
AWS Lambda launches auto scaling improvements for Amazon MSK and self-managed Apache Kafka as event sources. With this launch, customers will now experience faster scaling and more consistent throughput when processing Kafka messages with Lambda functions.
The Kafka event source mapping has been a great choice for Lambda customers that need a Kafka consumer that can scale horizontally without impacting other workloads. However, customers using the Kafka event source mapping observed that the event source mapping did not scale up quickly enough, leading to increased latency and elevated OffsetLag. These issues were especially pronounced in workloads where a large number of messages arrive in short bursts. Our recent enhancements address these issues by allowing Kafka event source mappings to reach maximum throughput up to twice as fast. These changes will reduce processing delays across many workloads, with the largest improvements appearing in latency sensitive workloads with large bursts of messages that arrive in irregular intervals.
Kafka event source mapping auto scaling improvements are generally available in all commercial regions where AWS Lambda and Amazon MSK are available except China (Beijing), China (Ningxia) and the AWS GovCloud (US) Regions. For more information on availability, please see the AWS Region table.
These auto scaling improvements are applied to new and existing Kafka event source mappings. See these improvements in action in the AWS Compute Blog. For more details, please see the documentation for the MSK event source mapping and Apache Kafka event source mapping.