Posted On: Aug 17, 2022
AWS Lambda now supports custom Consumer Group IDs when using Amazon Managed Streaming for Apache Kafka (MSK) or Self-Managed Kafka as an event source. Kafka uses Consumer Group IDs to identify consumer membership and record consumer checkpoints. Using a custom Consumer Group ID is ideal for customers with workloads that require disaster recovery or fail‑over support.
Lambda makes it easy to consume events from Kafka Topics at scale. When Lambda starts consuming from a topic, it presents a Consumer Group ID, this has always been a randomly generated unique value which ensures that Lambda will be identified as a new group consumer to the topic and that processing will start at the specified position (Latest or Trim horizon). Now, with a specified Consumer Group ID, Lambda does not need to be identified as a new consumer group. When Kafka identifies Lambda as an existing consumer group, consuming will instead start from where Kafka recorded the consumer group left off, or from the Trim horizon if the offset is no longer valid. In disaster recovery workflows, customers using Apache MirrorMaker2 can use Lambda with a custom Consumer Group ID to resume processing from a mirrored Kafka cluster.
You can get started with custom Consumer Group IDs for Amazon MSK and Self-Managed Kafka via AWS Management Console, AWS CLI, AWS SAM, or AWS SDK for Lambda. It can be used at no additional cost in all regions where AWS Lambda is available. For more information on where AWS Lambda is available, see the AWS region table. To learn more about building an Apache Kafka consumer application with Lambda, read the Lambda Developer Guide and the AWS Compute Blog.