Posted On: Jun 21, 2023
AWS Lambda now supports starting from a specific timestamp when using Amazon Managed Streaming for Apache Kafka (MSK) or Self-Managed Kafka as an event source. Previously, Kafka event source mappings could only have starting positions of trim horizon or latest. Now with starting from a timestamp, you can start processing messages at a precise point in time. This is useful for situations like Disaster Recovery, where you need a new consumer to quickly start processing where you previously left off.
When a Kafka event source mapping is configured to start from a specific timestamp, the event source mapping will start processing messages in a topic at or the first message after the specified timestamp. To use this feature, create a new Kafka event source mapping, set the StartingPosition to AT_TIMESTAMP, and set StartingPositionTimestamp to the desired starting position. The StartingPositionTimestamp needs to be formatted in Unix time seconds. Please note that in Kafka, the starting position is only used for a new consumer group or when an existing consumer group points to an offset that is invalid (expires). New Kafka event source mappings will generate a new consumer group ID if not otherwise configured with a specific consumer group ID.
This feature incurs no additional charge. You pay for the Lambda invocations triggered by the event source mapping connected to Kafka. To learn more, see the Lambda Developer Guide for Amazon MSK and Apache Kafka.