Posted On: Jun 21, 2023
AWS Lambda now supports starting from a specific timestamp when using Amazon Managed Streaming for Apache Kafka (MSK) or Self-Managed Kafka as an event source. Previously, Kafka event source mappings could only have starting positions of trim horizon or latest. Now with starting from a timestamp, you can start processing messages at a precise point in time. This is useful for situations like Disaster Recovery, where you need a new consumer to quickly start processing where you previously left off.
When a Kafka event source mapping is configured to start from a specific timestamp, the event source mapping will start processing messages in a topic at or the first message after the specified timestamp. To use this feature, create a new Kafka event source mapping, set the StartingPosition to AT_TIMESTAMP, and set StartingPositionTimestamp to the desired starting position. The StartingPositionTimestamp needs to be formatted in Unix time seconds. Please note that in Kafka, the starting position is only used for a new consumer group or when an existing consumer group points to an offset that is invalid (expires). New Kafka event source mappings will generate a new consumer group ID if not otherwise configured with a specific consumer group ID.