Posted On: Nov 16, 2023
AWS Lambda now supports failed-event destinations for Amazon Managed Streaming for Apache Kafka (MSK) and self-managed Apache Kafka event source mappings. Failed-event destinations enable your Lambda function to continue processing events on a topic even when it returns an error.
Previously, if an AWS Lambda function failed to process a Kafka message, the Lambda function would retry the record until the message expired. This approach could cause unnecessary costs and complicate error handling. With failed-event destinations, failing batches of events will be sent to SQS, SNS, or S3 after only a few retries. This ensures Lambda functions don’t stall on a single failing batch and enables the ability to re-drive events later. Kafka messages larger than 6MB will also be sent to the failed-event destination. When SQS or SNS are selected as the destination, Lambda will send metadata including offset information to the destination. When S3 is selected as the destination, Lambda will send the invocation record of the failing batch to the destination.
This feature is available in all AWS Regions where the Amazon MSK or self-managed Apache Kafka event source mappings are available. This feature is off by default and requires no additional charge to use, but you will pay for charges associated with the desired destination (SQS/SNS/S3). To learn more see the Lambda Developer Guide for Amazon MSK or Kafka.