Posted On: Nov 25, 2019
AWS Lambda now supports four failure-handling features for processing Kinesis and DynamoDB streams: Bisect on Function Error, Maximum Record Age, Maximum Retry Attempts, and Destination on Failure. These new features allow you to customize responses to data processing failures and build more resilient stream processing applications.
Lambda processes data records from Kinesis and DynamoDB streams in batches. Previously when your function returns an error, Lambda stops processing any data in the impacted shard and retries the entire batch of records. These records are continuously retried until they are successfully processed by Lambda or expired by the event source.
Bisect on Function Error
With Bisect on Function Error enabled, Lambda breaks the impacted batch of records into two when a function returns an error, and retries them separately. This allows you to easily separate the malformed data record from the rest of the batch, and process the rest of data records successfully.
Maximum Record Age
Your Lambda function can skip processing a data record when it has reached its Maximum Record Age, which can be configured from 60 seconds to 7 days.
Maximum Retry Attempts
Your Lambda function can skip retrying a batch of records when it has reached the Maximum Retry Attempts, which can be configured from 0 to 10,000.
Destination on Failure
Now your Lambda function can continue processing a shard even when it returns an error. When a data record reaches the Maximum Retry Attempts or Maximum Record Age, you can send its metadata like shard ID and stream ARN to one of these two destinations for further investigation: an SQS queue or SNS topic.
You can get started with these features with AWS Management Console, AWS CLI, AWS CloudFormation, or AWS SDK for Lambda. They can be used at no additional cost in all AWS Regions where AWS Lambda is available. For more information on where AWS Lambda is available, see the AWS Region table. To learn more, see Using AWS Lambda with Amazon Kinesis and Using AWS Lambda with Amazon DynamoDB in the AWS Lambda Developer Guide.