Why is my Lambda function retrying valid Amazon SQS messages and placing them in my dead-letter queue?
Last updated: 2021-11-01
I configured my AWS Lambda function to process messages in an Amazon Simple Queue Service (Amazon SQS) queue. Now, some of my valid Amazon SQS messages are received multiple times up to the maxReceiveCount and end up in my dead-letter queue. Why is this happening, and how do I make sure that my Lambda function processes all of my valid Amazon SQS messages?
If your Lambda function is throttled, returns an error, or doesn't respond when reading an Amazon SQS message batch, then the messages return to your queue. After the visibility timeout occurs, your Lambda function receives the message batch again. If your function fails to process valid messages multiple times, then Amazon SQS sends the messages to your dead-letter queue, if you've configured one.
To prevent valid messages from being placed in a dead-letter queue, your function code must be idempotent and capable of handling messages multiple times. For more information, see How can I prevent an Amazon SQS message from invoking my Lambda function more than once?
Verify that your Lambda function's code is idempotent
For idempotency best practices and example function logic, see How do I make my Lambda function idempotent?
Verify that your Amazon SQS queue's visibility timeout is at least six times longer than your Lambda function's timeout setting
Set your source queue's visibility timeout to at least six times longer than your function's timeout. The extra time allows your function to retry processing a batch if the function is throttled while processing a previous batch.
For more information, see Setting the visibility timeout in the Amazon SQS Developer Guide.
Note: If your function doesn't receive messages because the queue's visibility timeout isn't long enough, then the messages won't be recorded in your Amazon CloudWatch Logs.
Verify that the maxReceiveCount attribute is set to at least five on your source queue's redrive policy
Set the maxReceiveCount on the source queue's redrive policy to at least five. If your function returns an error, or can't be invoked because it's at maximum concurrency, processing might succeed with additional attempts. A maxReceiveCount of at least five gives your messages more chances to be processed before they're sent to your dead-letter queue.
Identify and resolve any errors that your Lambda function returns
Follow the instructions in How do I troubleshoot Lambda function failures? Your function automatically removes messages from your queue only if the function doesn't return an error.