Using Amazon SQS Dead Letter Queues

After Jason Fulghum recently posted a blog entry about using Amazon SQS dead letter queues with the AWS SDK for Java, I thought his post would be interesting for .NET developers as well. Here is Jason’s post with the code replaced with the C# equivalent.

Amazon SQS recently introduced support for dead letter queues. This feature is an important tool to help your applications consume messages from SQS queues in a more resilient way.

Dead letter queues allow you to set a limit on the number of times a message in a queue is processed. Consider an application that consumes messages from a queue and does some sort of processing based on the message. A bug in your application may only be triggered by certain types of messages or when working with certain data in your application. If your application receives one of these messages, it won’t be able to successfully process it and remove it from the queue. Instead, your application will continue to try to process the message again and again. While this message is being continually retried, your queue is likely filling up with other messages, which your application is unable to process because it’s stuck repeatedly processing the bad message.

Amazon SQS dead letter queues enable you to configure your application so that if it can’t successfully process a problematic message and remove it from the queue, that message will be automatically removed from your queue and delivered to a different SQS queue that you’ve designated as a dead letter queue. Another part of your application can then periodically monitor the dead letter queue and alert you if it contains any messages, which you can debug separately.

Using Amazon SQS dead letter queues is easy. You just need to configure a RedrivePolicy on your queue to specify when messages are delivered to a dead letter queue and to which dead letter queue they should be delivered. You can use the AWS Management Console, or you can access the Amazon SQS API directly with the AWS SDK for .NET.

// First, we'll need an Amazon SQS client object.
IAmazonSQS sqs = new AmazonSQSClient(RegionEndpoint.USWest2);

// Create two new queues:
//     one main queue for our application messages
//     and another to use as our dead letter queue
string qUrl = sqs.CreateQueue(new CreateQueueRequest()
{
    QueueName = "MyApplicationQueue"
}).QueueUrl;

string dlqUrl = sqs.CreateQueue(new CreateQueueRequest()
{
    QueueName = "MyDeadLetterQueue"
}).QueueUrl;

// Next, we need to get the the ARN (Amazon Resource Name) of our dead
// letter queue so we can configure our main queue to deliver messages to it.
IDictionary attributes = sqs.GetQueueAttributes(new GetQueueAttributesRequest()
{
    QueueUrl = dlqUrl,
    AttributeNames = new List() { "QueueArn" }
}).Attributes;

string dlqArn = attributes["QueueArn"];

// The last step is setting a RedrivePolicy on our main queue to configure
// it to deliver messages to our dead letter queue if they haven't been
// successfully processed after five attempts.
string redrivePolicy = string.Format(
    "{{"maxReceiveCount":"{0}", "deadLetterTargetArn":"{1}"}}",
    5, dlqArn);

sqs.SetQueueAttributes(new SetQueueAttributesRequest()
{
    QueueUrl = qUrl,
    Attributes = new Dictionary()
    {
        {"RedrivePolicy", redrivePolicy}
    }
});

There’s also a new operation in the Amazon SQS API to help you identify which of your queues are set up to deliver messages to a specific dead letter queue. If you want to know what queues are sending messages to a dead letter queue, just use the IAmazonSQS.ListDeadLetterSourceQueues operation.

IList sourceQueues = sqs.ListDeadLetterSourceQueues(
    new ListDeadLetterSourceQueuesRequest()
    {
        QueueUrl = dlqUrl
    }).QueueUrls;

Console.WriteLine("Source Queues Delivering to " + qUrl);
foreach (string queueUrl in sourceQueues)
{
    Console.WriteLine(" * " + queueUrl);
}

Dead letter queues are a great way to add more resiliency to your queue-based applications. Have you set up any dead letter queues in Amazon SQS yet?