AWS Compute Blog

Maintaining a Healthy Email Database with AWS Lambda, Amazon SNS, and Amazon DynamoDB

Carlos Sanchiz
Sr. Solutions Architect

Mike Deck
Partner Solutions Architect

Reputation in the email world is critical to achieve reasonable deliverability rates (the percentage of emails that arrive to inboxes); if you fall under certain levels, your emails end up in the spam folder or rejected by the email servers. To keep these numbers high, you have to constantly improve your email quality, but most importantly, you have to take action when a delivery fails or a recipient doesn’t want to receive your email.

Back in 2012, we showed you how to automate the process of handling bounces and complaints with an scheduled task, using Amazon SNS, Amazon SQS, Amazon EC2, and some C# code. We have released many AWS services since then, so this post shows a different approach towards the same goal of a clean and healthy email database.

To set a little bit of context about bounces and complaints processing, I’m reusing some of the previous post:

Amazon SES assigns a unique message ID to each email that you successfully submit to send. When Amazon SES receives a bounce or complaint message from an ISP, we forward the feedback message to you. The format of bounce and complaint messages varies between ISPs, but Amazon SES interprets these messages and, if you choose to set up Amazon SNS topics for them, categorizes them into JSON objects.

Amazon SES will categorize your hard bounces into two types: permanent and transient. A permanent bounce indicates that you should never send to that recipient again. A transient bounce indicates that the recipient’s ISP is not accepting messages for that particular recipient at that time and you can retry delivery in the future. The amount of time you should wait before resending to the address that generated the transient bounce depends on the transient bounce type. Certain transient bounces require manual intervention before the message can be delivered (e.g., message too large or content error). If the bounce type is undetermined, you should manually review the bounce and act accordingly.

A complaint indicates the recipient does not want the email that you sent them. When we receive a complaint, we want to remove the recipient addresses from our list.

In this post, we show you how to use AWS Lambda functions to receive SES notifications from the feedback loop from ISPs email servers via Amazon SNS and update an Amazon DynamoDB table with your email database.

Here is a high-level overview of the architecture:

Using the combination of Lambda, SNS and DynamoDB frees you from the operational overhead of having to run servers and maintain them. You focus on your application logic and AWS handles the undifferentiating heavy lifting behind the operations, scalability, and high availability.

Workflow

  1. Create the SNS topic to receive the SES bounces, deliveries and complaints.
  2. Create the DynamoDB table to use for our email database.
  3. Create the Lambda function to process the bounces, deliveries and complaints and subscribe it to the SNS topic
  4. Test & start emailing!

Create an SNS topic

First, create an SNS topic named “ses-notifications”. You subscribe your Lambda function to the topic later.

Create a DynamoDB table

Create a simple DynamoDB table called “mailing” to store the email database. Use the UserId (email address) as the partition key.

Create the Lambda function

Set up your Lambda function that will process all the notifications coming from SES through your SNS topic.

Note: This post uses Node.js 4.3 as the Lambda runtime but at the time of publication, you can also use Python 2.7, Java 8 or Node.js 0.10.

For the Lambda function code, I used the recently published blueprint (ses-notification-nodejs) and adapted it to work with the DynamoDB table. The following code has the modifications highlighted:

'use strict';
console.log('Loading function');

let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
let tableName = 'mailing';

exports.handler = (event, context, callback) => {
    //console.log('Received event:', JSON.stringify(event, null, 2));
    const message = JSON.parse(event.Records[0].Sns.Message);

    switch(message.notificationType) {
        case "Bounce":
            handleBounce(message);
            break;
        case "Complaint":
            handleComplaint(message);
            break;
        case "Delivery":
            handleDelivery(message);
            break;
        default:
            callback("Unknown notification type: " + message.notificationType);
    }
};

function handleBounce(message) {
    const messageId = message.mail.messageId;
    const addresses = message.bounce.bouncedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });
    const bounceType = message.bounce.bounceType;

    console.log("Message " + messageId + " bounced when sending to " + addresses.join(", ") + ". Bounce type: " + bounceType);

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleComplaint(message) {
    const messageId = message.mail.messageId;
    const addresses = message.complaint.complainedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });

    console.log("A complaint was reported by " + addresses.join(", ") + " for message " + messageId + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleDelivery(message) {
    const messageId = message.mail.messageId;
    const deliveryTimestamp = message.delivery.timestamp;
    const addresses = message.delivery.recipients;

    console.log("Message " + messageId + " was delivered successfully at " + deliveryTimestamp + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "enable");
    }
}

function writeDDB(id, payload, tableName, status) {
    const item = {
            UserId: id,
            notificationType: payload.notificationType,
            from: payload.mail.source,
            timestamp: payload.mail.timestamp,
            state: status
        };
    const params = {
            TableName:tableName,
            Item: item
        };
    dynamo.putItem(params,function(err,data){
            if (err) console.log(err);
            else console.log(data);
    });
}

Assign the function a role with execute and DynamoDB permissions so it can run and update the DynamoDB table accordingly and use index.handler as the function Handler.

This is the lambda_dynamo IAM role policy to use for the three functions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1428341300017",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:UpdateItem"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:dynamodb:us-east-1:ACCOUNT-ID:table/mailing"
        },
        {
            "Sid": "",
            "Resource": "*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow"
        }
    ]
}

You also set the corresponding SNS topic as an event source so that the Lambda function is triggered when notifications arrive.

Test the Lambda function

After everything is in place, it’s time to test. Publish notifications to your SNS topics using Amazon SNS notification examples for Amazon SES, and see how your DynamoDB table is updated by your Lambda functions.

Here’s an example of publishing a complaint notification to the ses-complaints-topic SNS topic using the CLI:

$ aws sns publish --topic-arn "arn:aws:sns:us-east-1:xxxxxxxxxxx:ses-notifications" --message file://message_complaints.txt --subject Test --region us-east-1

{
    "MessageId": "f7f5ad2d-a268-548d-a45c-e28e7624a64d"
}
$ cat message_complaints.txt

{
      "notificationType":"Complaint",
      "complaint":{
         "userAgent":"Comcast Feedback Loop (V0.01)",
         "complainedRecipients":[
            {
               "emailAddress":"recipient1@example.com"
            }
         ],
         "complaintFeedbackType":"abuse",
         "arrivalDate":"2009-12-03T04:24:21.000-05:00",
         "timestamp":"2012-05-25T14:59:38.623-07:00",
         "feedbackId":"000001378603177f-18c07c78-fa81-4a58-9dd1-fedc3cb8f49a-000000"
      },
      "mail":{
         "timestamp":"2012-05-25T14:59:38.623-07:00",
     "messageId":"000001378603177f-7a5433e7-8edb-42ae-af10-f0181f34d6ee-000000",
         "source":"email_1337983178623@amazon.com",
         "sourceArn": "arn:aws:sns:us-east-1:XXXXXXXXXXXX:ses-notifications",
         "sendingAccountId":"XXXXXXXXXXXX",
         "destination":[
            "recipient1@example.com",
            "recipient2@example.com",
            "recipient3@example.com",
            "recipient4@example.com"
         ]
      }
   }

And here is what you’d start seeing coming in your DynamoDB items list:

After you are done with your tests, you can point your SES notifications to the SNS topic you created and start sending emails.

Conclusion

In this post, we showed how you can use AWS Lambda, Amazon SNS, and Amazon DynamoDB to keep a healthy email database, have a good email sending score, and of course, do all of it without servers to maintain or scale.

While you’re at it, why not expose your DynamoDB table with your email database using Amazon API Gateway? For more information, see Using Amazon API Gateway as a proxy for DynamoDB.

If you have questions or suggestions, please comment below.