Amazon SES – How to track email deliverability to domain level with CloudWatch

Why is it important to track email deliverability per domain with Amazon Simple Email Service (SES)?

Amazon Simple Email Service (Amazon SES) is a scalable cloud email service provider that enables businesses to build a large-scale email solution and host multiple domains from the same SES account for different purposes ex: one domain for sending marketing emails such as special offers, another domain to send transactional emails such as order confirmations, and other types of correspondence such as newsletters.

As your product, service or solution built on Amazon SES grows and you require multiple domains verified, it is important to track email deliverability for emails you send from each domain for business continuity, billing purposes or incidents investigations. This can be useful to identify if you have low email deliverability for your business domain or if you have a domain generating high bounce or complaint rates and take proactive actions before impacting the account’s ability to send emails from any other domains.

SES offers features that automatically manage deliverability per domain through Virtual Deliverability Manager. Virtual Deliverability Manager helps enhance email deliverability and provides insights into sending and delivery data, as well as offering solutions to fix negative email sending reputation. You can learn more about Virtual Deliverability Manager here.

Solution Walkthrough

Amazon SES provides a way to monitor sender reputation metrics such as bounce and complaint rates per account or configuration sets using event publishing. This blog will discuss how you can use Amazon SES message auto-tags to monitor and publish email deliverability events (Send, Delivery, Bounce, Complaints) to CloudWatch custom metrics per domain. In addition, you will see how to create a custom CloudWatch dashboard that’s easy to access in a single view to monitor your domain metrics. This CloudWatch dashboard can help to provide guidance for your team members during operational events about how to respond to specific incidents for your sending domain.

What are Amazon SES Auto-Tags:

Message tags are a form of name/value pairs to categorize the email you are sending. For example, if you advertise books, you could name a message tag general, and assign a value of sci-fi or western, when you send an email for the associated campaign. Depending on which email sending interface you use, you can provide the message tag as a parameter to the API call (SendEmail, SendRawEmail) or as an Amazon SES-specific email header.

In addition to the message tags you add to any emails you send, Amazon SES adds a set of Auto-Tags that are automatically included in any emails you send. You don’t need to pass the parameters of the auto-tags to the API call or email headers since SES does this automatically.

The auto-tags in the list below are used to track the email deliverability for specific events ( ex: Send, Delivery, Bounce, Complaint). SES does this by using the name/value pairs of the auto-tag name as a dimension in CloudWatch metric to track the count of events of specific auto-tag. This blog post will use “ses:from-domain” auto-tag to configure event publishing for tracking and publish email deliverability events (Send, Delivery, Bounce, Complaints) you receive per domain to CloudWatch metrics and CloudWatch dashboard.

Prerequisites:

For this walkthrough, you should have the following prerequisites:

• An AWS account.
• A verified domain in Amazon SES.
• Administrative Access to Amazon SES Console and Amazon CloudWatch Console.
• Amazon CLI latest version installed and configured.

Configure Amazon SES to publish email deliverability events to CloudWatch destination:

To configure event publishing for tracking email deliverability events, you first need to create a configuration set. Configuration sets in SES are groups of rules, that you can apply to your verified identities. When you apply a configuration set to an email, all of the rules in that configuration set are applied to the email.

After your configuration set is created, you need to create Amazon SES event destination. Amazon SES will send all email deliverability events you intend to track to this event destination. In this blog the event destination is Amazon CloudWatch.

1. 1. Sign in to the Amazon SES console.
   2. In the navigation pane, under Configuration, choose Configuration sets. Choose Create set.
   3. Enter Configuration set name, leave the rest of fields to default, scroll to the send and click on Create set.
   4. Under configuration set home page click on Event destinations tab and select Add destination
   5. 
   6. Under Select event types, check Sends, Deliveries, Hard bounces and Complaints boxes and click Next.
   7. 
   8. Under Specify destination, Select Amazon CloudWatch.
   9. 
   10. Name – enter the name of the destination for this configuration set. The name can include letters, numbers, dashes, and hyphens. (example : Tracking_per_Domain)
   11. Under Amazon CloudWatch dimensions, Select Value source: Message tag , Dimension name: ses:from-domain and Default value: example.com (you will need to add the verified domain name you want to track) as shown below:
   12. 
   13. Review, When you are satisfied that your entries are correct, Click Add destination to add your event destination.

Send a test email via Amazon SES mailbox simulator to trigger events in CloudWatch custom metric.

After selected Amazon CloudWatch as event destination , Amazon CloudWatch will create a custom metric with the auto-tag dimension and value you chose. For this custom metric to appear in CloudWatch Console, you must send an email to trigger each selected event. We recommend using the Amazon SES Mailbox Simulator to avoid generating real bounces or complaints that could impact your account’s reputation.

In the below section, This blog will show how to send those test emails to the following recipients manually using CLI. If you would like to use the console method to send those emails. you will need to send three separate test emails since the console will only allow one recipient per message:

Amazon SES Mailbox Simulator recipients to trigger the events in CloudWatch metrics:
success@simulator.amazonses.com
bounce@simulator.amazonses.com
complaint@simulator.amazonses.com

Note: You must pass the name of the configuration set when sending an email. This can be done by either specifying the configuration set name in the headers of emails, or specifying it as a default configuration set. This can be done at the time of identity creation, or later while editing a verified identity.

The following example uses send-emailCLI command to send a formatted email to the Amazon SES simulator recipients:

Before you run any commands, set your default credentials by following Configuring the AWS CLI. The IAM user must has “ses:SendEmail” permission to send email.

1. Navigate to your terminal where the AWS CLI is installed and configured. Create message.json file for the message to send and add the following content:

2. {
   "Subject": {
   "Data": "Testing CW events with email simulator",
   "Charset": "UTF-8"
   },
   "Body": {
   "Text": {
   "Data": "This is the message body of testing CW events with email similulator.",
   "Charset": "UTF-8"
   }
   }
   }

3. Create a destination.json file to add Amazon SES simulator recipients for bounces, complaints and delivery events as shown below:

4. { 
   
   "ToAddresses": ["complaint@simulator.amazonses.com", "bounce@simulator.amazonses.com" , "success@simulator.amazonses.com"]
   
   }

5. Send a test email using send-email CLI command to send a formatted email to the Amazon SES simulator recipients:

6. aws ses send-email --from test@example.com --destination file://destination.json --message file://message.json --configuration-set-name SES_Config_Set --region <AWS Region>

7. After the message sent, you are expected to see the following output:

8. {
   
   "MessageId": "EXAMPLEf3a5efcd1-51adec81-d2a4-4e3f-9fe2-5d85c1b23783-000000"
   
   }

Now you sent a test email to trigger the events you want to track in CloudWatch custom metrics. Lets create the CloudWatch dashboard to see those metrics.

Create CloudWatch dashboard to track the email deliverability events for my domain.

1. Sign in to the Amazon CloudWatch console.
2. In the navigation pane, choose Dashboards, and then choose Create dashboard.
3. In the Create new dashboard dialog box, enter a name like ‘CW_Domain_Tracking’ for the dashboard, and then choose Create dashboard.
4. In the Add Widget dialog box, Choose Number to add a number displaying a metric to the dashboard and then choose Next
5. Under Add metric graph, click on edit sign to rename the graph with your domain example.com . this will make it easy for you to select the dashboard of the domain if you have multiple domains.
6. In the Browse tab , Select the AWS region where you are running your SES account and in the search bar, search for “ses:from-domain”.
7. You will get four metrics returned with your domain name “example.com”. Select checkbox beside the four metrics and click Create widget.
8. 
9. Save dashboard in the top right corner of the dashboard page to save the widget settings.
10. 

After the CloudWatch dashboard created, for any email you send from example.com domain with configuration set name passed in the email header, The email deliverability events will be counted in your CloudWatch metrics and you will be able to see them in the CloudWatch dashboard.

As an additional step. You can also setup a CloudWatch alarms for this custom metrics and add a threshold for each metric. When the metric breach the threshold, the alarm goes on and send an SNS notification to you to take the necessary actions.

Cleaning Up:

This setup includes Amazon CloudWatch and Amazon SES service charges. To avoid incurring any extra charges, remember to delete any resources created manually if you no longer need them for monitoring.

Resources to delete from Amazon SES console.

1. In the navigation pane, under Configuration, choose Configuration sets.
2. Check the box beside Configuration set you created and select Delete.

Resources to delete from Amazon CloudWatch console.

1. In the navigation pane, choose Dashboards, and then choose the dashboard you created.
2. In the upper-right corner of the graph that you want to remove, choose Actions, and then choose Delete Dashboard.
3. Save dashboard.

Conclusion:

You have now seen how to configure Amazon SES to track email deliverability at domain level with CloudWatch dashboard. Tracking email deliverability for emails you send from each domain is essential for business continuity, billing purposes or incidents investigations. Using SES message auto-tags and CloudWatch metrics you can identify the domains that have low email deliverability quickly and take necessary actions to maximize your email deliverability and take proactive actions before impacting the account’s ability to send emails from any other domains.

About the author:

Alaa Hammad is a Senior Cloud Support Engineer at AWS and subject matter expert in Amazon Simple Email Service and AWS Backup service. She has a 10 years of diverse experience in supporting enterprise customers across different industries. She enjoys cooking and try new recipes from different cuisines.