AWS Messaging & Targeting Blog

Introducing Sending Metrics

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

The SES team is pleased to announce sending metrics. You will now be able to segment your sending statistics and publish them to CloudWatch or Kinesis Firehose. We want to help you improve your deliverability, and this new feature makes it easier to pinpoint problems hidden by aggregate account-level statistics.

Why use sending metrics?

With the launch of sending metrics, SES enables you to better segment your send statistics, so that you can see how specific types of emails perform. You will find it easier to diagnose problems quickly before they threaten the reputation of your whole domain.

For example, you may want to see if a specific email campaign is targeting the right people by tracking the bounce and complaint rates separately from your other campaigns and transactional email. Alternatively, you may manage sending for a larger company and may want to get statistics on the email that each team sends to make sure they all uphold the same standards.

For what types of events can I see metrics?

Five types of events are available through this feature:

  • Send – The API call to SES was successful and SES will attempt to deliver the email.
  • Reject – SES rejected the email after the initial send API call was successful. This typically happens when we detect that the email contains a virus.
  • Delivery – The remote server accepted the email from SES. It is still possible that the remote server will send an asynchronous bounce or that the recipient will complain later on.
  • Bounce – The remote server rejected the email. Note that SES automatically retries when getting a “soft” bounce. Only permanent failures to deliver the email will show up in this statistic.
  • Complaint – The recipient marked the incoming email as spam.

For best practices in dealing with bounces and complaints, please see the Amazon SES Best Practices.

How can I see my sending metrics?

SES supports publishing to Kinesis Firehose and CloudWatch.

CloudWatch is best suited for situations where you want aggregate metrics. SES will publish metrics to your account in CloudWatch. Your metrics will be aggregate event counts segmented based on dimensions you specify in your setup in SES. You can see event counts by type and combination of specified dimensions. For example, if you have a dimension called Campaign, you could tag your emails with a message tag called Campaign and see the sending metrics for each individual campaign separately in CloudWatch.

Firehose enables you to see individual events sent by SES published as records to a stream that will save it to S3, feed it into Kinesis Analytics, make it available in an Elasticsearch Service cluster, or push it to Redshift. The event will include information like the type, message tags, time stamp, and message ID associated with the message.

How do I enable sending metrics?

With the release of this feature, SES will begin publishing basic account-level metrics to CloudWatch on your behalf. In order to start getting more detailed metrics, you must first create a configuration set, which will contain event destinations that specify where to publish the events. You can use either the SES console or APIs.

To get started with sending metrics using the console:

  1. Click on the Configuration Sets link from the navigation pane on the left, and then on the Create Configuration Set button. Name your new configuration set.
  2. After creating the configuration set, click on your new configuration to go to a page where you can add event destinations to define where your events should be published. You can select either CloudWatch or Firehose.
  3. Give your new destination a name and select which event types SES will send to this destination. You can send a particular type of event to multiple destinations if you want.
  4. Configure destination-specific info:
  • For CloudWatch, this means adding the dimensions. The values of the message tags or email headers with names that match your dimensions will be associated with that event. (See the Amazon CloudWatch Concepts documentation for more information about how this will work in CloudWatch.) If the tag or header is not present for a given message, SES will use the default value you specify instead.
  • For Firehose, specify the name of the stream and the role SES will assume to publish the events to your Firehose stream. SES will put a record containing a JSON representation of each event to that stream. You can have SES create a role with permissions to publish to Firehose on your behalf, select an existing role, or create the role yourself. SES will verify that it can assume the specified role and use it to publish to the specified stream before allowing you to save your Firehose destination.

When you are finished with the definition, you can start setting the new optional parameter called ConfigurationSetName on the SendEmail and SendRawEmail APIs or passing in the optional X-SES-CONFIGURATION-SET header when using SendRawEmail and the SMTP endpoint. If both the parameter and header are specified, the configuration set from the parameter will take precedence. Emails sent with a configuration set specified will trigger events, which will be published to the event destinations associated with that configuration set.

How do message tags work?

Like headers, message tags are message-specific key-value pairs, but you specify them as a parameter to the SendEmail/SendRawEmail API call. They will be used as value for a CloudWatch dimension or show up in the record for the event published to Firehose.

For example, you could tag your emails with a tag called Type and use values like Transactional, Marketing, Customer_Support, etc. Then, you could add a Type dimension in your CloudWatch event destination and see metrics broken down by each of those email types.

To help you get started, SES will automatically tag all emails with:

  • configuration set name (ses:configuration-set)
  • caller identity (ses:caller-identity)
  • MAIL FROM domain (ses:from-domain)
  • source IP (ses:source-ip)
  • outgoing IP (ses:outgoing-ip)

Additional SES-provided tags may be available in the future. See our documentation for an up to date list of auto-tags.

These auto-tags will be present when appropriate (for example, the source IP will be available when you use the SMTP interface) and you can use them the same way you would use any custom tags you add.

For example, if you manage an AWS account to send email on behalf of multiple users, you could set up a configuration set with a CloudWatch destination that has a ses:caller-identity dimension. You could then add this configuration set as a parameter to all outgoing email sending calls and get a detailed breakdown by the IAM user that was used to send the email for the various event metrics. Alternatively, if you use multiple domains to send emails, you could use the ses:from-domain tag instead to get sending statistics by domain.

How do these events look at the various event destinations?

In CloudWatch, you will find new choices inside the SES namespace based on your event destinations as well as a new Account Metrics option. The Account Metrics option contains basic, account-wide metrics equivalent to the ones currently available in the Sending Statistics section of the SES Console or the GetSendStatistics API. If you choose one of the options based on your configuration sets, you will see a metric per unique combination of tag values, which you can graph or set alarms on.

With Firehose, you can configure your stream to put events to various services.

You can consume the events directly from S3. If you choose this option, you should see files containing one JSON object per line representing an individual event. The file contains an aggregation of events based on a time interval which you can configure in the stream setup. The format looks like:

{"eventType":"Send","mail":{"timestamp":"2016-10-17T18:46:39.808Z","source":"source","sourceArn":"sourceArn","sendingAccountId":"XXXXXXXXXXXX","messageId":"574af616-10fc-4298-abdc-16be2cf77afc","destination":["a@example.com"],"headersTruncated":false,"headers":[{"name":null,"value":null}],"commonHeaders":{"messageId":"574af616-10fc-4298-abdc-16b02cf77afc"},"tags":{"ses:configuration-set":["FirehoseTest"],"ExampleTagName":["ExampleTagValue"]}},"send":{}}
{"eventType":"Bounce","bounce":{"bounceType":"Permanent","bounceSubType":"ContentRejected","bouncedRecipients":[{"emailAddress":"fake.person@fakeDomain.com"}],"timestamp":"2016-10-17T18:46:39.808Z","feedbackId":"63e7a3ba-ad42-4d6a-b736-262562369be4","reportingMTA":"reportingMta.com"},"mail":{"timestamp":"2016-10-17T18:46:39.808Z","source":"source","sourceArn":"sourceArn","sendingAccountId":"XXXXXXXXXXXX","messageId":"71ae6d3f-540b-4310-80ae-5f34d63dcb81","destination":["a@example.com"],"headersTruncated":false,"headers":[{"name":null,"value":null}],"commonHeaders":{"messageId":"71ae6d3f-54cb-4310-80ae-5f34d63dcb81"},"tags":{"ses:configuration-set":["FirehoseTest"],"ExampleTagName":["ExampleTagValue"]}}}

Alternatively, you can have Redshift load the data from your S3 bucket. You can specify a schema that will convert the JSON objects into rows in a relational database that you can query using SQL. For more information on how to set this up, see our Redshift event publishing tutorial.

You can also feed the data into Amazon Elasticsearch Service, which is a managed service that makes it easy to create a domain and deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. Elasticsearch is a popular open-source search and analytics engine. To learn more, see the Amazon Elasticsearch Service documentation.

One example of a tool built on top of Elasticsearch is Kibana, which you can use to visualize data. Here’s a simple example:

 

For more details on how to set up Elasticsearch and Kibana, see our tutorial.

This is not an exhaustive list, and there may be additional options for consuming and analyzing data pushed through Firehose.

How much will this cost me?

Although you can now access basic account-level email sending statistics in CloudWatch for free, normal CloudWatch prices apply for any metrics you collect using configuration sets. You may also incur additional costs by using other AWS services. For example, if you use a Firehose stream to push event data to an S3 bucket, standard S3 charges will apply. SES itself will not charge for this feature beyond current SES pricing.

What if I don’t specify a configuration set on my outgoing emails?

You can still get the usual metrics from the GetSendStatistics API and CloudWatch at the account level, but none of the new, more granular metrics will be available. SES will ignore any tags on an email with no configuration set specified.

We hope that you enjoy this feature! As always, please leave any comments and questions you may have in the SES Forum or here in the comments section of the blog.