We’ve finally reached the conclusion of our deep dive into how you can capture SMTP conversations should you need to debug an issue that lies deeper than your application. Now that we’ve gone over SMTP conversation basics and getting the easiest to decipher bits of a TCP conversation with TCP Flow, let’s look at all the information contained in a TCP conversation using TCP Dump and Wireshark.
Using TCP Dump
TCP Dump is an open source network packet analyzer (licensed under a 3-clause BSD license) which, in conjunction with the libpcap library, can also be used for capturing network traffic. It is one of the most widely used packet analyzers around because it provides a raw level of detail that solutions like TCP Flow don’t provide. It’s essentially a fire hose of data, so it’s sometimes used to capture data that is then read in using Wireshark, which is licensed under GNU GPL v2 and provides you with a great GUI for filtering and analyzing packets.
Amazon EC2 instances running an Amazon Linux AMI come with TCP Dump (tcpdump) pre-installed, so you don’t need to do anything there. The TCP Dump manual is even more intimidating than the TCP Flow manual, so here’s a simple base command you can start and experiment with:
sudo tcpdump -i any -w ~/captures/capture_%Y-%m-%d-%H-%M-%S.cap -G 30 -n -X -Z $USER “port 25”
- The -i option specifies what network interface to listen on, just as in TCP Flow. For most folks, “any” is going to work just fine.
- The -w option writes the raw packets to the file instead of printing to the console, and it’s followed by the file path and format. You can specify the time in plain old strftime format – in this example a file would look like ~/captures/capture_2014-04-30-19-15-00.cap for the time 2014-04-30T19:15:00Z if your machine’s time zone is UTC.
- The -G option is very useful if your application processes large amounts of data – it lets you specify, in seconds, how often the dump file is rotated. In this case, it’ll create a new capture file every 30 seconds (and the file naming will follow what you specified in the -w option).
- The -n option will forego printing FQDNs of host names referenced in the dump. If your application logs print IP addresses instead of host names, this option will make your life much easier.
- The -X option will print each packet in hex and ASCII, which may come in handy in Wireshark.
- The -Z option drops privileges to the user name specified, which means that you’ll own the captures instead of root owning them.
- The end of the command line is, once again, a filtering expression as defined in the pcap filter manual.
A TCP Dump and Wireshark Example: START TLS
Unlike TCP Flow output, your TCP Dump capture file(s) will probably be very hard to read. This is where Wireshark comes in handy. Wireshark actually comes with the command-line tool tshark, which you could use instead of TCP Dump (it’s built on top of TCP Dump), but it doesn’t provide a lot of added value for the general use case. If your own computer is Linux, you should be able to just install Wireshark with yum:
sudo yum -y install wireshark wireshark-gnome
There are Windows and OS X installers available from the Wireshark website, which also has detailed documentation on the suite of features that you can take advantage of. There’s a lot of documentation there, so before you browse that it’s not a bad idea to play with the program a bit to get your feet wet.
Once you have Wireshark installed, transfer your TCP Dump capture from your EC2 instance to your own computer, fire up Wireshark, and open your TCP Dump capture. On Linux, you can simply pass the capture file to Wireshark as a command-line argument (you may or may not need sudo privileges to run it):
sudo wireshark ~/capture_2014-04-16-23-52-29.cap
I recommend immediately going to “View” -> “Time Display Format” and then changing the date format from epoch time (the default) to something more readable. You’ll see a table in the center pane of the GUI that displays one row per packet of data, followed by deeper details of a selected packet, followed by a pane with a hex and ASCII view of the packet. Above all this, you’ll see a blank field that you can fill in with a filter expression, of which Wireshark has an impressive array. You can explore them with the “+ Expression” button (you should see some familiar filters under the “TCP” section from the TCP Flow filter expressions) and then choose one to slice your traffic to just what you’re interested in. There are also some default filter expressions to choose from in “Analyze” -> “Display Filters”. Let’s once again take a look at a STARTTLS conversation with Amazon SES:
There are a few items of note here:
- Everything is color coded as ingress (light blue) or egress (black). You also see grey for the ACK packet. Speaking of which…
- Notice that you can see where the connection was established with the SMTP server – the first three packets. The first packet is the SYN packet from the SMTP client to the SMTP server to open a TCP connection. The second packet is the SYN ACK from the server to the client that it received the SYN packet. The third packet is the ACK from the client to the server that it received the SYN ACK and the connection is established. This is especially useful if you’re trying to determine if there are high latencies during connection establishment or between when the connection is established and receiving the SMTP greeting (the fourth packet) or latencies between all that and when your client starts sending an EHLO, etc.
- In the pane below this packet table, you can select slices of the packet to highlight in the last pane where the packet is displayed.
- Just as in the TCP Flow output, everything under “Ready to start TLS” is unreadable. Again, if all you care about is the timing of packets or if you’re a relay that receives email in plaintext and then transmits the messages to Amazon SES, what we’ve discussed up to now will work just fine for your use case. You can still add logging in your application to print out the decrypted packets.
- The above screenshot shows just one conversation, but depending on your TCP Dump filtering expression, you can capture everything that’s going on to try and detect congestion or interference from other network traffic that your machine is dealing with.
A TCP Dump and Wireshark Example: TLS Wrapper
TLS wrapper mode encrypts everything right from the get-go, so your ability to peek into what’s happening is very limited. There is a way to get Wireshark to tell you which packets are used in the TLS negotiation, though. At a high level, the TLS setup process involves these steps:
- The client and server negotiate security capabilities to determine what to use.
- The server transmits digital certificates and public/private key information to the client so that the client can verify the identity of the server.
- The client exchanges public/private key information with the server (including a pre-master secret) and may send a digital certificate to show that the client is who it says it is (if the server asked for this).
- The server authenticates the client and then the client and server both turn the pre-master secret into master secrets that are used to generate the session key (the same key is generated independently on both sides). This session key is used to encrypt/decrypt communications from here on out.
If you’re deeply curious, you can read more in RFC 2246 (TLS 1.0), RFC 4346 (TLS 1.1), and RFC 5246 (TLS 1.2). Since both the client and server use a public/private key pair as part of this set-up process, you’ll need at least the client’s private key in order for Wireshark to understand the handshake. If you use Open SSL, you can generate and supply this pretty easily:
openssl genrsa -out ~/rsa_key.pem
openssl s_client -crlf -connect email-smtp.us-east-1.amazonaws.com:465 -key ~/rsa_key.pem
The first command line creates the key and the second command line is the one shown in the Amazon SES Developer Guide but with a private key supplied. With this connection set up, you can have your SMTP conversations and use the same TCP Dump command as before, only with port 465:
sudo tcpdump -i any -w ~/captures/capture_%Y-%m-%d-%H-%M-%S.cap -G 30 -n -X -Z $USER “port 465”
Then, just transfer the capture file and rsa_key file to your own computer and fire up Wireshark:
Note that all you see are SYNs and ACKs. We can fix this. Go to “Edit” -> “Preferences”, expand “Protocols” and then select “SSL”. For the above example, if the rsa_key file is in my home directory I’d put the following in “RSA keys list”:
The first element is the server IP address (visible in the Wireshark GUI), the second element is the server port, the third element is the application protocol, and the last element is the location of the private key file. You’ll see something like this:
Now you’ll notice in the “Info” column that you can see more information at a high level and the protocol is specified in the previous column as TCP, SSL, or TLSv1 (whereas before it was just TCP). Additionally, the next pane with the packet breakdown has bits like “Secure Socket Layer” or whatever the protocol is that highlights the part of the packet involved. The “Application Data” rows are just encrypted SMTP messages. The real value though is being able to debug any TLS wrapper issues you may have by comparing good negotiations with bad ones or timestamps in good negotiations versus bad ones and getting to the bottom of whatever is going wrong.
We hope that these posts have given you a better understanding of what’s happening behind the scenes when you interact with Amazon SES, and empowered you to better debug problems that you may experience at the transport or network layer. Thanks for being an Amazon SES customer! Happy debugging!
If your email-sending application has problems communicating with the Amazon SES SMTP interface (or your customers are having problems connecting to your SMTP server that proxies requests to Amazon SES), you’ll first probably check your application logs to see what’s going on. If you’re not able to find a smoking gun in your application logs though, what else can you do? Last week, we went over the basics of SMTP conversations and today we’ll explore how you can debug deeper than the application logs.
You may consider setting up an application layer wire log that shows all of the messages you’re sending and receiving, but one unlucky day you may find yourself with a lower-level issue on your hands. It could be a problem in the link between you and your ISP, between your ISP and the next hop, between your application and your kernel, or any number of other things.
A great way to get more data to help you figure out what’s going on is to go lower in the networking stack to the transport layer. Two well-known, freely available tools that can help you with this are TCP Flow and TCP Dump. TCP Flow is a great next step when you just want to see plaintext data packets in a human-readable format, while TCP Dump is more adept at giving you the kitchen sink so to speak (i.e., all the TCP packets in a variety of formats). In today’s post we’ll talk about TCP Flow. Since many of our customers use EC2 Linux-backed instances, we’ll focus on how to use TCP Flow from Linux.
Installing TCP Flow
TCP Flow lets you get your feet wet in transport layer debugging without overwhelming you with data. You can get the latest version using git clone:
sudo yum -y install git
mkdir ~/tcpflow && cd ~/tcpflow
git clone –recursive git://github.com/simsong/tcpflow.git
Currently, the latest version is 1.3, and the steps in the README work on a standard EC2 with a 64-bit AMI (tested on ami-bba18dd2 and ami-2f726546), though you may also need to yum install openssl-devel. If you encounter any problems doing this you can also try downloading the latest version from the GitHub site, though the install instructions may be in the NEWS file instead of README.
If you can run sudo /usr/local/bin/tcpflow -h and see the usage information, then the install was a success and you’re ready to boogie. Otherwise, double check the console output to see if some step failed. You can get more detailed usage information from man tcpflow.
Using TCP Flow
As you can see in the TCP Flow usage information, there are a lot of options to help you toggle what you’re looking for; these can be overwhelming at first glance. Let’s look at a reasonable set of options to start you off on the right track:
sudo /usr/local/bin/tcpflow -i any -g -FT -c port 25 > ~/tcpflow_out
- The -i option specifies what network interface to listen on (‘any’ is a reasonable default to start you off)
- The -g option was renamed in a recent version (it used to be -J), but it’s just to give you information in different colors, which you’ll soon see is nice to have.
- The -c option prints to the console instead of creating individual files. By default, TCP Flow creates two files for each TCP conversation – one file for the packets coming in and one for the packets being transmitted. The -c option can be a useful alternative because the console interleaves the input and output packets.
- The -F option is all about the format of the output files, and the ‘T’ prepends each file name with an ISO-8601 timestamp. If you output to the console using the -c option, it will still prepend all the lines of your conversation with the timestamp to the millisecond even though you’re not creating any files.
- The “port 25” bit is a filtering expression, as defined in the pcap filter manual. Depending on what your instance is up to, listening to all traffic can be overwhelming so it’s a good idea to filter on what you care about. You can filter on dozens of things including the source or destination host/port, port ranges, and protocol.
Once you have your TCP Flow output, you can look at it with the color coding preserved (there’s one color for packets sent and one for packets received) using less:
less -R ~/tcpflow_out
You can pipe grep, too, if you’re trying to isolate an incident via a specific source/destination port or address:
grep 18.104.22.168 ~/tcpflow_out | less –R
A TCP Flow Example
If you establish a STARTTLS connection with the Amazon SES SMTP endpoint on port 25 and you use the above TCP Flow command, the output from less might look something like this:
You’ll notice that the output is actually readable – there’s a timestamp for each packet in ISO 8601 format followed by the source IP and port of the packet and then the destination IP and port of the packet. You don’t get TCP packet headers or SYN/ACK packets or any of those details, but maybe your problem doesn’t require that much information.
From this point on, however, the conversation will look like gibberish since it’s just a TLS handshake and then all the packets are encrypted. If you use TLS wrapper mode, all the packets will look like gibberish. The nature of TLS makes it tough to decrypt these packets, but TCP Dump and Wireshark will allow us to decrypt at least some of the handshake (we’ll go over these in the next blog post of this series). TCP Flow is still useful on its own, though, if you’re receiving plaintext SMTP conversations from your customers and then proxying messages to Amazon SES for final delivery.
One last thing to note on TCP Flow – you can use the -r option to read in a TCP Dump capture and make it look readable for you.
We hope that you’ve found these tips handy, but the best is yet to come – in the next post of this series we’ll show you how to milk your TCP connections for all the data they’ve got. Thanks again for being a customer!
Amazon SES strives to make your email sending as simple and quick as possible, which means that users of our HTTP API don’t even have to worry about what an SMTP conversation is or how to capture one. Even a lot of our SMTP interface users outsource the problem to software like Microsoft Outlook or PHP Mailer that takes care of these details for them. But if you’re experiencing an issue sending mail that can’t be explained by a recent code change or an error message from SES, understanding how an SMTP conversation works can be helpful. Or maybe you’re just curious as to what those bits flying around look like. In today’s blog post, the first in a series to help you debug these issues, we’ll go over the basics of an SMTP conversation.
The Simple Mail Transfer Protocol (SMTP) was first officially put into writing in 1982 in RFC 821 as a way to “transfer mail reliably and efficiently”, but the protocol that the majority of ISPs use today is described in RFC 5321. The protocol includes a basic handshake, supported commands and responses at each step of the conversation, and finally transmission of message content. Here is a summary of the various steps of a typical conversation, without any extensions involved:
- An SMTP client opens a connection with an SMTP server. Generally, this is on port 25 or 587.
- The SMTP server responds with a 220 code and may follow that with a header that describes the server. It could also respond with a 554 status code to reject the connection, and then the client’s only option would be the QUIT command.
- The SMTP client sends either an EHLO or HELO command to the server. Either command must be followed by a space and then the domain of the client. Contemporary clients and servers should support EHLO, so EHLO is what we’ll use in this example.
- The SMTP server should respond to the EHLO with the 250 status code, its domain name, and a server greeting, and one line for every SMTP extension it supports.
- Now the SMTP client is in business and can start defining the mail to be sent, starting with what’s commonly referred to as the “envelope from” header. The client sends “MAIL FROM:” followed by a reverse-path address, which defines where bounce messages are sent if the message can’t be delivered after it’s accepted (receiving MTAs add this to incoming mail with the Return-Path header). If the mail being sent is a bounce message, this address should be empty (i.e., “<>”). The reverse-path address can optionally be followed by mail parameters defined by a supported SMTP extension (as advertised in the EHLO reply). The conversation cannot proceed until this MAIL FROM command is sent and accepted.
- The SMTP server must accept the MAIL FROM with a “250 OK” reply if the format is good and the address is deemed acceptable. Otherwise, the server typically responds with a 550 or 553 error response to indicate whether the failure is temporary or permanent. Other acceptable error status codes are 552, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions).
- The SMTP client can now define whom the email is for using the RCPT TO command. The syntax is very similar to MAIL FROM: it must be the literal “RCPT TO:” followed by the forward-path address surrounded by angle brackets. This can also optionally be followed by any parameters necessary to use an SMTP extension advertised in the EHLO reply. The RCPT TO command can only define a single recipient. If there are multiple recipients, the command can be issued multiple times, but the client needs to wait for a response from the server each time before supplying another destination.
- The SMTP server usually validates that the address is deliverable and responds with “250 OK” if it is. Otherwise, it typically returns a 550 reply. Other acceptable error status codes are 551, 552, 553, 450, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions). Some servers will accept all mail and only validate the destination after the SMTP conversation has completed.
- Finally, the SMTP client can initiate sending the body of the email by issuing the DATA command with no other text after it.
- The SMTP server responds with a 354 reply if it is ready to accept the message, or else a 503 or 554 if there was no valid MAIL FROM or RCPT TO command sent.
- If the SMTP server responded with a 354 reply, the client submits the message text, followed by the end of mail data indicator, which is a line containing only a period. The message generally starts with headers (one per line, e.g., “Header-name: header-value”) and then is followed by the body of the message.
- If the message is accepted, the SMTP server replies with a “250 OK” reply.
- The client can now initiate a new conversation with the server or send the “QUIT” command to politely close out the connection.
- If the SMTP server receives a QUIT, it is supposed to send a “221 OK” reply and then close the connection.
For deeper details of SMTP, please refer to RFC 5321. You can communicate with SES via SMTP over a number of clients including Microsoft Outlook – see the developer guide for more details. If you’d like to see the conversation yourself, you can also use telnet. There are a few additional things worth noting:
- The server must not intentionally close the connection unless it sees a QUIT command except in the case of a timeout or if the server has to go down.
- At any time during the conversation, the SMTP client can send the RSET command (just “RSET”, no additional parameters) to abort the current mail transaction so that the conversation can start fresh.
- Other supported commands are VRFY to verify that a string represents a valid user or mailbox, EXPN to confirm that a string identifies a mailing list and returns membership of that list, NOOP to get a “250 OK” reply from the server, and HELP to get help information.
- One notable extension that Amazon SES supports for secure communication on ports 25, 587, and 2587 is STARTTLS, as detailed in RFC 3207. After the EHLO, the client sends the command “STARTTLS”, the server replies with “220 Ready to start TLS” (or 501 or 454 error codes), and then TLS negotiation occurs to set up encryption keys. The conversation is then reset and must start with EHLO all over again with all transactions from here on out encrypted. Amazon SES also supports TLS wrapper mode on ports 465 and 2465.
- Another notable extension that Amazon SES requires for authentication is AUTH PLAIN LOGIN, which is where you would enter your SMTP credentials. This is explained in some detail in the Developer’s Guide, but a recent blog post also goes into authentication in general.
Here is a sample SMTP conversation with SES in TLS wrapper mode with the conversation contents decrypted. The blue text comes from Amazon SES and the red text comes from a sample client:
PROXY TCP4 22.214.171.124 10.44.15.76 14659 465
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-793939519 iqAfLvOj6BjiiCjSnD6S
250-AUTH PLAIN LOGIN
235 Authentication successful.
354 End data with <CR><LF>.<CR><LF>
Subject: Test message
From: Senior Tester <email@example.com>
Content-Type: text/html; charset=”UTF-8″
<b>Cool email body</b>
250 Ok 0000012345678e09-123a4cdc-b56c-78dd-b90e-d123be456789-000000
We hope that you feel better informed now on how email works at a high level! In the next post of this series we’ll go over how you can easily capture a live SMTP conversation. Thanks for being a customer of Amazon SES!
Amazon SES provides a native SMTP interface, so that you can use a wide variety of SMTP-enabled programming languages and software packages to send email through Amazon SES. More information is available in the Amazon SES Developer Guide. In this post, we would like to provide you with three recommendations that will improve your SMTP sending.
- If you are sending to Amazon SES from an EC2 instance via port 25, please submit a request to remove email sending limitations. EC2 imposes default sending limits on email sent via port 25 and throttles outbound connections if you exceed those limits. You can also connect to Amazon SES via port 465 and port 587, neither of which is throttled.
- SMTP is a verbose protocol and submitting an email using this protocol requires several network round-trips. Because of the nature of this protocol, the potential of transient network errors increases. Ensure that your application uses retry logic when connecting to the SMTP end point, and that it can detect and retry message delivery in case of a network error. A message is accepted by Amazon SES for delivery only when it responds with an Amazon SES Message ID.
- Do not maintain long-lived connections with the Amazon SES SMTP end point. The Amazon SES SMTP end point runs on a fleet of EC2 instances behind an Elastic Load Balancer (ELB). In order to ensure the system is up-to-date and fault tolerant, active instances are periodically terminated and replaced with new instances. Since your application connects to an EC2 instance through the ELB, the connection becomes invalid when the instance is terminated. Ideally, you should establish a new SMTP connection after you have delivered a fixed number of messages via a single SMTP connection, or if the SMTP connection has been active for some amount of time. You will need to experiment to find appropriate thresholds depending on where your application is hosted and how it submits email to Amazon SES.
If you have more questions about the Amazon SES SMTP end point, please continue to post them in the Amazon SES forum.
This post will help you configure Microsoft SQL Server 2008 R2 to send email through Amazon SES. The instructions apply if you are running SQL Server in Amazon EC2 or hosted elsewhere. This post assumes you are running on Microsoft Windows Server 2008 R2 and SQL Server Management Studio is installed. If you need installation instructions, please refer to the Microsoft SQL Server documentation. The instructions may vary slightly if SQL Server Database Mail profiles or accounts are already set up.
Using Microsoft SQL Server Management Studio
Open Microsoft SQL Server Management Studio and expand the Management node. Right-click Database Mail and select Configure Database Mail in the context menu.
This will launch the Database Mail Configuration Wizard. On the Select Configuration Task wizard page, select Set up Database Mail by performing the following tasks and click Next.
If prompted, enable the Database Mail feature.
On the New Profile wizard page, set Profile name to Amazon SES SMTP.
Now, add a new SMTP account to this profile by clicking the Add… button on the same screen.
- Set Account Name to Amazon SES.
- Set Outgoing SMTP Email to a verified email address associated with your Amazon SES account.
- Set Outgoing SMTP Reply Email to a verified email address associated with your Amazon SES account. This can be the same as the Outgoing SMTP Email.
- Set Server Name to email-smtp.us-east-1.amazonaws.com.
- Set Server Port to 587.
- Set This server requires a secure connection SSL to checked.
- Select Basic Authentication and enter your Amazon SES SMTP username and password. The SMTP credentials are different from your AWS keys.
- Click OK to add this account and associate it with the Amazon SES SMTP profile.
Click Next to get to the Manage Profile Security wizard page,
- Select the Public check box to make Amazon SES SMTP a public profile.
- Select Yes for the Default Profile.
Click Next to get to the System Parameters page. Set Logging Level to Verbose and click Next.
Verify the actions that the wizard will take and click Finish to apply your changes. After the wizard is done and reports success, click Close.
To test your configuration, right click Management » Database Mail » Send Test E-Mail… Enter a recipient address and click Send Test E-Mail.
To verify the email was sent, right click Management » Database Mail » View Database Mail Log. Look for a log entry with Mail successfully sent as the message. This indicates that the email from SQL Server was accepted by Amazon SES for delivery.
We hope that you now have the configuration to use SQL Server 2008 R2 with Amazon SES. If you have comments or feedback about this service, please post them in the Amazon SES forums. We actively monitor the forum and frequently engage with customers. Happy sending with Amazon SES!