Increasing Amazon S3 Data Transfer Performance

The Amazon S3 team is now beta-testing support for an important low-level networking feature which has the potential to significantly increase the performance of large data transfers to and from S3, particularly (but not limited to) for long distance data transfers.

In particular, there is a new beta endpoint which supports the RFC 1323 model for TCP window scaling. With this option in effect, a larger amount of data can be in transit across the network at any given time, reducing the impact of speed-of-light delays as data is broken down in to chunks, sent across the internet, verified, and acknowledged. When the two nodes in question are far apart, the time that it takes for the data to travel from sender to receiver (and for the acknowledgments to travel back) turns out to be just as important as the raw data transfer rate.

Per the thread in the Amazon S3 Forum, early results from the beta testers are quite good with reported speedups of 4x to 18x! If you are moving large amounts of data into or out of S3 then you will definitely want to implement this feature.

Note that we are supporting this through a beta endpoint. The endpoint will be present for the duration of the test and will then vanish. Of course you won’t want to embed this endpoint in any shrink-wrapped software.

You will need to dig deep into your operating system’s networking setting in order to take advantage of this new feature. For Linux systems the information here will get you started (there’s also some information for other operating systems in there too). Windows 2000 and Windows 2003 users can start at this Knowledge Base article. I know nothing about the Mac except that my youngest daughter keeps asking me for one.

I would also recommend that you read this very informative article by Brian Tierney for further background information and some formulas and recommendations on how to calculate and set the optimal window size.

— Jeff;

PS – Did you know that Amazon.com sells actual windows?

Modified 2/1/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.