AWS Storage Update – Amazon S3 Transfer Acceleration + Larger Snowballs in More Regions

Several AWS teams are focused on simplifying and accelerating the process of moving on-premises data to the cloud. We started out with the very basic PUT operation and multipart upload in the early days. Along the way we gave you the ability to send us a disk, and made that process even easier by launching the AWS Import/Export Snowball at last year’s AWS re:Invent (read AWS Import/Export Snowball – Transfer 1 Petabyte Per Week Using Amazon-Owned Storage Appliances to learn more).

Today we are launching some important improvements to Amazon S3 and to Snowball, both designed to further simplify and accelerate your data migration process. Here’s what’s new:

Amazon S3 Transfer Acceleration – This new feature accelerates Amazon S3 data transfers by making use of optimized network protocols and the AWS edge infrastructure. Improvements are typically in the range of 50% to 500% for cross-country transfer of larger objects, but can go ever higher under certain conditions.

Larger Snowballs in More Regions – A new, larger-capacity (80 terabyte) Snowball appliance is now available. In addition, the appliances can now be used with two additional US Regions and two additional international Regions.

Amazon S3 Transfer Acceleration
The AWS edge network has points of presence in more than 50 locations. Today, it is used to distribute content via Amazon CloudFront and to provide rapid responses to DNS queries made to Amazon Route 53. With today’s announcement, the edge network also helps to accelerate data transfers in to and out of Amazon S3. It will be of particular benefit to you if you are transferring data across or between continents, have a fast Internet connection, use large objects, or have a lot of content to upload.

You can think of the edge network as a bridge between your upload point (your desktop or your on-premises data center) and the target bucket. After you enable this feature for a bucket (by checking a checkbox in the AWS Management Console), you simply change the bucket’s endpoint to the form BUCKET_NAME.s3-accelerate.amazonaws.com. No other configuration changes are necessary! After you do this, your TCP connections will be routed to the best AWS edge location based on latency.  Transfer Acceleration will then send your uploads back to S3 over the AWS-managed backbone network using optimized network protocols, persistent connections from edge to origin, fully-open send and receive windows, and so forth.

Here’s how I enable Transfer Acceleration for my bucket (jbarr-public):

My bucket endpoint is https://jbarr-public.s3.amazonaws.com/. I simply change it to https://jbarr-public.s3-accelerate.amazonaws.com/ in my upload tool and/or code. Then I initiate the upload and take advantage of S3 Transfer Acceleration with no further effort. I can use the same rule to construct a URL that can be used for accelerated downloads.

Because network configurations and conditions vary from time to time and from location to location, you pay only for transfers where Transfer Acceleration can potentially improve upload performance. Pricing for accelerated transfers begins at $0.04 per gigabyte uploaded. As is always the case with S3, there’s no up-front fee or long-term commitment.

You can use our new Amazon S3 Transfer Acceleration Speed Comparison to get a better understanding of the value of Transfer Acceleration in your environment. I ran it from my Amazon WorkSpace in the US West (Oregon) Region and this is what I saw:

As you can see, the opportunity for improvement grew in rough proportion to the distance between my home Region and the target.

To learn more about this feature, read Getting Started with Amazon S3 Transfer Acceleration in the Amazon S3 Developer Guide. It is available today in all Regions except China (Beijing) and AWS GovCloud (US).

Larger Snowballs in More Regions
Many AWS customers are now using AWS Snowball to move large amounts of data in and out of the AWS Cloud. For example, here’s a Snowball on-site at GE Oil & Gas:

Ben Wilson (CTO of GE Oil & Gas) posted the picture to LinkedIn with the following caption:

“PIGs and Snowballs – a match made in heaven!! AWS Snowball 25 TB of Pipeline PIG Data to be managed at AWS. That is our GE PIG we pulled some of the data from. It’s always fun to try new AWS features and try to break them!!”

Today we are making Snowball available in four new Regions: AWS GovCloud (US), US West (N. California), Europe (Ireland), and Asia Pacific (Sydney). We expect to make Snowball available in the remaining AWS Regions in the coming year.

The original Snowball appliances had a capacity of 50 terabytes. Today we are launching a newer appliance with 80 terabytes of capacity. If you are transferring data in or out of the US East (N. Virginia), US West (Oregon), US West (N. California), or AWS GovCloud (US) Regions using Snowball you can choose the desired capacity. If you are transferring data in or out of the Europe (Ireland) or Asia Pacific (Sydney) Regions, you will use the 80 terabyte appliance. Here’s how you choose the desired capacity when you are creating your import job:

To learn more about Snowball, read the Snowball FAQ and the Snowball Developer Guide.

— Jeff;