Category: Amazon S3


What Can I Say? Another Amazon S3 Price Reduction!

We’ve reduced the prices for Amazon S3 storage again. As is always the case, the cost to store your existing data will go down. This is markedly different than buying a hard drive at a fixed cost per byte and is just one of the many advantages of using cloud-based storage. You can also count on Amazon S3 to deliver essentially infinite scalability, eleven nines of durability (99.999999999%), and your choice of four distinct geographic locations for data storage.

So, starting November 1, 2010, you’ll see a reduction of up to19% in your overall storage charges on a monthly basis. We’ve created a new pricing tier at the 1 TB level, and we have removed the current 50 – 100 TB tier, thereby extending our volume discounts to more Amazon S3 customers.

The new prices for standard storage in the US Standard, EU – Ireland, and APAC – Singapore regions are as follows:

    Old   New
First 1 TB   $0.150   $0.140 per GB
Next 49 TB   $0.150   $0.125 per GB
Next 50 TB   $0.140   $0.110 per GB
Next 400 TB   $0.130   $0.110 per GB
Next 500 TB   $0.105   $0.095 per GB
Next 4000 TB   $0.080 per GB   $0.080 per GB (no change)
Over 5000 TB   $0.055 per GB   $0.055 per GB (no change)

Reduced Redundancy storage will continue to be priced 1/3 lower than standard storage in all regions.

The full price list can be found on the Amazon S3 page. We’ll continue to work relentlessly to drive our costs down so that we can pass the savings along to you!

We’ve got several more announcements related to S3 coming up in the near future, so stay tuned.

The S3 team is hiring Software Development Engineers, a Technical Program Manager, System Engineers, Administrators, and Product Managers. More information and instructions for applying can be found on the Amazon S3 Jobs page.

— Jeff;

 

Now Available: Host Your Web Site in the Cloud

I am very happy to announce that my first book, Host Your Web Site in the Cloud is now available! Weighing in at over 355 pages, this book is designed to show developers how to build sophisticated AWS applications using PHP and the CloudFusion toolkit.

Here is the table of contents:

  1. Welcome to Cloud Computing.
  2. Amazon Web Services Overview.
  3. Tooling Up.
  4. Storing Data with Amazon S3.
  5. Web Hosting with Amazon EC2.
  6. Building a Scalable Architecture with Amazon SQS.
  7. EC2 Monitoring, Auto Scaling, and Elastic Load Balancing.
  8. Amazon SimpleDB: A Cloud Database.
  9. Amazon Relational Database Service.
  10. Advanced AWS.
  11. Putting It All Together: CloudList.

After an introduction to the concept of cloud computing and a review of each of the Amazon Web Services in the first two chapters, you will set up your development environment in chapter three. Each of the next six chapters focuses on a single service. In addition to a more detailed look at each service, each of these chapters include lots of full-functional code. The final chapter shows you how to use AWS to implement a simple classified advertising system.

Although I am really happy with all of the chapters, I have to say that Chapter 6 is my favorite. In that chapter I show you how to use the Amazon Simple Queue Service to build a scalable multistage image crawling, processing, and rendering pipeline. I build the code step by step, creating a queue, writing the code for a single step, running it, and then turning my attention to the next step. Once I had it all up and running, I opened up five PuTTY windows, ran a stage in each, and watched the work flow through the pipeline with great rapidity. Here’s what the finished pipeline looks like:

I had a really good time writing this book and I hope that you will have an equally good time as you read it and put what you learn to good use in your own AWS applications.

Today (September 21) at 4 PM PT I will be participating in a webinar with the good folks from SitePoint. Sign up now if you would like to attend.

— Jeff;

PS – If you are interested in the writing process and how I stayed focused, disciplined, and organized while I wrote the book, check out this post on my personal blog.

 

Amazon S3: Console Enhancements and Import/Export of 8TB Devices

I’ve got two items that will be of interest to Amazon S3 users: console support for RRS notifications, and AWS Import/Export support for 8TB devices.

Console Support for RRS

A couple of months ago I blogged about the integration between Amazon S3 and Amazon SNS. This integration allows S3 to send a notification to an SNS topic if an object stored with Reduced Redundancy Storage is lost. You can now enable this feature for an S3 bucket by selecting Properties on the bucket’s right-click menu:

Then you select the Notifications tab, enable notifications, and enter the name of an SNS topic:

AWS Import/Export Support for 8 TB Devices

The AWS Import/Export Service now supports devices with capacities of up to 8 Terabytes, reducing the number of devices needed for large data transfers. We’ve been working a lot with the LaCie 4big Quadra:

We are also interested in speaking with users of even larger devices such as the Micronet 10TB Platinum RAID SATA and the Data Robotics Drobo S. If you are interested in using a device that can hold more than 8 TB, please contact us at awsimportexport@amazon.com.

— Jeff;

Amazon S3 and Amazon SNS – Best Friends Forever

We’re starting to wire various AWS services to each other, with interesting and powerful results. Today I’d like to talk to you about a brand new connection between Amazon S3 and the Amazon Simple Notification Service.

When I introduced you to SNS earlier this year I noted that “SNS is also integrated with other AWS services” and said that you could arrange to deliver notifications to an SQS message queue.

We’re now ready to take that integration to a new level. Various parts of AWS will now start to publish messages to an SNS topic to let your application know that a certain type of event has occurred. The first such integration is with Amazon S3, and more specifically, with S3’s new Reduced Redundancy Storage option.

You can now configure any of your S3 buckets to publish a message to an SNS topic of your creation (permissions permitting) when S3 detects that it has lost an object that was stored in the bucket using the RRS option.Your application can subscribe to the topic and (when the event is triggered) respond by regenerating the object and storing it back in S3. The message will include the event, a timestamp, the name of the bucket, the object’s key and version id, and some internal identifiers.

Let’s say that you are using S3 to store an original image and some derived images. You would use the STANDARD storage class for the original image and the REDUCED_REDUNDANCY storage class for the derived images. You would also need to store the information needed to regenerate a derived image from the original image. You could store this in SimpleDB or you could create a naming convention for your S3 object keys and then extract the needed information from the URL.

Consider this image:

It is the original image and would be stored with the STANDARD storage class. Derived images (scaled to a new size in this case) would use a suffix containing the needed information, and would be stored with REDUCED_REDUNDANCY:

A notification would be stored on the faces bucket and routed to a topic such as faces_web_app_errors. Your application need only await events on the topic and respond as follows:

  • Confirm the event is of the expected type (s3:ReducedRedundancyLostObject)
  • Extract the bucket and key name from the event
  • Parse the key name to identify the key of the original object and the transform to be applied
  • Fetch the original object
  • Apply the transform (image scaling in this case)
  • Store the derived object in S3 using the REDUCED_REDUNDACY storage class

Over time, we’ll wire up additional events (for S3 and for other services) to SNS. You can prepare for this now by creating general purpose event handlers in your application, and by keeping your code properly factored so that it is easy to create an object when needed. For the case listed above, I would think about structuring my application so that the only way to create a derived object is in response to an event. I would then generate synthetic “lost” events and use them to materialized the derived objects for the first time.

— Jeff;

AWS Management Console Support for S3 RRS

The AWS Management Console now supports Amazon S3’s Reduced Redundancy Storage. You can view and change the storage class of an S3 object in the object’s Properties pane:

You can also select multiple objects and change the storage class for all of them at the same time.

Finally, you can set the option when you upload one or more objects:

Are you putting RRS to use in your application? I’d like to learn more. Send me an email or leave me a comment.

— Jeff;

Amazon S3 Bucket Policies – Another Way to Protect Your Content

Update August 22, 2011:

We are aware of the existence of a tool that scans S3 looking for buckets that allow anonymous users READ permission. Bucket level READ permissions only allow an user to list the objects within a bucket. However, users with the ability to list could probe into the bucket looking for unprotected content, potentially resulting in undesirable access to content as well as usage charges. We have inspected the permissions of all S3 buckets and have sent an email to the owner of buckets that appear to have excessively permissive access controls granting the READ permission for anonymous users. We have also emailed the owners of all buckets that grant the WRITE or WRITE ACL permission to anonymous users. With WRITE permissions granted to Everyone, an anonymous user can access, modify or delete any object in a bucket.

We strongly encourage you to inspect and, if necessary, restrict the permissions on your buckets and on the objects in each bucket. If you are concerned about the integrity of your objects, you can inspect the modification dates in the buckets. You can also inspect the Server Access Logs for the buckets in question. The easiest way to secure your bucket is by using the AWS Management Console. First select a bucket and click the Properties option within the Actions drop down box. Now select the Permissions tab of the Properties panel. Verify that there is no grant for Everyone or Authenticated Users. If one or both exist, remove the Everyone and Authenticated Users grantees. Your bucket will now be inaccessible to anonymous users.

Earlier this year we launched a popular feature enabling our users host static websites on S3. Bucket level READ permissions for everyone the permission configuration weve warned our users about – is not required for S3 website hosting. However, S3 website hosting does require READ permissions at a per-object level. To verify that your bucket is secured against anonymous operations, use the instructions above. To verify individual objects have anonymous READ, you can use the S3 Console to view the permissions on individual objects and verify that Everyone is granted READ permission. Granting anonymous READ on every object is commonly done using ACLs but can also be done using Bucket Policies.

Users of Amazon S3 have been looking for additional ways to control access to their content. We’ve got something new (and very powerful), and I’ll get to it in a moment. But first, I’d like to review the existing access control mechanisms to make sure that you have enough information to choose the best option for your application.

The two existing access control mechanisms are query string authentication and access control lists or ACLs.

The query string authentication mechanism gives you the ability to create a URL that is valid for a limited amount of time. You simply create a URL that references one of your S3 objects, specify an expiration time for the query, and then compute a signature using your private key.

The Access Control List (ACL) mechanism allows you to selectively grant certain permissions (read, write, read ACL, and write ACL) to a list of grantees. The list of grantees can include the object’s owner, specific AWS account holders, anyone with an AWS account, or to the public at large.

Each of these mechanisms controls access to individual S3 objects.

Today, we are adding support for Bucket Policies. Bucket policies provide access control management for Amazon S3 buckets and for the objects in them using a single unified mechanism. The policies are expressed in our Access Policy Language (introduced last year to regulate access to Amazon SQS queues) and enable centralized management of permissions.

Unlike ACLs which can only be used to add (grant) permissions on individual objects, policies can either add or deny permissions across all (or a subset) of the objects within a single bucket. You can use regular expression operators on Amazon resource names (“arns”) and other values, so that you can control access to groups that begin with a common prefix or end with a given extension such as “.html”.

Policies also introduce new ways to restrict access to resources based on the request. Policies can include references to IP addresses, IP address ranges in CIDR notation, dates, user agents, the HTTP referrer, and transports (http and https).

Finally, with bucket policies we have expanded your ability to control access based on specific S3 operations such as GetObject, GetObjectVersion, DeleteObject, or DeleteBucket

When you put all of this together, you can create policies that give you an incredible amount of access control.

You could set up a bucket policy to do any or all of the following:

  • Allow write access…
  • To a particular S3 bucket…
  • Only from your corporate network…
  • During business hours…
  • From your custom application (as identified by a user agent string).

You can grant one application limited read and write access, but allow another to create and delete buckets as well. You could allow several field offices to store their daily reports in a single bucket, allowing each office to write only to a certain set of names (e.g. “Nevada/*” or “Utah/*” and only from the office’s IP address range).

Policies and ACLs interact in a well-defined way and you can choose to use either one (or both) to control access to your content. You can also convert your existing ACLs to bucket policies if you’d like.

Read more in the new Using Bucket Policies section of the Amazon S3 Developer Guide. We’ll also be holding an Introduction to Bucket Policies webcast on July 13th.

What do you think? How will you use this exciting and powerful new feature?

— Jeff;

Amazon Web Services for Backup and Disaster Recovery

Since the launch of the Amazon Web Services platform in 2006, many companies have decided to offer solutions on top of it. We collect a list of these solution providers, being them System Integrators (SI) or Independent Software Vendors (ISV), on our main AWS website. These Solution Providers offer a vast range of solution for many different use cases.

Backup and Disaster Recovery scenarios are great to highlight the advantages of the Cloud versus traditional solutions: if you need Backup and Disaster Recovery for your business, you can avoid the expensive burden of relying on physical tapes and tape management in favor of cloud-based storage.

Data written on tapes typically goes in a vault somewhere and remains useless until the tape is transported back in case of a disaster. Cost of over- or under-provisioning in this case is very high.
On the Cloud, however, you can get rid of tapes and use Amazon S3, a highly durable storage solution: it provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. All the complexities of scalability, reliability, durability, performance and cost-effectiveness are hidden behind a very simple programming interface.

Amazon S3 is intentionally built with a minimal feature set, and if you have very specific needs, that’s where our Solution Providers can help.
For example, If you want to use Amazon S3 to backup Windows servers, desktops and live applications such as Microsoft Exchange and SQL Server to Amazons highly dependable online storage, you might consider using a solution from one of our Solution Providers, Zmanda.
Their Zmanda Cloud Backup automates the steps needed to backup your data to the cloud, through a GUI based backup configuration and management.
They just announced their third generation Cloud Backup, which fully supports the AWS Asia Pacific Region (check out the pricing list).

Zmanda-backup-where-locations

I’ve seen many customer interested in Backup solutions here in Asia Pacific, and I’m sure that they’ll be interested in the solution offered by Zmanda.

But what happens when large amounts of data need to be transferred, and the internet simply isn’t fast enough to do it in a reasonable amount of time?
AWS Import/Export accelerates moving large amounts of data into and out of AWS, using portable storage devices for transport. AWS transfers your data directly onto and off of storage devices using Amazons high-speed internal network and bypassing the Internet.
For significant data sets, AWS Import/Export is often faster than Internet transfer and more cost effective than upgrading your connectivity.

If you are using Amazon S3 and/or AWS Import/Export for Backup or Disaster Recovery, let us know your story and tell us what do you like the most about these services.
You might also be interested in reading Best Practices for Using Amazon S3.

– Simone Brunozzi (@simon)

AWS Management Console Now Supports Amazon S3

The AWS Management Console now includes support for Amazon S3. You can upload and download individual objects and groups of objects. You can control permissions and metadata, and you can create sub-folders to further organize the contents of your buckets.

Here’s a rundown on the major features…

Your buckets are shown on the left; the objects in the selected bucket are shown on the right, as are any sub-folders:

The bucket menu contains the following actions:

The object menu contains the following actions:

You can copy or cut an object from one bucket or folder and then paste it in to another one. The Make Public command is a one-click way to set an object’s ACL for public-read.

Here’s a close-up view of the bucket list:

You can select multiple files for upload:

You can set file permissions as part of the upload process:

And you can also set metadata for each S3 object. You can chose from a number of common HTTP header names (and suggested values) or you can set custom headers:

The console supports all of the Amazon S3 regions and is ready to be used now.

–Jeff;

AWS Import/Export API

We added full programmability to AWS Import/Export in order to make it even easier for you to move large amounts of data in to or out of Amazon S3.

The new API is very straightforward. You use CreateJob to create a new Import or Export job, and ListJobs to retrieve a list of your jobs. You call GetStatus to find out where the job is in the processing pipeline, UpdateJob to make changes to a job, or CancelJob to cancel it entirely (read the API Reference to learn more).

Our customers are using AWS Import/Export in a number of interesting ways. Here’s a sampling:

Sonian has developed a workflow to make it easy to bring in customer data (Exchange server databases, Outlook PST files, Lotus Notes databases, and so forth) in to the cloud.

The customer copies their database to physical media and Sonian manages the details of the import process. Within days of receipt the information contained in the files is archived in Sonian. The average Sonian job size is now 750 GB and the largest Sonian job to date was 4 TB.

 

Classical Archives imported 14.3 TB of data using AWS Import/Export.

After bringing the data in as encrypted 4 GB chunks, they copied it to a set of Elastic Block Store volumes for direct access. They noted that “The use of large files allowed us to improve transfer speed from the drives, and to save time any money.”

 

Complete Genomics automatically transmits completed customer sequencing and analysis data sets via a fast, dedicated, secure network connection from Complete Genomics offsite storage location to the Amazon Simple Storage Service (S3).

By using Amazon S3, Complete Genomics has access to a highly scalable and reliable data storage infrastructure that permits secure storage and retrieval of any amount of data, at any time, from anywhere on the web. AWS then transfers the anonymized Complete Genomics customer data set onto a hard disk drive, which it ships directly to the customer using Amazons distribution and logistics network.

 

The following third-party tools* are already using the AWS Import/Export API:

CloudBerry S3 Explorer now has the ability to create, list and manage Import/Export jobs. Here are some screen shots:

CloudBuddy Personal also includes complete support for Import/Export. Here are some more screen shots:

— Jeff;

* Cloudberry S3 Explorer and CloudBuddy Personal are third party tools provided by companies unaffiliated with AWS.

Amazon CloudFront: HTTPS Access, Another Edge Location, Price Reduction

We continue to enhance Amazon CloudFront at a rapid pace. Here’s the latest and greatest:

  1. We’ve added a new edge location in New York City. This location will provide even better performance to users requesting your content from New York and the northeastern United States.
  2. We’ve reduced pricing for CloudFront HTTP requests by 25%. The prices now start at $0.0075 per 10,000 requests.
  3. You can now deliver content over an HTTPS connection by replacing the “http:” with “https:” in the links to your CloudFront content.
  4. You can configure any of your CloudFront distributions so that the content must be accessed by an HTTPS connection.

The first three items should be pretty much self-explanatory, so let’s take a look at the fourth…

You can now configure an Amazon CloudFront distribution such that access to the Amazon S3 objects represented by the distribution is limited to HTTPS connections. You can do this to protect your content as it travels from a CloudFront edge location to your client application, or to avoid the dreaded “mixed content” warning issued by many web browsers.

Here’s a simple diagram:

To configure your distribution in this matter you simply set the distribution’s RequiredProtocols attribute to the value “https”. If you do not set this attribute, the contents of the distribution will be accessible via both HTTP and HTTPS. You cannot currently make HTTPS requests via a CNAME.

The following third-party applications provide simple tools to set up your distributions for HTTPS access:

These third-party tools are provided by companies unaffiliated with AWS.

— Jeff;