Storage Space, The Final Frontier

Developers who have found our cloud computing model attractive have been asking us to be a little bit more open about what we are planning to do in the future. To date we’ve simply announced new additions to the Amazon Web Services lineup, with immediate beta availability at the time of announcement.

Earlier this year we started to post specifications for new features along with requests for feedback. We did this for the Amazon S3 Copy feature and for Amazon S3 Post Support . We received a lot of helpful feedback in both cases.

Now it is time for the next step…

I am excited to be able to tell you about an entire new feature, a feature so new that it doesn’t even have a proper name, and that you can’t use just yet. But you can read about it and you can start thinking about the best way to incorporate it into your system architecture.

If you have taken a close look at Amazon EC2, you know that the instances are ephemeral. The instances have anywhere from 160 GB to 1.7 TB of attached storage. The storage is there as long as the instance is running, but of course it disappears as soon as the instance is shut down. Applications with a need for persistent storage could store data in Amazon S3 or in Amazon SimpleDB, but they couldn’t readily access either one as if it was an actual file system.

As you can read in our forum post, we’ve been working on addressing this.

In the same way that your running EC2 instances, your Elastic IP addresses, your S3 buckets and your SQS queues can be thought of as items contained within the scope of your AWS account, our forthcoming persistent storage feature will give you the ability to create reliable, persistent storage volumes for use with EC2. Once created, these volumes will be part of your account and will have a lifetime independent of any particular EC2 instance.

These volumes can be thought of as raw, unformatted disk drives which can be formatted and then used as desired (or even used as raw storage if you’d like). Volumes can range in size from 1 GB on up to 1 TB; you can create and attach several of them to each EC2 instance. They are designed for low latency, high throughput access from Amazon EC2. Needless to say, you can use these volumes to host a relational database.

You will also be able to perform “snapshot” backups of your volumes to Amazon S3. You can use these snapshots to create new volumes or to roll back your stored data to an earlier point in time.

The volumes are accessible via a new set of APIs, with functions like CreateVolume, DeleteVolume, AttachVolume, and CreateSnapshot. The same functionality is also available via the EC2 Command-Line tools.

I spent some time experimenting with this new feature on Saturday. In a matter of minutes I was able to create a pair of 512 GB volumes, attach them to an EC2 instance, create file systems on them with mkfs, and then mount them. When I was done I simply unmounted, detached, and then finally deleted them.

First I created the volumes from the command line of my Windows desktop:

Then I attached them to my EC2 instance:

Then I switched over to my instance, formatted and mounted them, and I was all set:

Perhaps I am biased, but the ability to requisition this much storage on an as-needed basis seems pretty cool.

A few EC2 customers are already using these new volumes and we will be opening it up to a wider audience later this year. You should sign up now if you are interested in gaining access to this cool new feature. If you don’t already have an Amazon Web Services account, get one today before you sign up for the waiting list.

We’ll be releasing more information as soon as possible and I’ll do my best to cover it here when we do.

Updated: Here is some additional coverage:

• Amazon’s Werner  Vogels talks about  this feature in Persistent Storage for Amazon EC2.
• RightScale’s Thorsten vok Eiken does too, in Amazon takes EC2 to the next level with persistent storage volumes.

— Jeff;

Modified 1/25/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.