Category: Amazon S3


Amazon S3 Copy API Ready for Testing

Copying_s3_objects A few weeks ago we asked our developer community for feedback on a proposed Copy feature for Amazon S3. The feedback was both voluminous and helpful to us as we finalized our plans and designed our implementation.

This feature is now available for beta use; you can find full documentation here (be sure to follow the links to the detailed information on the use of this feature via SOAP and REST). Copy requests are billed at the same rate as PUT requests: $.01 for 1000 in the US, and $.012 for 1000 in Europe.

In addition to the obvious use for this feature — creating a new S3 object from an existing one — you can also use it to rename an object within a bucket or to move an object to a new bucket. You can also update the metadata for an object by copying it to itself while supplying new metadata.

Still on the drawing board is support for copying between US and Europe, and a possible conditional copy feature. Both of these items surfaced as a result of developer feedback.

Tool and library support for this new feature is already starting to appear; read more about that in this discussion board thread.

— Jeff;

Your Input Please: Amazon S3 Copy

There are any number of references on the Web to Amazons focus on being the earths most customer centric company; that passion and determination comes straight from Jeff Bezos.

It’s in this traditon that the product team posted a design spec for enhancements to Amazon S3. As Alyssa Henry says in the forum post, we’ve received a significant amount of customer feedback asking that we add support to Amazon S3 for copying an object, as well as related operations such as move and rename. The post points to a design document and asks for community input. (The document is not final–don’t design code around it!)

And the input is coming in! I encourage each of you to read the spec and send us your thoughts on whether the team’s work meets your needs. If not, please take the time to provide suggestions on changes you’d like to see in the spec. We’re serious about making certain our Web Services are the best on earth for you, the customer.

Mike

Amazon S3 Plugs Into a Road Trip

Tim Heuer at Microsoft just emailed me to tell me about The Code Trip — and more particularly S3 Browser for Windows Live Writer, which is a plugin for Windows Live Writer. If you use Windows Live to host your blog, this is a desktop app that makes it easy to create blog posts which include formatted content, photos, and videos, etc. Not only is the plug-in a cool implementation, it’s the first Codeplex contribution from the Code Trip :)

The source code for this plug-in is posted to www.codeplex.com/s3browser if you want to check it out. Aaron Lerch  and Time are the core contibutors behind the code–love the spit and polish on this UI! And that sounds like a seriously cool road trip that Tim is on!

— Mike

Zmanda Webinar

Zmanda At 10 AM on Wednesday, February 13th, Dmitri Joukovski from Zmanda will explain how backup to Amazon S3 complements traditional backup to disk and tape. He will also demonstrate how easy it is to configure Amanda Enterprise for backup and recovery to Amazon S3.

Dmitri will talk about the benefits of using S3 for off-site backup and archiving. He will also list the advantages of using Amanda Enterprise with S3 vs online backup services. The webinar will show how easy it is to configure and backup to S3 using Amanda Enterprise.

The webinar is free but pre-registration is a must.

— Jeff;

Increasing Amazon S3 Data Transfer Performance

The Amazon S3 team is now beta-testing support for an important low-level networking feature which has the potential to significantly increase the performance of large data transfers to and from S3, particularly (but not limited to) for long distance data transfers.

Amazon_window In particular, there is a new beta endpoint which supports the RFC 1323 model for TCP window scaling. With this option in effect, a larger amount of data can be in transit across the network at any given time, reducing the impact of speed-of-light delays as data is broken down in to chunks, sent across the internet, verified, and acknowledged. When the two nodes in question are far apart, the time that it takes for the data to travel from sender to receiver (and for the acknowledgments to travel back) turns out to be just as important as the raw data transfer rate.

Per the thread in the Amazon S3 Forum, early results from the beta testers are quite good with reported speedups of 4x to 18x! If you are moving large amounts of data into or out of S3 then you will definitely want to implement this feature.

Note that we are supporting this through a beta endpoint. The endpoint will be present for the duration of the test and will then vanish. Of course you won’t want to embed this endpoint in any shrink-wrapped software.

You will need to dig deep into your operating system’s networking setting in order to take advantage of this new feature. For Linux systems the information here will get you started (there’s also some information for other operating systems in there too). Windows 2000 and Windows 2003 users can start at this Knowledge Base article. I know nothing about the Mac except that my youngest daughter keeps asking me for one.

I would also recommend that you read this very informative article by Brian Tierney for further background information and some formulas and recommendations on how to calculate and set the optimal window size.

— Jeff;

PS – Did you know that Amazon.com sells actual windows?

Amazon S3 POST Support

S3_post The Amazon S3 team is looking for feedback on their proposal to support browser-based file uploads. This feature will allow developers to build web-based applications which support upload of local files to Amazon S3 without the need for an intermediate server.

If you are interested in helping to shape the definition this feature, you should definitely read the proposal and then visit this thread to let them know what you think.

— Jeff;

Notes From Berlin

Another post from the road In this case the road is a three-week trip to Ireland, Germany, Portugal, and Canada before returning to Seattle. This week Im in Berlin for Web 2.0 Expo and Interop; both are under one roof.

Todays announcement that were launching Amazon S3 in Europe has me really excited, given where Im posting this entry from. You can guess what the #1 request from developers has been on this trip: When will Amazon S3 host data in the EU?. Developers in the United States were also asking for S3 in Europe, because they were looking for lower latency. Of course the requests for hosting in the EU are as old as the service itselfand checking off a top developer request makes the Amazon S3 product team as happy as the customers who asked.

To support Europe we enhanced Amazon S3s API to include a new parameter for locality. The name of the parameter is LocationConstraint, and if you use it then buckets are created in the location you specify. Existing applications will not be affected, and will create buckets in the United States, because thats the default behavior. Pricing is slightly different in Europe; although not but much. Use the AWS Price calculator if you want to estimate what your costs will be.

ISVs Already Support Amazon S3 in Europe
Existing ISVs who built products on top of Amazon S3 have been effusive. For example, Saurabh Dani from Bucket Explorer said “We built Bucket Explorer as a simple and easy to use, yet powerful Interface for S3. The majority of our customers represent three segments: people looking to use Amazon S3 as an online collaboration tool, as a reliable store for off-site backups, or as a content delivery network. And our EU customers from all three segments expressed the need for an EU node of S3 to keep their data confined to the EU. With the integration of Bucket Explorer with S3/EU, we are very excited to be able to address to our customer needs.” The latest version of Bucket Explorer can be downloaded at www.bucketexplorer.com/be-download.html.

Bucket Explorer and a popular Firefox add-in known as S3 Organizer (available at www.rjonna.com) have enhanced their products to enable users to specify whether buckets are created in North America or Europe. Of course youll need to download the latest version in order to take advantage of this new feature.

Real-World S3 Examples
The launch of S3 in Europe is one entry in a long list of milestones. To me, real-world implementations are always the most interesting. One really interesting example that demonstrates the power of scale on demand is in a recent blog post by software developers who work for the New York Times.

The post is about their project to convert old newspaper articles to PDF format for access online (check this out by searching for Titanic iceberg on the front page of the Times, and choose the 1851-1980 archive in the drop-down box). From the blog: [I] determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3. (In fact, it worked so well that we ran it twice, since after we were done we noticed an error in the PDFs.)

Wow

Mike

Cool Real Estate Application Using Amazon S3

The other day I blogged that we’re looking for applications that use Amazon Web Services, and that were developed on the other side of the Atlantic… Had some interesting replies; and want to tell you about one of them.

Metropix is based in England, and is the leading supplier of floor plans to the UK real estate industry. They’ve built a server-based engine that converts 2D floor plans into 3D views of homes and other buildings. Of course, the interior as rendered makes a few assumptions regarding decorating and furnishing, but the point of doing this is to help people understand the space that might be their new home, and offer them a glimpse without having to visit the property.

Max Christian is technical director for MetroPix, and emailed me the following validation of the hard work our product team put into building the Amazon S3 web service platform.

“We’re in production and have around 17,000 users in any one week. As well as basic floor plans, we use S3 to host 3D floor plans and are starting to use it to store Google Earth KMZ files of 3D models of houses for sale. “

“The company started in 2004 and as the customer base grew we responded by adding additional web servers. Although using additional servers to cope with demand was reasonably straightforward, the storage needed for the thousands of 3D Floor Plans and 3D Walkthroughs hosted on each server was also growing rapidly. It became increasingly impractical to keep expanding the storage capacity of the web servers, which was consuming valuable technical resources that could be better spent on product development. In early 2007 we switched to Amazon’s S3 service for storage, shuttling images and video backwards and forwards between S3 and our servers so that only recently-demanded content need be kept locally. This proved so successful, and so reliable, that in March 2007 we went one step further and are now serving floor plans directly from S3 to all major UK real estate portals. As well as vastly improving the scalability of our systems, S3 is already saving us over a thousand pounds a month, as Amazon’s data transfer charges are 5% of what we paid previously and have proved to be just as reliable.”

Max pointed me to a KMZ file for a house that’s for sale in Southern England. After downloading the 12 MB file, Google Earth did its usual zoom in to the location of the house, and then rendered the building. As you can see below, I was able to check and uncheck “layers” to add and remove floors in the building. Really neat way to bring a 2D floor plan to life!


Initially both floors were displayed, showing a 3D rendition of the interior of this home.


By unclicking the upper floor layer, the ground floor was revealed

Sounds like entire towns might be next — evidence that content is still king on the Net.

— Mike

Serving KML, KMZ files from Amazon S3

Map-based Mashups are not new. Overlaying your data on maps are also not new. But serving your Google Earth’s Keyhole Markup Language (KML) and its zipped (KMZ) files right off from Amazon S3 is new and innovative.

Few months ago, I blogged about Microsoft’s MapCruncher tool that generates requisite tiles/files of your map mashed with Microsoft Virtual Earth and publishes them to Amazon S3 using the “Publish to S3” button in the tool. My Lake Roosevelt and Grand Coulee Dam crunched map is available here.

Likewise, some smart developers utilized Google Earth Network Links to publish files on Amazon S3.

Google Earth has the ability to display data imported from outside sources. Users can create KML, an XML syntax used to create data files, dynamically and overlay the information on Google Earth.

Sam Curren,  the creator of ActiveTrails.com, a site that lets you upload GPS data of your hikes to create trail maps on Google Earth, wrote a nice article – Wicked Fast Google Earth Network Links using KML, Zip files, and Amazon S3. In the article, he nicely explains how he used all these technologies together to create a meaningful app that serves files real fast from the cloud. The article is nice read even though you are not a Google Earth user.

Arc2Earth V2 converts and publishes your ArcGIS data to view in Google Earth, Google Maps or Virtual Earth with a click of a button, right from the app. In the article, they mention what files they publish on Amazon S3 and how.

Likewise, Ogle Earth‘s Brian talks about a similar app that uses Amazon S3 for Google Earth’s Network regions.

Of course, my hot favorite is WeoGeo – one-stop marketplace for mapping content. WeoGeo is a killer Ruby on Rails app, built and hosted on Amazon EC2 and uses Amazon S3 to host their KMZ files that you purchase off their website. These high-resolution professional mapping content can be terabytes in size and because Amazon S3 scales as you go, WeoGeo is able to reach to a larger audience while keeping their infrastructure costs down. Do checkout WeoGeo‘s cool, very intuitive, iPod-style interface, if you haven’t already done so.

— Jinesh

More Choices – All backed by Amazon S3

It is always great to have more choices:

Db DigitalBucket – Your Windows Explorer On the Web

DigitalBucket is now Live – really cool windows-explorer-style online file management, sharing and publishing backed by Amazon S3. Users can sign up for an account with 1GB storage and 3GB bandwidth for free.

Greg Hacobian, one of the developers, in his own words:

AWS gave us the opportunity to start a business on top of S3 without worrying about buying Terabytes of storage. Running this kind of business in the past was impossible without raising funds from venture capitals. Now, digitalbucket.net is privately funded and has 2 employees.  It was like dream come true when we were first introduced to S3.  The opportunities are endless with AWS.

What I liked about digitalbucket was its intuitive UI and its Windows-Explorer-Look-and-Feel that has zero-learning curve for any windows users. Its packed with AJAX, optimized by JSON, built on ASP.NET with MS SQL Server to store user metadata and relationships and uses Amazon S3 as primary storage for files. It has all the features that you can imagine – rename, move, copy, share, publish, view videos, slideshows, subscribe to RSS feeds, edit office documents online (aha!), zipping and more. It also has a “Sub Accounts” feature – great for startups and small business firms where employees can share documents in one master “company account”.

File123 File123 – Fax-in, Email-in, Scan-in your files to appear online automatically

Users can simply choose to email their files as attachments to specified email address or choose to fax their documents to 888 number or install a small utility and choose to scan and upload files with one click. All the files then appear automatically in their own online secure storage vault. Of course they can also upload digital files manually, manage and share documents using the online file manager too. It automatically gets indexed for searching. All documents are stored on Amazon S3.

Matt Dusig, one of the co-founders, File123 says:

You can store tax returns, wills, business leases, insurance claims, auto registrations, contracts, pay stubs, bills, receipts and other documents you deem important.  You can also backup all your digital photos or video files.  File123 makes organizing and backing up your life easy.

Users can also sign up for an File123 account with different plans (inlcuding free 1GB storage plan).

— Jinesh