AWS Developer Blog

AWS re:Invent Slides Posted

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

AWS re:Invent was an amazing event this year. We are glad that we could connect with the PHP developers that were there.

If you weren’t there, be sure to watch the Day 1 Keynote with AWS Sr. Vice President Andy Jassy and Day 2 Keynote with Amazon CTO, Werner Vogels. Well known AWS customers like Netflix shared their experiences, and there were a few new services announced, including AWS CloudTrail, Amazon Kinesis, Amazon Workspaces, and Amazon AppStream. AWS CloudTrail support is available in the AWS SDK for PHP as of Version 2.4.10.

We want to let you know that the slides for our presentation, Mastering the AWS SDK for PHP are now available on SlideShare. You can find the slides for all of the AWS re:Invent presentations here.

We’ll share more about AWS re:Invent with you, including the video of our presentation, during the next few weeks.

Efficient Amazon S3 Object Concatenation Using the AWS SDK for Ruby

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Today’s post is from one of our Solutions Architects: Jonathan Desrocher, who coincidentally is also a huge fan of the AWS SDK for Ruby.


There are certain situations where we would like to take a dataset that is spread across numerous Amazon Simple Storage Service (Amazon S3) objects and represent it as a new object that is the concatenation of those S3 objects. A real-life example might be combining individual hourly log files from different servers into a single environment-wide concatenation for easier indexing and archival. Another use case would be concatenating outputs from multiple Elastic MapReduce reducers into a single task summary.

While it is possible to download and re-upload the data to S3 through an EC2 instance, a more efficient approach would be to instruct S3 to make an internal copy using the new copy_part API operation that was introduced into the SDK for Ruby in version 1.10.0.

Why upload when you can copy?

Typically, new S3 objects are created by uploading data from a client using AWS::S3::S3Object#write method or by copying the contents of an existing S3 object using the AWS::S3::Object#copy_to method of the Ruby SDK.

While the copy operation offers the advantage of offloading data transfer from the client to the S3 back-end, it is limited by its ability to only produce new objects with the exact same data as the data specified in the original. This limits the usefulness of the copy operation to those occasions where we want to preserve the data but change the object’s properties (such as key-name or storage class) as S3 objects are immutable.

In our case, we want to offload the heavy lifting of the data transfer to S3’s copy functionality, but at the same time, we need to be able to shuffle different source objects’ contents into a single target derivative—and that brings us to the Multipart Upload functionality.

Copying into a Multipart Upload

Amazon S3 offers a Multipart Upload feature that enables customers to create a new object in parts and then combine those parts into a single, coherent object.

By its own right, Multipart Upload enables us to efficiently upload large amounts of data and/or deal with an unreliable network connection (which is often the case with mobile devices) as the individual upload parts can be retried individually (thus reducing the volume of data retransmissions). Just as importantly, the individual upload parts can be uploaded in parallel, which can greatly increase the aggregated throughput of the upload (note that the same benefits also apply when using byte range GETs).

Multipart Upload can be combined with the copy functionality through the Ruby SDK’s AWS::S3::MultipartUpload#copy_part method—which results in the internal copy of the specified source object into an upload part of the Multipart Upload.

Upon the completion of the Multipart Upload job the different upload parts are combined together such that the last byte of an upload part will be immediately followed by the first byte of the subsequent part (which could be the target of a copy operation itself)— resulting in a true in-order concatenation of the specified source objects.

Code Sample

Note that this example uses Amazon EC2 roles for authenticating to S3. For more information about this feature, see our “credential management” post series.


require 'rubygems'
require 'aws-sdk'

s3 = AWS::S3.new()
mybucket = s3.buckets['my-multipart']

# First, let's start the Multipart Upload
obj_aggregate = mybucket.objects['aggregate'].multipart_upload

# Then we will copy into the Multipart Upload all of the objects in a certain S3 directory.
mybucket.objects.with_prefix('parts/').each do |source_object|

  # Skip the directory object
  unless (source_object.key == 'parts/')
    # Note that this section is thread-safe and could greatly benefit from parallel execution.
    obj_aggregate.copy_part(source_object.bucket.name + '/' + source_object.key)
  end

end

obj_completed = obj_aggregate.complete()

# Generate a signed URL to enable a trusted browser to access the new object without authenticating.
puts obj_completed.url_for(:read)

Last Notes

  • The AWS::S3::MultipartUpload#copy_part method has an optional parameter called :part_number. Omitting this parameter (as in the example above) is thread-safe. However, if multiple processes are participating in the same Multipart Upload (as in different Ruby interpreters on the same machine or different machines altogether), then the part number must be explicitly provided in order to avoid sequence collisions.
  • With the exception of the last part, there is a 5 MB minimum part size.
  • The completed Multipart Upload object is limited to a 5 TB maximum size.
  • It is possible to mix-and-match between upload parts that are copies of existing S3 objects and upload parts that are actually uploaded from the client.
  • For more information on S3 multipart upload and other cool S3 features, see the “STG303 Building scalable applications on S3” session from AWS re:Invent 2012.

Happy concatenating!

The Three Different APIs for Amazon S3

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

The AWS SDK for .NET has three different APIs to work with Amazon S3. The low-level API found in the Amazon.S3 and Amazon.S3.Model namespaces provides complete coverage of the S3 APIs. For easy uploads and downloads, there is TransferUtility, which is found in the Amazon.S3.Transfer namespace. Finally the File I/O API in the Amazon.S3.IO namespace gives the ability to use filesystem semantics with S3.

Low-level API

The low-level API uses the same pattern used for other service low-level APIs in the SDK. There is a client object called AmazonS3Client that implements the IAmazonS3 interface. It contains methods for each of the service operations exposed by S3. Here are examples of performing the basic operations of putting a file in S3 and getting the file back out.

s3Client.PutObject(new PutObjectRequest
{
    BucketName = bucketName,
    FilePath = @"c:datalog.txt"
});

var getResponse = s3Client.GetObject(new GetObjectRequest
{
    BucketName = bucketName,
    Key = "log.txt"
});

getResponse.WriteResponseStreamToFile(@"c:datalog-low-level.txt");

TransferUtility

The TransferUtility runs on top of the low-level API. For putting and getting objects into S3, I would recommend using this API. It is a simple interface for handling the most common uses of S3. The biggest benefit comes with putting objects. For example, TransferUtility detects if a file is large and switches into multipart upload mode. The multipart upload gives the benefit of better performance as the parts can be uploaded simultaneously as well, and if there is an error, only the individual part has to be retried. Here are examples showing the same operations above in the low-level API.

var transferUtility = new TransferUtility(s3Client);

transferUtility.Upload(@"c:datalog.txt", bucketName);

transferUtility.Download(@"c:datalog-transfer.txt", bucketName, "log.txt");

File I/O

The File I/O API is the third API that you’ll find in the Amazon.S3.IO namespace. This API is useful for applications that want to treat S3 as a file system. It does this by mimicking the .NET base classes FileInfo and DirectoryInfo with the new classes S3FileInfo and S3DirectoryInfo. For example, this code shows how similar creating a directory structure in an S3 bucket is to doing so in the local filesystem.

// Create a directory called code at c:code	
DirectoryInfo localRoot = new DirectoryInfo(@"C:");
DirectoryInfo localCode = localRoot.CreateSubdirectory("code");
	
// Create a directory called code in the bucket
S3DirectoryInfo s3Root = new S3DirectoryInfo(s3Client, "bucketofcode");
S3DirectoryInfo codeS3Dir = s3Root.CreateSubdirectory("code");

The following code shows how to get a list of directories and files from the root of the bucket. While going through the enumeration of directories and files all the paging for Amazon S3 calls is handled behind the scenes so there is no need to keep track of a next token.

// Print out the names of the subdirectories under the root directory
foreach (S3DirectoryInfo subDirectory in s3Root.GetDirectories())
{
    Console.WriteLine(subDirectory.Name);
}

// Print the names of the files in the root directory
foreach (S3FileInfo file in s3Root.GetFiles())
{
    Console.WriteLine(file.Name);
}

To write to a file in Amazon S3, you simply open a stream for write from S3FileInfo and write to it. Once the stream is closed, the in-memory data for the stream will be committed to Amazon S3. To read the data back from Amazon S3, just open the stream for read from the S3FileInfo object.

// Write file to Amazon S3
S3DirectoryInfo artDir = s3Root.CreateSubdirectory("asciiart");
S3FileInfo artFile = artDir.GetFile("aws.txt");
using (StreamWriter writer = new StreamWriter(artFile.OpenWrite()))
{
    writer.WriteLine("   _____  __      __  _________");
    writer.WriteLine("  /  _  /      /  /   _____/");
    writer.WriteLine(" /  /_     //   /_____   ");
    writer.WriteLine("/    |            / /        ");
    writer.WriteLine("____|____/__/__/ /_________/");
}	

// Read file back from Amazon S3
using (StreamReader reader = artFile.OpenText())
{
    Console.WriteLine(reader.ReadToEnd());
}

AWS Service Provider for Silex – Version 1.1.0

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the availability of version 1.1.0
of the AWS Service Provider for Silex. This release updates the package’s
dependencies to work with the latest versions of the AWS SDK for PHP and
Silex.

Let us know what you think! Please submit any issues or feature requests to our GitHub issue
tracker
.

High-Level APIs in the AWS SDK for Java

Today, at AWS re:Invent 2013, I’m talking about some of the high-level APIs for Amazon S3 and Amazon DynamoDB, but there are a whole lot more high-level APIs in the SDK that I won’t have time to demo. These high-level APIs are all aimed at specific common tasks that developers face, and each one can save you development time. To help you find all these high-level APIs, we’ve put together the list below. As an added bonus, I’ve thrown in some extra links to some of the more powerful features in the AWS Toolkit for Eclipse.

Take a few minutes to explore the SDK and Eclipse Toolkit features below. Are you already using any of these high-level APIs? What’s your favorite? Let us know in the comments below!

Amazon S3 TransferManager

TransferManager is an easy and efficient way to manage data transfers in and out of Amazon S3. The API is easy to use, provides asynchronous management of your transfers, and has several throughput optimizations.

Amazon S3 Encryption Client

This drop-in replacement for the standard Amazon S3 client gives you control over client-side encryption of your data. The encryption client is easy to use, but also has advanced features like hooks for integrating with existing key management systems.

Amazon DynamoDB Mapper

The DynamoDB Mapper handles marshaling your POJOs into and out of Amazon DynamoDB tables. Just apply a few annotations to your POJOs, and they’re ready to use with the mapper. The mapper also has support for running scans and queries on your data and for batching requests.

S3Link

This new type in the SDK allows you to easily store pointers to data in Amazon S3 inside your POJOs that you’re using with the DynamoDB Mapper. It also makes it easy to perform common operations on the referenced data in Amazon S3, such as replacing the contents, downloading them, or changing access permissions.

Amazon DynamoDB Tables Utility

This class provides common utilities for working with Amazon DynamoDB tables, such as checking if a table exists, and waiting for a new table to transition into an available state.

AWS Flow Framework

AWS Flow is an open-source framework that makes it faster and easier to build apps with Amazon Simple Workflow. The framework handles the interaction with Amazon SWF and keeps your application code simple.

Amazon SES JavaMail Provider

The SDK provides an easy to use JavaMail transport implementation that sends email through the Amazon Simple Email Service.

Amazon SQS Batched Client

This extension of the basic Amazon SQS client provides client-side batching when sending and deleting messages with your Amazon SQS queues. Batching can help reduce the number of round-trip queue requests your application makes and can therefore save you money.

Amazon SNS Topics Utility

This class provides common utilities for working with Amazon SNS topics, such as as subscribing an Amazon SQS queue to an SNS topic to receive published messages.

AWS Policy API

Writing JSON policies by hand can be difficult to maintain, but the Policy API in the AWS SDK for Java gives you an easy way to programmatically create JSON policies for AWS services.

Amazon Glacier ArchiveTransferManager

Glacier’s ArchiveTransferManager makes it easy to get data into and out of Amazon Glacier.

AWS Toolkit for Eclipse

Android Application Development Support

Developing Android applications that use AWS has never been easier. With the AWS Toolkit for Eclipse, you can create new AWS Android projects that have your security credentials configured, Android libraries present, AWS SDK for Android on your build path, and some sample source code to start from.

CloudFormation Support

Lots of new features in the Eclipse Toolkit make working with AWS CloudFormation easy. You can update your CloudFormation stacks directly from Eclipse and use a custom editor to make working with CloudFormation templates easy.

AWS Elastic Beanstalk Deployment

One of the most powerful features of the Eclipse Toolkit is being able to quickly deploy your Java web applications to AWS Elastic Beanstalk directly from within Eclipse. This three-part blog series demonstrates how to get started with AWS Java web projects in Eclipse, how to deploy them to AWS Elastic Beanstalk, and how to manage your applications running in AWS Elastic Beanstalk.

Release: AWS SDK for PHP – Version 2.4.10

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.4.10 of the AWS SDK for PHP. This release adds support for AWS CloudTrail, identity federation using SAML 2.0 for AWS Identity and Access Management (IAM), and a few new features to the Amazon Redshift client.

Changelog

  • Added support for AWS CloudTrail
  • Added support for identity federation using SAML 2.0 to the AWS STS client
  • Added support for configuring SAML-compliant identity providers to the AWS IAM client
  • Added support for HSM storage for encryption keys to the Amazon Redshift client
  • Added support for encryption key rotation to the Amazon Redshift client
  • Added support for database audit logging to the Amazon Redshift client

Install/Download the Latest SDK

AWS re:Invent 2013

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

AWS re:Invent is this week (November 12th-15th) in Las Vegas! We are excited to be here now, and to have an opportunity to talk to you in person.

There is going to be a lot of great technical content year. Loren Segal and I will be presenting a session on Thursday called Diving Into the New AWS SDK for Ruby. We will also be hanging out in the developer lounge area, so come by any time, especially during our Ruby development office hours.

See you there!

Client Side Data Encryption with AWS SDK for .NET and Amazon S3

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

What is client-side encryption, and when would you want to use it?

Version 2 of AWS SDK for .NET provides an easy-to-use Amazon S3 encryption client that allows you to secure your sensitive data before you send it to Amazon S3. Using the AmazonS3EncryptionClient class, the SDK automatically encrypts data on the client when uploading to Amazon S3, and automatically decrypts it when data is retrieved.

EncryptionMaterials encryptionMaterials = new EncryptionMaterials(RSA.Create());
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(encryptionMaterials);
PutObjectResponse putObjectResponse = client.PutObject(putObjectRequest);
GetObjectResponse getObjectResponse = client.GetObject(getObjectRequest);

The entire process of encryption and decryption is called "envelope encryption". AmazonS3EncryptionClient generates a one-time-use AES 256-bit symmetric key (the envelope symmetric key) to encrypt your data, then that key is encrypted by a master encryption key you supply and stored alongside your data in Amazon S3. When accessing your data with the Amazon S3 encryption client, the encrypted symmetric key is retrieved and decrypted with a master encryption key you supply, and then the data is decrypted. Your master encryption key can be a symmetric or asymmetric key.

You can also store your data in Amazon S3 with server-side encryption, but using client-side encryption has some added benefits. First, with server-side encryption, your data is encrypted and decrypted after reaching S3, whereas client-side encryption is performed locally and your data never leaves the execution environment unencrypted.

Another benefit is that client-side encryption allows you to use your own master encryption keys. This ensures that no one can decrypt your data without having access to your master encryption keys.

Encryption metadata storage location

You have the choice to store the encrypted envelope symmetric key either in object metadata or in an instruction file. The instruction file is stored at the same location as that of the object. The following code snippet shows how you can set the storage location.

AmazonS3CryptoConfiguration config = new AmazonS3CryptoConfiguration()
{
    StorageMode = CryptoStorageMode.InstructionFile
};
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(config, encryptionMaterials);

How simple is it to use the AmazonS3EncryptionClient?

The AmazonS3EncryptionClient class implements the same interface as the standard AmazonS3Client, which means it is easy to switch to the AmazonS3EncryptionClient class. In fact, your application code will not be aware of the encryption and decryption happening automatically in the client. All you have to do is create an EncryptionMaterials object that holds an instance of either an asymmetric algorithm (preferably RSA) or a symmetric algorithm. You then simply pass the EncryptionMaterials object to the constructor of AmazonS3EncryptionClient.

The following example shows how you can use AmazonS3EncryptionClient.

EncryptionMaterials encryptionMaterials = new EncryptionMaterials(RSA.Create());

AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(encryptionMaterials);

string bucketName = "YourBucketName";
string keyName = "YourKeyName";
client.PutBucket(new PutBucketRequest { BucketName = bucketName });
PutObjectRequest putObjectRequest = new PutObjectRequest
{
    BucketName = bucketName,
    Key = keyName,
    ContentBody = "Secret Message"
};
client.PutObject(putObjectRequest);
GetObjectRequest getObjectRequest = new GetObjectRequest
{
    BucketName = bucketName,
    Key = keyName
};
GetObjectResponse getObjectResponse = client.GetObject(getObjectRequest);
using (Stream decryptedStream = getObjectResponse.ResponseStream)
{
    using (StreamReader reader = new StreamReader(decryptedStream))
    {
        string decryptedContent = reader.ReadToEnd();
        Console.WriteLine("Decrypted data: {0}", decryptedContent);
    }
}

The AWS SDK for .NET supports client-side encryption for MultiPartUpload and TransferUtility as well, but since we use Cipher Block Chaining mode, TransferUtility uploads the parts sequentially rather than in parallel. Note that this means encrypted multi-part uploads cannot take advantage of multi-threading.

What happens if your master encryption keys are lost?

If your master encryption keys are lost, you will not be able to decrypt your data. Your master encryption keys are never sent to AWS; hence, it is important that you safely store them (e.g., as a file or using a separate key management system) and load them when needed for uploading or downloading objects.

The following example shows how you can use a master encryption key with an asymmetric algorithm.

Create an instance of an RSA algorithm and save the private key in a file.

RSA rsaAlgorithm = RSA.Create();
string privateKey = rsaAlgorithm.ToXmlString(true);
string filePath = @"c:tempPrivateKey.txt";
File.WriteAllText(filePath, privateKey);
EncryptionMaterials materials = new EncryptionMaterials(rsaAlgorithm);
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(materials);
// Perform your operations, such as PutObject, GetObject, etc.

Create an instance of an RSA algorithm and load it with the saved private key.

string filePath = @"c:tempPrivateKey.txt";
string privateKey = File.ReadAllText(filePath);
RSA rsaAlgorithm = RSA.Create();
rsaAlgorithm.FromXmlString(privateKey);
EncryptionMaterials materials = new EncryptionMaterials(rsaAlgorithm);
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(materials);
// Perform your operations, such as PutObject, GetObject, etc.

The following example shows how you can use a master encryption key with a symmetric algorithm.

Create an instance of an AES algorithm and save the symmetric key in a file.

Aes aesAlgorithm = Aes.Create();
File.WriteAllBytes(@"c:tempSymmetricKey.txt", aesAlgorithm.Key);
EncryptionMaterials materials = new EncryptionMaterials(aesAlgorithm);
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(materials);
//Perform your operations, such as PutObject, GetObject, etc.

Create an instance of an AES algorithm and load it with the saved SymmetricKey key.

Aes aesAlgorithm = Aes.Create();
aesAlgorithm.Key = File.ReadAllBytes(@"c:tempSymmetricKey.txt");
EncryptionMaterials materials = new EncryptionMaterials(aesAlgorithm);
AmazonS3EncryptionClient client = new AmazonS3EncryptionClient(materials);
//Perform your operations, such as PutObject, GetObject, etc.

The AmazonS3EncryptionClient class in the AWS SDK for .NET is fully compatible with the AmazonS3EncryptionClient class in the AWS SDK for Java and AWS SDK for Ruby. All you have to do is, using one SDK, store your private encryption keys in a commonly-accessible location (for example, a .pem file) and then load them in the second SDK.

GA Release of AWS SDK for .NET Version 2

by Wade Matveyenko | on | in .NET | Permalink | Comments |  Share

We are excited to announce the General Availability (GA) release of AWS SDK for .NET version 2! This is the next major release of the SDK, which adds support for Windows Store, Windows Phone, and .NET Framework 4.5 platforms. You can download it here.

Improvements

  • One of the most exciting new features of version 2 is the ability to have Windows Store and Windows Phone 8 Apps use our SDK. Like other SDKs for these new platforms, all method calls that make requests to AWS are asynchronous methods.
  • Another big improvement we made to the SDK for asynchronous programming is that when you target Windows Store, Windows Phone 8, or .NET 4.5, the SDK uses the new Task-based pattern for asynchronous programming instead of the IAsyncResult pattern using pairs of Begin and End methods. Version 2 of the SDK also consists of a version compiled for .NET 3.5 Framework that contains the Begin and End methods for applications that aren’t yet ready to move to .NET 4.5.
  • The AWS SDK for .NET provides four distinct assemblies for developers to target different platforms. However, not all SDK functionality is available on each of these platforms. This guide describes the differences in what is supported across these platforms. We have also put together a migration guide that describes how version 2 of AWS SDK for .NET differs from the first version of the SDK and how to migrate your code to use the new SDK.
  • We have also added a new Amazon S3 encryption client in this SDK. This client allows you to secure your sensitive data before you send it to Amazon S3. Using the AmazonS3EncryptionClient class, the SDK automatically encrypts data on the client when uploading to Amazon S3, and automatically decrypts it when data is retrieved.

Breaking Changes

Below are the breaking changes in version 2 of the AWS SDK for .NET that you need to be aware of if you are migrating from version 1 of the SDK.

The region parameter is now mandatory

The SDK now requires the region to be explicitly specified through the client constructor or by using the AWSRegion setting in the application’s app or web config file. Prior versions of the SDK implicitly defaulted to us-east-1 if the region was not set. Here is an example of setting a region in the app config file so applications that are not explicitly setting a region can take this update without making any code changes.

<configuration>
  <appsettings>
    <add key="AWSRegion" value="us-east-1">
  </add></appsettings>
</configuration>

Here is an example of instantiating an Amazon S3 client using the new method on AWSClientFactory that accepts a RegionEndpoint parameter.

var s3Client = AWSClientFactory.CreateAmazonS3Client(accessKey,secretKey,RegionEndpoint.USWest2);

Fluent programming methods are no longer supported

The "With" methods on model classes that are present in version 1 of the SDK are not supported in version 2. You can use constructor initializers when creating new instances.

Here is an example that demonstrates this change. Calling the "With" methods using version 1 of the SDK to set up a TransferUtilityUploadRequest object looks like this:

TransferUtilityUploadRequest uploadRequest = new TransferUtilityUploadRequest()
    .WithBucketName("my-bucket")
    .WithKey("test")
    .WithFilePath("c:test.txt");

In version 2 of the SDK, you can instead use constructor initializers like this:

TransferUtilityUploadRequest uploadRequest = new TransferUtilityUploadRequest
{
    BucketName = "my-bucket",
    Key = "test",
    FilePath = "c:test.txt"
};

Resources

Here are a few resources that you will find handy while working with the new SDK.

See You at AWS re:Invent 2013

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

AWS re:Invent is next week (November 12th-15th) in Las Vegas! We are excited to be there and to have an opportunity to talk to you in person.

There is going to be a lot of great technical content year. Michael Dowling and I will be presenting a session on Friday called Mastering the AWS SDK for PHP. We will also be hanging out in the developer lounge area, so come by any time, especially during our PHP development office hours.

Last year Michael and I spoke as well. In case you missed it, our presentation was called Using Amazon DynamoDB Effectively with the AWS SDK for PHP. We also have the slides for that presentation posted on Slideshare. We’ll be sure to post the slides and video for this year’s presentation as well after the conference.

See you there!