Category: Java


Client-side Encryption for Amazon DynamoDB

by Hanson Char | on | in Java | Permalink | Comments |  Share

We are thrilled to introduce one of the latest AWS Labs projects for enabling client-side encryption for Amazon DynamoDB in Java. This library is designed to support encryption and signing of your data when stored in Amazon DynamoDB.

A typical use of this library is when you are using DynamoDBMapper, where transparent encryption and signing of all objects serialized through the mapper can be enabled by configuring an AttributeEncryptor.

Getting Started

Suppose you have created (sample code) a DynamoDB table "MyStore", and want to store some Book objects. Let’s further suppose the security requirement involves classifying the attributes Title and Authors as sensitive information. Here is how the Book class might look like:

@DynamoDBTable(tableName="MyStore")
public class Book {
    private Integer id;
    private String title;
    private String ISBN;
    private Set bookAuthors;
    private String someProp;

    // Not encrypted because it is a hash key    
    @DynamoDBHashKey(attributeName="Id")  
    public Integer getId() { return id;}
    public void setId(Integer id) {this.id = id;}

    // Encrypted by default
    @DynamoDBAttribute(attributeName="Title")  
    public String getTitle() {return title; }
    public void setTitle(String title) { this.title = title; }

    // Specifically not encrypted
    @DoNotEncrypt
    @DynamoDBAttribute(attributeName="ISBN")  
    public String getISBN() { return ISBN; }
    public void setISBN(String ISBN) { this.ISBN = ISBN; }

    // Encrypted by default
    @DynamoDBAttribute(attributeName = "Authors")
    public Set<String> getBookAuthors() { return bookAuthors; }
    public void setBookAuthors(Set<String> bookAuthors) {
        this.bookAuthors = bookAuthors;
    }

    // Not encrypted nor signed
    @DoNotTouch
    public String getSomeProp() { return someProp;}
    public void setSomeProp(String someProp) {this.someProp = someProp;}
}

For a typical use case of DynamoDBMapper, you can easily save and retrieve a Book object to and from Amazon DynamoDB without encryption (nor signing). For example,

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
DynamoDBMapper mapper = new DynamoDBMapper(client);
Book book = new Book();
book.setId(123);
book.setTitle("Secret Book Title ");
// ... etc. setting other properties

// Saves the book unencrypted to DynamoDB
mapper.save(book);

// Loads the book back from DynamoDB
Book bookTo = new Book();
bookTo.setId(123);
Book bookTo = mapper.load(bookTo);

To enable transparent encryption and signing, simply specify the necessary encryption material via an EncryptionMaterialsProvider. For example:

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
SecretKey cek = ...;        // Content encrypting key
SecretKey macKey =  ...;    // Signing key
EncryptionMaterialsProvider provider = 
            new SymmetricStaticProvider(cek, macKey);
mapper = new DynamoDBMapper(client, DynamoDBMapperConfig.DEFAULT,
            new AttributeEncryptor(provider));
Book book = new Book();
book.setId(123);
book.setTitle("Secret Book Title ");
// ... etc. setting other properties

// Saves the book both encrypted and signed to DynamoDB
mapper.save(book);

// Loads the book both with signature verified and decrypted from DynamoDB
Book bookTo = new Book();
bookTo.setId(123);
bookTo = mapper.load(bookTo);

Note that by default all attributes except the primary keys are both encrypted and signed for maximum security. To selectively disable encryption, you can use the annotation @DoNotEncrypt as shown in the Book class above. To disable both encryption and signing, you can use the annotation @DoNotTouch.

There are a variety of existing EncryptionMaterialsProvider implementations that you can use to provide the encryption material, including KeyStoreMaterialsProvider which makes use of a Java keystore. Alternatively, you can also plug in your own custom implementation.

For more information, head over to aws-dynamodb-encryption-java, and give it a spin. Happy crypto, and may the power of security and Amazon DynamoDB be with you!

Introducing DynamoDB Document API (Part 2)

by Hanson Char | on | in Java | Permalink | Comments |  Share

In the previous blog, Introducing DynamoDB Document API (Part 1), we saw how to program against the DynamoDB Document API and produce code that is both easy to write and read.  But why is the API called the Document API, and how are JSON-style documents supported?

This perhaps can best be explained, well, with code! Using the same Game table from the previous blog, let’s start with a game object directly represented by a JSON document:

    {
        "Status" : "IN_PROGRESS",
        "GameId" : "abc",
        "Player1-Position" : 15,
        "Player2-Position" : 12
    }

First, we put this game to Amazon DynamoDB as a structured document:

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
DynamoDB dynamodb = new DynamoDB(client);
Table table = dynamo.getTable("Game");
    
String json = "{"
                    + ""Status" : "IN_PROGRESS","
                    + ""GameId" : "abc","
                    + ""Player1-Position" : 15,"
                    + ""Player2-Position" : 12"
                    + "}"
                    ;
Item jsonItem = Item.fromJSON(json);
table.putItem(jsonItem);

Suppose we need to update the game, changing the status to "SUSPENDED", and adding 1 to the first player’s position, but only if both players’ positions are less than 20 and if the current status is "IN_PROGRESS":

UpdateItemOutcome outcome = table.updateItem(new UpdateItemSpec()
            .withReturnValues(ReturnValue.ALL_NEW)
            .withPrimaryKey("GameId", "abc")
            .withAttributeUpdate(
                new AttributeUpdate("Player1-Position").addNumeric(1), 
                new AttributeUpdate("Status").put("SUSPENDED"))
            .withExpected(
                new Expected("Player1-Position").lt(20),
                new Expected("Player2-Position").lt(20),
                new Expected("Status").eq("IN_PROGRESS"))
        );

Finally, let’s get back the updated document as JSON:


Item itemUpdated = outcome.getItem();
String jsonUpdated = itemUpdated.toJSONPretty();
System.out.println(jsonUpdated);

Here is the output in JSON:

    {
      "Status" : "SUSPENDED",
      "GameId" : "abc",
      "Player1-Position" : 16,
      "Player2-Position" : 12
    }

As you can see, saving JSON as a structured document in Amazon DynamoDB, or updating, retrieving and converting the document back into JSON is as easy as 1-2-3. :)  You can find more examples in the A-Z Document API quick-start folder at GitHub. Happy coding until next time!

Introducing DynamoDB Document API (Part 1)

by Hanson Char | on | in Java | Permalink | Comments |  Share

Amazon DynamoDB has recently announced the support of storing entire JSON-style document as single DynamoDB items. What is as exciting is that the AWS SDK for Java has come up with a new Document API that makes it easy and simple to access all the feaures of Amazon DynamoDB, including the latest document support, but with less code!

The new Document API is designed from the ground up to be the next generation of API for accessing DynamoDB. It has an object-oriented API that provides full access to all the DynamoDB features including JSON data support, use of Document Path to access part of a document, new data types such as Map, List, etc.  The best part is, the resultant code is a lot less verbose, and is therefore both easier to write and read.

Alright, enough talking.  Perhaps the new API can best be illustrated with an example. Here I took the liberty of borrowing the code from a previous blog, Using Improved Conditional Writes in DynamoDB, and rewrite it using the new API. To begin with, the original code is copied here:

public static void main(String[] args) {
    // To run this example, first initialize the client, and create a table
    // named 'Game' with a primary key of type hash / string called 'GameId'.
    AmazonDynamoDB dynamodb; // initialize the client     
    try {
        // First set up the example by inserting a new item         
        // To see different results, change either player's
        // starting positions to 20, or set player 1's location to 19.
        Integer player1Position = 15;
        Integer player2Position = 12;
        dynamodb.putItem(new PutItemRequest()
                .withTableName("Game")
                .addItemEntry("GameId", new AttributeValue("abc"))
                .addItemEntry("Player1-Position",
                    new AttributeValue().withN(player1Position.toString()))
                .addItemEntry("Player2-Position",
                    new AttributeValue().withN(player2Position.toString()))
                .addItemEntry("Status", new AttributeValue("IN_PROGRESS")));
        // Now move Player1 for game "abc" by 1,
        // as long as neither player has reached "20".
        UpdateItemResult result = dynamodb.updateItem(new UpdateItemRequest()
            .withTableName("Game")
            .withReturnValues(ReturnValue.ALL_NEW)
            .addKeyEntry("GameId", new AttributeValue("abc"))
            .addAttributeUpdatesEntry(
                 "Player1-Position", new AttributeValueUpdate()
                     .withValue(new AttributeValue().withN("1"))
                     .withAction(AttributeAction.ADD))
            .addExpectedEntry(
                 "Player1-Position", new ExpectedAttributeValue()
                     .withValue(new AttributeValue().withN("20"))
                     .withComparisonOperator(ComparisonOperator.LT))
            .addExpectedEntry(
                 "Player2-Position", new ExpectedAttributeValue()
                     .withValue(new AttributeValue().withN("20"))
                     .withComparisonOperator(ComparisonOperator.LT))
            .addExpectedEntry(
                 "Status", new ExpectedAttributeValue()
                     .withValue(new AttributeValue().withS("IN_PROGRESS"))
                     .withComparisonOperator(ComparisonOperator.EQ))  
        );
        if ("20".equals(
            result.getAttributes().get("Player1-Position").getN())) {
            System.out.println("Player 1 wins!");
        } else {
            System.out.println("The game is still in progress: "
                + result.getAttributes());
        }
    } catch (ConditionalCheckFailedException e) {
        System.out.println("Failed to move player 1 because the game is over");
    }
}

Now, let’s rewrite the same code using the DynamoDB Document API:

public static void main(String[] args) {
    // Initialize the client and DynamoDB object
    AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
    DynamoDB dynamodb = new DynamoDB(client);

    try {
        Table table = dynamodb.getTable("Game");
        table.putItem(new Item()
            .withPrimaryKey("GameId", "abc")
            .withInt("Player1-Position", 15)
            .withInt("Player2-Position", 12)
            .withString("Status", "IN_PROGRESS"));
         
       UpdateItemOutcome outcome = table.updateItem(new UpdateItemSpec()
            .withReturnValues(ReturnValue.ALL_NEW)
            .withPrimaryKey("GameId", "abc")
            .withAttributeUpdate(
                new AttributeUpdate("Player1-Position").addNumeric(1))
            .withExpected(
                new Expected("Player1-Position").lt(20),
                new Expected("Player2-Position").lt(20),
                new Expected("Status").eq("IN_PROGRESS")));

        Item item = outcome.getItem();
        if (item.getInt("Player1-Position") == 20) {
            System.out.println("Player 1 wins!");
        } else {
            System.out.println("The game is still in progress: " + item);
        }
    } catch (ConditionalCheckFailedException e) {
        System.out.println("Failed to move player 1 because the game is over");
    }
}

As you see, the new Document API allows the direct use of plain old Java data types and has less boilerplate.  In fact, the Dynamo Document API can be used to entirely subsume what you can do with the low level client (i.e. AmazonDynamoDBClient) but with a much cleaner programming model and less code.

I hope this has whet your appetite in harnessing the power of Amazon DynamoDB using the Document API.  To see more examples, feel free to play with the code sample in the A-Z Document API quick-start folder at github, or check out the AWS blog.

AWS re:Invent 2014

by Jason Fulghum | on | in Java | Permalink | Comments |  Share

Just a little over a month from now AWS re:Invent 2014 kicks off! This year, you’ll have over 20 different tracks to choose from – from Big Data to Web Development, and everything in between.

The AWS SDK for Java team will be at AWS re:Invent this year, and presenting a session on using the AWS SDK for Java. In particular, we’ll be highlighting many enhancements we’ve built into the AWS SDK for Java in the past year.

We all look forward to AWS re:Invent every year because it’s such a great opportunity to connect with you, hear about what you need, and teach you about some of the new stuff we’ve been working on.

It’s not too late to register for AWS re:Invent 2014! See you in Las Vegas!

Introducing the AWS Resource APIs for Java

by David Murray | on | in Java | Permalink | Comments |  Share

Today we’re excited to announce the first developer preview release of the AWS Resource APIs for Java!

The AWS SDK for Java provides a set of Amazon[Service]Client classes, each exposing a direct mapping of each AWS service’s API. These client objects have a method for each operation that the service supports, with corresponding POJOs representing the request parameters and the response data. Using this "low-level" client API gives you full control over what requests you’re making to the service, allowing you to very tightly control the behavior and performance of your calls to AWS services, but it can also be a bit intimidating to a new user.

With the resource APIs, we’re hoping to improve this experience by providing a higher-level, object-oriented abstraction on top of the low-level SDK clients. Instead of a single client object exposing a service’s entire API, with the resource APIs we’ve defined a class representing each of the conceptual "resources" that you interact with while using a service. These classes expose getters for data about the resource, actions that can be taken on the resource, and links to other related resources. For example, using the EC2 API:

Instance instance = ec2.getInstance("i-xxxxxxxx");
System.out.println(instance.getDnsName());
instance.terminate();

Trying the API

First you’ll need to get your hands on the library. The easiest way to do this is via Maven:

    <dependency>
      <groupId>com.amazonaws.resources</groupId>
      <artifactId>aws-resources</artifactId>
      <version>0.0.1</version>
      <type>pom</type>
    </dependency>

Or alternatively, you can download the preview release here.

Creating a Service

A service object is the entry point for creating and interacting with resource objects. You create service instances by using the ServiceBuilder:

EC2 ec2 = ServiceBuilder.forService(EC2.class)
    .withCredentials(new ProfileCredentialsProvider("test-app"))
    .withRegion(Region.getRegion(Regions.US_WEST_2))
    .build();

Moving on to Resource objects

From the Service instance, you can follow references to resources exposed by the service:

// Get an instance reference by id
Instance instance = ec2.getInstance("i-xxxxxxxx");
System.out.println(instance.getDnsName());

// Enumerate all current instances for this account/region
for (Instance instance : ec2.getInstances()) {
    System.out.println(instance.getDnsName();
}

You can also follow references from one resource to another:

Instance instance = ...;
Subnet subnet = instance.getSubnet();
System.out.println(subnet.getCidrBlock());

for (Volume volume : instance.getVolumes()) {
    System.out.println(volume.getVolumeType() + " : " + volume.getSize());
}

Conclusion

This release is a developer preview, with support for Amazon EC2, AWS Identity and Access Management, and Amazon Glacier. You can browse through the API or go straight to the code. We’ll be adding support for more services and tweaking the API over the next couple months as we move toward GA. We’re very interested to hear your thoughts on the APIs. Please give it a try and let us know what you think on GitHub, Twitter, or here in the comments!

AWS Ant Tasks

by Jesse Duarte | on | in Java | Permalink | Comments |  Share

Introducing a new AWS Labs project: AWS Ant Tasks. These are custom tasks to use within your Ant builds that allow easy access to AWS services. Ant is a commonly used tool for building Java projects, and now you can use it to deploy your project to AWS within the same build. To use these tasks, simply reference “taskdefs.xml” in the project’s jar. The services currently available are Amazon S3 and Amazon Elastic Beanstalk, with AWS OpsWorks on its way soon. If you currently develop a Java application and deploy new versions to an AWS service often, consider trying these tasks out!

Here’s an example of an Ant build that updates an Elastic Beanstalk environment with a new war file:

<project basedir="." default="deploy" name="mybeanstalkproject">
    <taskdef resource="taskdefs.xml" 
             classpath="lib/aws-java-sdk-ant-tasks-1.0.0.jar" />
	
    <target name="compile">
        <mkdir dir="build/classes"/> 
        <javac srcdir="src" destdir="build/classes” />
    </target>

    <target name="war" depends="compile">
	    <war destfile="dist/MyProject.war" webxml="WebContent/WEB-INF/web.xml">
	        <fileset dir="WebContent" />
	        <lib dir="WebContent/WEB-INF/lib"/>
	        <classes dir="build/classes" />
	    </war>
    </target>

    <target name="deploy" depends="war">
         <deploy-beanstalk-app bucketName="mys3bucket" 
             versionLabel="version 0.2"
             versionDescription="Version 0.2 of my app" 
             applicationName="myapp" 
             environmentName="myenv" 
             file="dist/MyProject.war" />
    </target>
</project>

Now if you run ant deploy, this build file will compile, war, and deploy your project to your Elastic Beanstalk environment. You can immediately view the results by heading to your environment. If you use Ant to build your project, this task has the potential to be very helpful for deploying in one step as soon as your code is ready. For more in-depth documentation and example code, check out the README documentation on GitHub.

What’s the next AWS service you’d like to see with Ant integration? Is there an Ant task you’d like to contribute to the GitHub project?

Determining an Application’s Current Region

by Jason Fulghum | on | in Java | Permalink | Comments |  Share

AWS regions allow you to run your applications in multiple geographically distributed locations all over the world. This allows you to position your applications and data near your customers for the best possible experience. There are ten AWS regions available today:

  • 4 regions in North America
  • 4 regions in Asia Pacific
  • 1 region in South America
  • 1 region in Europe

When you host an application in one region, you typically want to use the AWS services available in that region, since they’ll give you lower latency and higher throughput. If your application is running on Amazon EC2 instances, the latest version of the AWS SDK for Java enables you to easily detect what AWS region those instances are in. Before being able to detect this, if you wanted to run your application in multiple regions, you needed to give your application a region-specific configuration so it knew what regional endpoints to use. The new Regions.getCurrentRegion() method makes this a lot easier. For example, if you start an Amazon EC2 instance in us-west-1 and run your application on that instance, it would know it’s running in us-west-1 and you could use that information to easily configure your application to talk to other services in us-west-1.

// When running on an Amazon EC2 instance, this method
// will tell you what region your application is in
Region region = Regions.getCurrentRegion();

// If you aren’t running in Amazon EC2, then region will be null
// and you can set whatever default you want to use for development
if (region == null) region = Region.getRegion(Regions.US_WEST_1);

// Then just use that same region to construct clients 
// for all the services you want to work with
AmazonDynamoDBClient dynamo = new AmazonDynamoDBClient();
AmazonSQSClient sqs = new AmazonSQSClient();
dynamo.setRegion(region);
sqs.setRegion(region);

How are you using AWS regions? Are you running any applications in multiple regions?

Accessing Private Content in Amazon CloudFront

by Jason Fulghum | on | in Java | Permalink | Comments |  Share

Amazon CloudFront is an easy to use, high performance, and cost efficient content delivery service. With over 50 worldwide edge locations, CloudFront is able to deliver your content to your customers with low latency in any part of the world.

In addition to serving public content for anyone on the Internet to access, you can also use Amazon CloudFront to distribute private content. For example, if your application requires a subscription, you can use Amazon CloudFront’s private content feature to ensure that only authenticated users can access your content and prevent users from accessing your content outside of your application.

Accessing private content in Amazon CloudFront is now even easier with the AWS SDK for Java. You can now easily generate authenticated links to your private content. You can distribute these links or use them in your application to enable customers to access your private content. You can also set expiration times on these links, so even if your application gives a link to a customer, they’ll only have a limited time to access the content.

To use private content with Amazon CloudFront, you’ll need an Amazon CloudFront distribution with private content enabled and a list of authorized accounts you trust to access your private content. From the Create Distribution Wizard in the Amazon CloudFront console, start creating a web distribution. In the ”’Origin Settings”’ section, select an Amazon S3 bucket that you’ve created for private content only, and make sure you select the options as below:

This will set the permissions on your Amazon S3 bucket to protect your content from being accessed publicly, but still allow CloudFront to access your content.

Continue creating your distribution, and at the bottom of the Default Cache Behavior Settings section, make sure you enable the Restrict Viewer Access option and select self as the trusted signer. These are called trusted signers because you’re trusting URLs that are signed by them and allowing them to access your private content. In our example, we’re using self as the only trusted signer, which means that only your account can sign URLs to access your CloudFront private content.

The last thing you need to set up in your account is a CloudFront key pair. This is the public/private key pair that you’ll use to sign requests for your CloudFront private content. Any trusted signer that you configure for your CloudFront distribution will need to set up their own CloudFront key pair for their account in order to sign requests for your CloudFront private content. You can configure your CloudFront key pair through the Security Credentials page in the IAM console. Make sure you download your private key, and make a note of the key pair ID listed in the AWS Management Console.

Now that your account and distribution are configured, you’re ready to use the SDK to generate signed URLs for accessing your CloudFront private content. The CloudFrontUrlSigner class in the AWS SDK for Java makes it easy to create signed URLs that you and your customers can use to access your private content. In the following example, we create a signed URL that expires in 60 seconds and allows us to access the private foo/bar.html content in our CloudFront distribution.

// the DNS name of your CloudFront distribution, or a registered alias
String distributionDomainName;   
// the private key you created in the AWS Management Console
File cloudFrontPrivateKeyFile;
// the unique ID assigned to your CloudFront key pair in the console    
String cloudFrontKeyPairId;   
Date expirationDate = new Date(System.currentTimeMillis() + 60 * 1000);

String signedUrl = CloudFrontUrlSigner.getSignedURLWithCannedPolicy(
           Protocol.https, 
           distributionDomainName, 
           cloudFrontPrivateKeyFile,   
           “foo/bar.html”, // the resource path to our content
           cloudFrontKeyPairId, 
           expirationDate);

You can also attach additional policy restrictions to the presigned URLs you create with CloudFrontUrlSigner. The following example shows how to create a policy to restrict access to a CIDR IP range, which can be useful to limit access to your private content to users on a specific network:

// the DNS name of your CloudFront distribution, or a registered alias
String distributionDomainName;   
// the private key you created in the AWS Management Console
File cloudFrontPrivateKeyFile;
// the unique ID assigned to your CloudFront key pair in the console   
String cloudFrontKeyPairId;   
// the CIDR range limiting which IP addresses are allowed to access your content
String cidrRange; 
// the resource path to our content
String resourcePath  = "foo/bar.html";  
Date expirationDate = new Date(System.currentTimeMillis() + 60 * 1000);

String policy = buildCustomPolicyForSignedUrl(
                    resourcePath,
                    expirationDate,
                    cidrRange,
                    null);

String signedUrl = CloudFrontUrlSigner.getSignedURLWithCustomPolicy(
                    resourcePath,
                    cloudFrontKeyPairId,
                    cloudFrontPrivateKey,
                    policy);

Are you already an Amazon CloudFront customer? Have you tried out Amazon CloudFront private content yet?

Pausing and Resuming transfers using Transfer Manager

by Manikandan Subramanian | on | in Java | Permalink | Comments |  Share

One of the really cool features that TransferManager now supports is pausing and resuming file uploads and downloads. You can now pause a very large file upload and resume it at a later time without having the necessity to re-upload the bytes that have been already uploaded. Also, this helps you survive JVM crashes as the operation can be resumed from the point at which it was stopped.

On an upload/download operation, TransferManager tries to capture the information that is required to resume the transfer after the pause. This information is returned as a result of executing the pause operation.

Here is an example of how to pause an upload:

// Initialize TransferManager.
TransferManager tm = new TransferManager();

// Upload a file to Amazon S3.
Upload myUpload = tm.upload(myBucket, myKey, myFile);

// Sleep until data transferred to Amazon S3 is less than 20 MB.
long MB = 1024 * 1024;
TransferProgress progress = myUpload.getProgress();
while( progress.getBytesTransferred() < 20*MB ) Thread.sleep(2000);

// Initiate a pause with forceCancelTransfer as true. 
// This cancels the upload if the upload cannot be paused.
boolean forceCancel = true;
PauseResult<PersistableUpload> pauseResult = myUpload.tryPause(forceCancel);

In some cases, it is not possible to pause the upload. For example, if the upload involves client-side encryption using AmazonS3EncryptionClient, then TransferManager doesn’t capture the encrypted context for security reasons and will not be able to resume the upload. In such cases, the user can decide to cancel the uploads by setting the forceCancelTransfers attribute of Upload#tryPause(boolean). The status of the pause operation can be retrieved using PauseResult#getPauseStatus() and can be one of the following.

  • SUCCESS – Upload is successfully paused.
  • CANCELLED – User requested to cancel the upload if the pause has no effect on the upload.
  • CANCELLED_BEFORE_START – User tried to pause the upload even before the start and cancel was requested.
  • NO_EFFECT – Pause operation has no effect on the upload. Upload continues to transfer data to Amazon S3.
  • NOT_STARTED – Pause operation has no effect on the upload because it has not yet started.

On a successful upload pause, PauseResult#getInfoToResume() returns an instance of PersistableUpload that can be used to resume the upload operation at a later time. To persist this information to a file, use the following code,

// Retrieve the persistable upload from the pause result.
PersistableUpload persistableUpload = pauseResult.getInfoToResume();

// Create a new file to store the information.
File f = new File("resume-upload");
if( !f.exists() ) f.createNewFile();
FileOutputStream fos = new FileOutputStream(f);

// Serialize the persistable upload to the file.
persistableUpload.serialize(fos);
fos.close();    

While the Upload#tryPause(boolean) returns a PauseResult when the pause operation succeeds or fails, there is an Upload#pause() that throws an PauseException in case the upload cannot be paused.

Here is an example of how to resume an upload.

// Initialize TransferManager.
TransferManager tm = new TransferManager();

FileInputStream fis = new FileInputStream(new File("resume-upload"));

// Deserialize PersistableUpload information from disk.
PersistableUpload persistableUpload = PersistableTransfer.deserializeFrom(fis);

// Call resumeUpload with PersistableUpload.
tm.resumeUpload(persistableUpload);

fis.close();

TransferManager skips the parts of the file that was uploaded previously and uploads the rest to Amazon S3.

Similar to the upload example, the following example pauses an Amazon S3 object download and persists the PersistableDownload to a file.

// Initialize TransferManager.
TransferManager tm = new TransferManager();

//Download the Amazon S3 object to a file.
Download myDownload = tm.download(myBucket, myKey, new File("myFile"));

// Sleep until the progress is less than 20 MB.
long MB = 1024 * 1024;
TransferProgress progress = myDownload.getProgress();
while( progress.getBytesTransferred() < 20*MB ) Thread.sleep(2000);

// Pause the download.
PersistableDownload persistableDownload = myDownload.pause();

// Create a new file to store the information.
File f = new File("resume-download");
if( !f.exists() ) f.createNewFile();
FileOutputStream fos = new FileOutputStream(f);

// Serialize the persistable download to a file.
persistableDownload.serialize(fos);
fos.close();

To resume a download, use the following code

// Initialize TransferManager.
TransferManager tm = new TransferManager();

FileInputStream fis = new FileInputStream(new File("resume-download"));

// Deserialize PersistableDownload from disk.
PersistableDownload persistDownload = PersistableTransfer.deserializeFrom(fis);

// Call resumeDownload with PersistableDownload.
tm.resumeDownload(persistDownload);

fis.close();

TransferManager performs a range GET operation during the resumeDownload operation to download the remaining Amazon S3 object contents. ETag’s are returned only when downloading whole Amazon S3 objects and hence ETag validation is skipped during resumeDownload operation. Also, resuming a download for an object encrypted using CryptoMode.StrictAuthenticatedEncryption would result in AmazonClientException because authenticity cannot be guaranteed for a range GET operation.

In order to support resuming uploads/downloads during JVM crashes, PersistableUpload or PersistableDownload must be serialized to disk as soon as it is available. You can achieve this by passing an instance of S3SyncProgressListener to TransferManager#upload or TransferManager#download that serializes the data to disk. The following example shows how to serialize the data to a file without calling a pause operation.

// Initialize TransferManager.
TransferManager tm = new TransferManager();

PutObjectRequest putRequest = new PutObjectRequest(myBucket,myKey,file);

// Upload a file to Amazon S3.
tm.upload(putRequest, new S3SyncProgressListener() {

    ExecutorService executor = Executors.newFixedThreadPool(1);

    @Override
    public void onPersistableTransfer(final PersistableTransfer persistableTransfer) {

       executor.submit(new Runnable() {
          @Override
          public void run() {
              try {
                  File f = new File("resume-upload");
                  if (!f.exists()) {
                      f.createNewFile();
                  }
                  FileOutputStream fos = new FileOutputStream(f);
                  persistableTransfer.serialize(fos);
                  fos.close();
              } catch (IOException e) {
                  throw new RuntimeException("Unable to persist transfer to disk.", e);
              }
          }
       });
    }
});

As the name indicates, S3SyncProgressListener is executed in the same thread as the upload/download operation. It should be very fast and return control to TransferManager since it will affect the performance of the upload/download. Note that the above example code is for illustrative purposes only, so in your progress listener implementation you must avoid blocking operations such as writing to disk.

Do you like the new Pause and Resume functionality supported by TransferManager? Let us know your feedback in the comments.

Follow up on Base64 Codec Performance

by Hanson Char | on | in Java | Permalink | Comments |  Share

After we posted the previous blog, A Fast and Correct Base64 Codec, some readers expressed interest in getting more details about the comparison of various codecs’ performance. So this blog post is a quick follow-up with a side-by-side decode/encode performance comparison of various Base64 codec’s, including AWS SDK for Java, DataTypeConverter, Jakarta Commons Codec and Java 8.

To generate the performance data, we used the same test suite as in the previous blog, except this time we ran the tests in a Java 8 VM (instead of Java 7 as in the original blog) so as to include the Java 8-specific Base 64 codec for comparison. (The particular JVM is Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode.)

As you can see from the graph and statistics below, in terms of raw performance, the DataTypeConverter is the fastest in terms of Base64 decoding, whereas Java 8 is the clear winner in terms of Base64 encoding. However, as explained in the previous blog, decoding speed is not the only factor in selecting the Base64 decoder for use in the AWS SDK for Java, but also security and correctness.

Hope you find this information useful.

Decoder Performance

Base 64 decoder speed density plots

                (Decoder performance statistics associated with the graph generated via R above.)
                              vars      n  mean   sd median trimmed  mad  min max range skew  kurtosis  se
Commons                1 1000 24.28 4.36     24        23.88  4.45  18   50    32      1.06     2.12    0.14
DataTypeConverter 2 1000  8.51 2.55        8          8.14  1.48    6   51    45      5.83    79.84   0.08
Java8                       3 1000 11.43 2.96     11         11.11  2.97    8   60    52      4.85    70.63   0.09
SDK                         4 1000 13.77 2.80     13        13.47  2.97  10   39    29      2.27    13.50   0.09

    Commons      DataTypeConverter     Java8            SDK       
 Min.     :18.00     Min.   :  6.000           Min.      : 8.00    Min.     :10.00  
 1st Qu.:21.00   1st Qu. : 7.000           1st Qu.  : 9.00   1st Qu. :12.00  
 Median :24.00  Median : 8.000           Median :11.00  Median :13.00  
 Mean   :24.28   Mean   :  8.506           Mean   :11.43   Mean   :13.77  
 3rd Qu.:27.00   3rd Qu.:  9.000           3rd Qu.:13.00   3rd Qu.:16.00  
 Max.    :50.00    Max.    :51.000           Max.    :60.00   Max.    :39.00  
 

Encoder Performance

Base 64 encoder speed density plots
 

                  (Encoder performance statistics associated with the graph generated via R above.)
                                 vars    n  mean   sd median trimmed  mad min max range  skew kurtosis   se
Commons                 1 1000 21.94 3.68     21   21.46       2.97  17  69     52      2.99     27.04  0.12
DataTypeConverter  2 1000 11.88 9.70     10    0.91        1.48    7 148   141    11.06   127.96  0.31
Java8                        3 1000  7.22  3.56      6     6.91       1.48    5   99     94    18.08   447.36  0.11
SDK                          4 1000 11.84 9.66     10   10.84       1.48    5 139   134    10.52   114.39  0.31

    Commons      DataTypeConverter       Java8                 SDK        
 Min.      :17.00   Min.      :   7.00    Min.      : 5.000   Min.      :    5.00  
 1st Qu. :19.00   1st Qu. : 10.00    1st Qu.  : 6.000   1st Qu. :  10.00  
 Median :21.00   Median : 10.00    Median : 6.000   Median :  10.00  
 Mean    :21.94   Mean    : 11.88    Mean    : 7.225   Mean   :  11.84  
 3rd Qu. :24.00   3rd Qu. : 12.00    3rd Qu. : 8.000   3rd Qu.:  12.00  
 Max.     :69.00   Max.     :148.00    Max.    :99.000   Max.    :139.00