Introducing Amazon S3 Transfer Manager in the AWS SDK for Java 2.x

We are pleased to announce the Developer Preview release of the Amazon S3 Transfer Manager – a high level file transfer utility for the Amazon Simple Storage Service (Amazon S3) in the AWS SDK for Java 2.x.

Using Transfer Manager’s simple API, you can now perform accelerated uploads and downloads of objects to and from Amazon S3 and benefit from enhanced throughput and reliability, which is achieved through concurrent transfers of a set of small parts from a single object. The Transfer Manager is built on top of the Java bindings of the AWS Common Runtime S3 client and leverages Amazon S3 multipart upload and byte-range fetches for parallel transfers.

Parallel upload via multipart upload

For upload operation, the Transfer Manager uses the Amazon S3 multipart upload API; it converts one single PutObjectRequest to multiple MultiPartUpload requests and sends those requests concurrently to achieve high performance.

Parallel download via byte-range fetches

For download operation, the Transfer Manager utilizes byte-range fetches. It splits a GetObjectRequest to multiple smaller requests, each of which retrieves a specific portion of the object. Those requests are also executed through concurrent connections to Amazon S3.

If you are using Transfer Manager 1.x, an object must be uploaded using multipart upload for it to be eligible to be downloaded with optimized performance. If an object was originally uploaded as a single object, the Transfer Manager will not be able to accelerate the downloading process.

With the new Transfer Manager 2.x, this is no longer a limitation – downloading an object does not depend on how the object was originally uploaded, and all downloads can benefit from high throughput and concurrency.

Getting Started

Add a dependency for the Transfer Manager

First, you need to include the separate dependency in your project.

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>s3-transfer-manager</artifactId>
  <version>2.17.123-PREVIEW</version>
</dependency>

Instantiate the Transfer Manager

You can instantiate the Transfer Manager easily using the default settings

S3TransferManager transferManager = S3TransferManager.create();

If you wish to configure settings, we recommend using the builder instead:

S3TransferManager transferManager =
    S3TransferManager.builder()
                     .s3ClientConfiguration(cfg -> cfg.credentialsProvider(credentialProvider)
                                                      .region(Region.US_WEST_2)
                                                      .targetThroughputInGbps(20.0)
                                                      .minimumPartSizeInBytes(10 * MB))
                     .build();

Upload a file to Amazon S3

To upload a file to S3, you need to provide the source file path and the PutObjectRequest that should be used for the upload.

FileUpload upload = transferManager.uploadFile(b -> b.source(Paths.get("myFile.txt"))
                                                     .putObjectRequest(req -> req.bucket("bucket")
                                                                         .key("key")));

upload.completionFuture().join();

Download an Amazon S3 object to a file

To download an object, you need to provide the destination file path and the GetObjectRequest that should be used for the download.

FileDownload download = 
    transferManager.downloadFile(b -> b.destination(Paths.get("myFile.txt"))
                                       .getObjectRequest(req -> req.bucket("bucket")
                                                                   .key("key")));
download.completionFuture().join();

Conclusion

To learn how to set up and begin using the Transfer Manager for the AWS SDK for Java 2.x, visit our Developer Guide. If you want to learn more, you can check out the source code on GitHub. Try out the new Transfer Manager today and let us know what you think via the GitHub issues page!