Announcing the Developer Preview of Amazon S3 Transfer Manager in Rust

We are excited to announce the Developer Preview of the Amazon S3 Transfer Manager for Rust, a high-level utility that speeds up and simplifies uploads and downloads with Amazon Simple Storage Service (Amazon S3). Using this new library, developers can efficiently transfer data between Amazon S3 and various sources, including files, in-memory buffers, memory streams, and directories. The Transfer Manager automatically handles complex scenarios such as parallel multipart uploads and byte-range downloads to provide enhanced throughput and reliability.

Key features and benefits

• Automatic Request Splitting: Improves throughput by automatically splitting large requests into part-sized chunks and processing them in parallel across multiple connections. This overcomes single-connection bandwidth limitations.
• Automatic Retries: Enhances reliability by automatically retrying failed chunks independently, eliminating the need to restart entire transfers when intermittent errors occur.
• Optimized Parallel Reads: Implements parallel read operations across different sections of large files during uploads, providing superior performance compared to sequential reading.
• Simplified Directory Operations: Transfer entire directories or objects sharing a common prefix with a single API call, streamlining bulk transfer operations.
• Advanced Load Balancing: Automatically optimizes throughput by dynamically adjusting parallelism based on current workload and network bandwidth.

Getting started

To begin using the Transfer Manager, follow these examples:

Add project dependency

First, you need to add a new dependency in your Cargo.toml file.

aws-sdk-s3-transfer-manager = "0.1.1"

Create Transfer-Manager

Create a transfer manager with the default recommended settings:

let config = aws_sdk_s3_transfer_manager::from_env()
    .load()
    .await;
let transfer_manager = aws_sdk_s3_transfer_manager::Client::new(config);

Upload file example

This will upload the file by automatically splitting the request into part-sized chunks and uploading them in parallel.

`// Upload a single file to S3`
let bucket = "<BUCKET-NAME>";
let key = "<OBJECT-KEY>";
let file_to_upload = "<OBJECT-PATH>";

let stream = InputStream::from_path(file_to_upload)?;
let response = transfer_manager
    .upload()
    .bucket(bucket)
    .key(key)
    .body(stream)
    .initiate()?
    .join()
    .await;

Download file example

This will split the download into part-sized chunks, download them in parallel, and then deliver them in order.

// Download a single object from S3
let bucket = "<BUCKET-NAME>";
let key = "<OBJECT-KEY>";

let mut handle = transfer_manager
    .download()
    .bucket(bucket)
    .key(key)
    .initiate()?;

while let Some(chunk_result) = handle
    .body_mut()
    .next()
    .await {
    let chunk = chunk_result?.data.into_bytes();
    println!("Received {} bytes", chunk.len());
}

Upload directory example

This will recursively upload all files in the directory, combining the given key_prefix with each file’s path from the filesystem. For example, if your key_prefix is “prefix” and the file’s path is “test/docs/key.json“, the file will be uploaded with the object key of ”prefix/test/docs/key.json“.

// Upload a directory to S3
let bucket = "<BUCKET-NAME>";
let source_dir = "<SOURCE-DIRECTORY-PATH>";
let key_prefix = "<KEY-PREFIX>";

let handle = transfer_manager
    .upload_objects()
    .key_prefix(key_prefix)
    .bucket(bucket)
    .source(source_dir)
    .recursive(true)
    .send()
    .await?;

let response = handle.join().await?;

Download directory example

This will download every object under the prefix and will create a local directory with similar hierarchy.

// Download objects with a common prefix to a local directory
let bucket = "<BUCKET-NAME>";
let destination_dir = "<DESTINATION-DIRECTORY-PATH>";
let key_prefix = "<KEY-PREFIX>";

let handle = transfer_manager
    .download_objects()
    .key_prefix(key_prefix)
    .bucket(bucket)
    .destination(destination_dir)
    .send()
    .await?;

let response = handle.join().await?;

Conclusion

The Amazon S3 Transfer Manager for Rust simplifies Amazon S3 operations with efficient resource management and an easy-to-use API. We invite you to try the Developer Preview and share your feedback through GitHub issues. Your input will help shape the future development of this library to meet the needs of the Rust community.

About the author: