Transfer Manager Directory Support for AWS SDK for Ruby

Managing bulk file transfer to Amazon Simple Storage Service (Amazon S3) can be complex when transferring directories containing multiple files and subdirectories. AWS SDK for Ruby Transfer Manager (aws-sdk-s3 version 1.215) now supports directory upload and download. This feature can help streamline bulk transfers by providing multipart handling and parallelism options.

Previously, uploading directories to Amazon S3 required manual iteration and handling. You also had to manage multipart uploads for large files and implement parallelism for performance. With directory support in Transfer Manager, you can handle this with a single method call that automates the process. In this post, we show you how to upload and download directories using Transfer Manager, customize transfer with filtering options and handle results effectively.

Getting started

This support requires aws-sdk-s3 version 1.215 or higher. Add aws-sdk-s3 to your Gemfile:

gem 'aws-sdk-s3', '>= 1.215'

Initialize the Transfer Manager

To initialize a Transfer Manager with a default S3 client:

require 'aws-sdk-s3' 
tm = Aws::S3::TransferManager.new

Or you could create a custom S3 client to pass to the Transfer Manager.

client = Aws::S3::Client.new(region: 'us-east-1')
tm = Aws::S3::TransferManager.new(client: client)

Upload a directory

Upload a local directory to an S3 bucket by providing a source path and bucket name:

tm.upload_directory('/path/to/directory', bucket: 'my-bucket')

By default, only files in the specified directory are uploaded. To include subdirectories, set recursive: true:

tm.upload_directory('/path/to/directory', bucket: 'my-bucket', recursive: true)

Download a directory

Download objects from an S3 bucket to a local directory by providing a destination path and bucket name:

tm.download_directory('/local/path', bucket: 'my-bucket')

To download only objects with a specific prefix, set s3_prefix. The full object key is preserved in the local path. For example, given s3_prefix: 'photos/':

• Object key: photos/vacation/beach.jpg
• Resolved local path: /local/path/photos/vacation/beach.jpg

tm.download_directory('/local/path', bucket: 'my-bucket', s3_prefix: 'photos/2026/')

Filtering contents

You can also filter transfers by using filter_callback:

# Upload only .txt files 
filter = proc { |_path, name| name.end_with?('.txt') } 
tm.upload_directory('/path/to/directory', bucket: 'my-bucket', filter_callback: filter) 

# Download only .jpg files 
filter = proc { |obj| obj.key.end_with?('.jpg') } 
tm.download_directory('/local/path', bucket: 'my-bucket', filter_callback: filter)

Handling results

On success, both operations return a hash containing completed and failed transfer details:

result = tm.upload_directory('/path/to/directory', bucket: 'my-bucket') 
# => { completed_uploads: 7, failed_uploads: 0 }

By default, an error raises an exception, which stops the transfer but does not clean up completed transfers. You can set ignore_failure: true to continue transferring remaining files and see what errors occurred in the results hash.

result = tm.upload_directory(
  '/path/to/directory', 
  bucket: 'my-bucket', 
  ignore_failure: true
)
# => { completed_uploads: 5, failed_uploads: 2, errors: [...] }

Conclusion

Directory upload and download support in the AWS SDK for Ruby Transfer Manager can help streamline bulk S3 transfers with built-in parallelism and multipart handling.

Key takeaways:

• Use upload_directory and download_directory for bulk transfers with a single method call
• Customize behavior with options like recursive, s3_prefix, and filter_callback
• Handle errors gracefully with ignore_failure and inspect results for details

These are only a few of the available options. See the API documentation for a full list.

Next steps: Try implementing directory transfers in your applications and explore other Transfer Manager features like upload_file and download_file for single-object transfers.

Share your questions, comments, and issues with us on GitHub.

About the author