Announcing the general availability of Amazon S3 Transfer Manager for Swift

We are pleased to announce the general availability of the Amazon S3 Transfer Manager for Swift – a high level file and directory transfer utility for the Amazon Simple Storage Service (Amazon S3) built with the AWS SDK for Swift.

Using Transfer Manager’s simple API, you can perform accelerated uploads of local files and directories to Amazon S3 and accelerated downloads of objects and buckets from Amazon S3 and benefit from enhanced throughput and reliability, which is achieved through concurrent transfers of a set of small parts from a single object. The Transfer Manager is built on top of the AWS SDK for Swift and leverages Amazon S3 multipart upload and byte-range / part-number fetches for parallel transfers. You can also track the progress of transfers in real-time as well.

Parallel upload via multipart upload

For the upload operation, the Transfer Manager uses the Amazon S3 multipart upload API; it sends multiple UploadPart  requests concurrently behind the scenes to achieve high performance.

Parallel download via byte-ranges or part numbers

For the download operation, the Transfer Manager utilizes byte-range fetches or part number fetches. Byte range fetches download the object with byte ranges and works on all objects, regardless of whether it was originally uploaded using multipart upload or not. Part number fetches download the object in parts, using the part number assigned to each object part during upload. The transfer manager splits one GetObject request to multiple smaller requests, each of which retrieves a specific portion of the object. Those requests are also executed through concurrent connections to Amazon S3.

Getting Started

To get started with Amazon S3 Transfer Manager for Swift, complete the following steps:

Add the dependency to your Xcode project

1. Open your project in Xcode and click on your .xcodeproj file, located at the top of the file navigator on the left pane.
2. Click the project name that appears on the left pane of the .xcodeproj file window.
3. Click on Package Dependencies tab, and click + button.
4. In Search or Enter Package URL search bar, enter git@github.com:aws/aws-sdk-swift-s3-transfer-manager.git.
5. Wait for package to load, and once it’s loaded, choose the target you want to add the S3TransferManager module to.

Add the dependency to your Swift package

1. Add the below to your package definition:

dependencies: [
    .package(
        url: "https://github.com/aws/aws-sdk-swift-s3-transfer-manager.git",
        from: "<VERSION_STRING>"
    )
],

2. Add an S3TransferManager module dependency to the target that needs it. For example:

targets: [
    .target(
        name: "YourTargetThatUsesS3TM",
        dependencies: [
            .product(
                name: "S3TransferManager",
                package: "aws-sdk-swift-s3-transfer-manager"
            )
        ]
    )
]

Initialize the S3 Transfer Manager

You can initialize an S3TM instance with all-default settings by simply doing this:

// Creates and uses default S3TM config & S3 client.
let s3tm = try await S3TransferManager()

Or you can pass a configuration object to the initializer to customize S3TM, like this:

// Create the custom S3 client config that you want S3TM to use.
let customS3ClientConfig = try S3Client.S3ClientConfig(
    region: "us-west-2",
    . . . custom S3 client configurations . . .
)

// Create the custom S3TM config with the S3 client config initialized above.
let s3tmConfig = try await S3TransferManagerConfig(
    s3ClientConfig: customS3ClientConfig,
    targetPartSizeBytes: 10 * 1024 * 1024, // 10MB part size.
    multipartUploadThresholdBytes: 100 * 1024 * 1024, // 100MB threshold.
    multipartDownloadType: .part
)

// Finally, create the S3TM using the custom S3TM config.
let s3tm = S3TransferManager(config: s3tmConfig)

For more information about what each configuration does, please refer to the documentation comments on S3TransferManagerConfig.

Upload an object

To upload a file to Amazon S3, you need to provide the input struct UploadObjectInput, which contains a subset of PutObjectInput struct properties and an array of transfer listeners. You must provide the destination bucket, the S3 object key to use, and the object body.

When object being uploaded is bigger than the threshold configured by multipartUploadThresholdBytes (16MB default), S3TM breaks them down into parts, each with the part size configured by targetPartSizeBytes (8MB default), and uploads them concurrently using S3’s multipart upload feature.

// Construct UploadObjectInput.
let uploadObjectInput = UploadObjectInput(
    body: ByteStream.stream(
        FileStream(fileHandle: try FileHandle(forReadingFrom: URL(string: "file-to-upload.txt")!))
    ),
    bucket: "destination-bucket",
    key: "some-key"
)

// Call .uploadObject and save the returned task.
let uploadObjectTask = try s3tm.uploadObject(input: uploadObjectInput)
let uploadObjectOutput = try await uploadObjectTask.value

Download an object

To download an object from Amazon S3, you need to provide the input struct DownloadObjectInput, which contains the download destination, a subset of GetObjectInput struct properties, and an array of transfer listeners. The download destination is an instance of Swift’s Foundation.OutputStream. You must provide the download destination, the source bucket, and the S3 object key of the object to download.

When object being downloaded is bigger than the size of a single part configured by targetPartSizeBytes  (8MB default), S3TM downloads the object in parts concurrently using either part numbers or byte ranges as configured by multipartDownloadType (.part default).

// Construct DownloadObjectInput.
let downloadObjectInput = DownloadObjectInput(
    outputStream: OutputStream(toFileAtPath: "destination-file.txt", append: true)!,
    bucket: "source-bucket",
    key: "s3-object.txt"
)

// Call .downloadObject and save the returned task.
let downloadObjectTask = try s3tm.downloadObject(input: downloadObjectInput)
let downloadObjectOutput = try await downloadObjectTask.value

 Conclusion

To learn more about how to use the Amazon S3 Transfer Manager for Swift including how to upload a directory, download a bucket, and track transfer progress, visit our README.md on GitHub. Try out the new Transfer Manager today and let us know what you think via the GitHub issues page!

About the authors