Introducing Multipart Download Support for AWS SDK for .NET Transfer Manager

The new multipart download support in AWS SDK for .NET Transfer Manager improves the performance of downloading large objects from Amazon Simple Storage Service (Amazon S3). Customers are looking for better performance and parallelization of their downloads, especially when working with large files or datasets. The AWS SDK for .NET Transfer Manager (version 4 only) now delivers faster download speeds through automatic multipart coordination, eliminating the need for complex code to manage concurrent connections, handle retries, and coordinate multiple download streams.

In this post, we’ll show you how to configure and use these new multipart download capabilities, including downloading objects to files and streams, managing memory usage for large transfers, and migrating from existing download methods.

Parallel download using part numbers and byte-ranges

For download operations, the Transfer Manager now supports both part number and byte-range fetches. Part number fetches download the object in parts, using the part number assigned to each object part during upload. Byte-range fetches download the object with byte ranges and work on all objects, regardless of whether they were originally uploaded using multipart upload or not. The transfer manager splits your GetObject request into multiple smaller requests, each of which retrieves a specific portion of the object. The transfer manager executes your requests through concurrent connections to Amazon S3.

Choosing between part numbers and byte-range strategies

Choose between part number and byte-range downloads based on your object’s structure. Part number downloads (the default) work best for objects uploaded with standard multipart upload part sizes. If the object is a non-multipart object, choose byte-range downloads. Range downloads enable greater parallelization when objects have large parts (for example, splitting a 5GB part into multiple 50MB range requests for concurrent transfer) and work with any S3 object regardless of how it was uploaded.

Keep in mind that smaller range sizes result in more S3 requests. Each API call incurs a cost beyond the data transfer itself, so balance parallelism benefits against the number of requests for your use case.

Now that you understand the download strategies, let’s set up your development environment.

Getting started

To get started with multipart downloads in the AWS SDK for .NET Transfer Manager, follow these steps:

Add the dependency to your .NET project

Update your project to use the latest AWS SDK for .NET:

dotnet add package AWSSDK.S3 -v 4.0.17 

Or add the PackageReference to your .csproj file:

<PackageReference Include="AWSSDK.S3" Version="4.0.17" />; 

Initialize the Transfer Manager

You can initialize a Transfer Manager with default settings for typical use cases:

var s3Client = new AmazonS3Client(); 
var transferUtility = new TransferUtility(s3Client); 

You can customize the following options:

// Create custom Transfer Manager configuration 
var config = new TransferUtilityConfig 
{ 
    ConcurrentServiceRequests = 20,  // Maximum number of concurrent HTTP requests 
    BufferSize = 8192  // Buffer size in bytes for file I/O and HTTP responses 
}; 
 
// Create Transfer Manager with custom configuration 
var transferUtility = new TransferUtility(s3Client, config); 

Experiment with these values to find the optimal configuration for your use case. Factors like object size, available network bandwidth, and your application’s memory constraints will influence which settings work best. For more information about configuration options, please refer to the documentation on TransferUtilityConfig.

Download an object to file

To download an object from an Amazon S3 bucket to a local file, use the DownloadWithResponseAsync method. You must provide the source bucket, the S3 object key, and the destination file path.

// Download large file with multipart support (Part GET strategy) 
var downloadResponse = await transferUtility.DownloadWithResponseAsync( 
    new TransferUtilityDownloadRequest 
    { 
        BucketName = "amzn-s3-demo-bucket", 
        Key = "large-dataset.zip", 
        FilePath = @"C:\downloads\large-dataset.zip", 
        MultipartDownloadType = MultipartDownloadType.PART  // Default - uses S3 part numbers 
    }); 

// Download using Range GET strategy (works with any S3 object) 
var downloadResponse = await transferUtility.DownloadWithResponseAsync( 
    new TransferUtilityDownloadRequest 
    { 
        BucketName = "amzn-s3-demo-bucket", 
        Key = "any-object.dat", 
        FilePath = @"C:\downloads\any-object.dat", 
        MultipartDownloadType = MultipartDownloadType.RANGE,  // Uses HTTP byte ranges 
        PartSize = 16 * 1024 * 1024  // 16MB parts (default is 8MB) 
    }); 

Download an object to stream

To download an object from Amazon S3 directly to a stream, use the OpenStreamWithResponseAsync method. This is useful when you want to process data as it downloads without saving it to disk first. You must provide the source bucket and the S3 object key. The OpenStreamWithResponseAsync method performs parallel downloads by buffering parts in memory until they are read from the stream. See the configuration options below for how to control memory consumption during buffering.

// Stream large file with multipart coordination and memory control 
var streamResponse = await transferUtility.OpenStreamWithResponseAsync( 
    new TransferUtilityOpenStreamRequest 
    { 
        BucketName = "amzn-s3-demo-bucket", 
        Key = "large-video.mp4", 
        MaxInMemoryParts = 512,  // Maximum number of parts buffered in memory  (default is 1024)
                                  // Total memory = MaxInMemoryParts × PartSize 
        MultipartDownloadType = MultipartDownloadType.PART,  // Uses S3 part numbers 
        ChunkBufferSize = 64 * 1024  // Size of individual memory chunks (64KB) 
                                      // allocated from ArrayPool for buffering. (default is 64KB) 
    }); 
 
using var stream = streamResponse.ResponseStream; 
// Process stream data as it downloads concurrently 
var buffer = new byte[8192]; 
int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length); 

Memory management for streaming downloads: The MaxInMemoryParts parameter controls how many parts can be buffered simultaneously, and ChunkBufferSize determines the size of individual memory chunks allocated for buffering. You can experiment with different values for both parameters to find the optimal configuration for your specific use case.

Download a directory

To download multiple objects from an S3 bucket prefix to a local directory, use the DownloadDirectoryWithResponseAsync method. This method automatically applies multipart download to each individual object in the directory.

// Download entire directory with multipart support for large files 
await transferUtility.DownloadDirectoryWithResponseAsync( 
    new TransferUtilityDownloadDirectoryRequest 
    { 
        BucketName = "amzn-s3-demo-bucket", 
        S3Directory = "datasets/", 
        LocalDirectory = @"C:\data\" 
    }); 

Migration path

The new WithResponse methods provide both multipart performance and access to S3 response metadata. Here’s how to migrate your existing code:

For file downloads: 

// Existing code (still works, but returns void) 
await transferUtility.DownloadAsync(downloadRequest); 
 
// Enhanced version (new capabilities + metadata access) 
var response = await transferUtility.DownloadWithResponseAsync(downloadRequest); 
Console.WriteLine($"Downloaded {response.ContentLength} bytes"); 
Console.WriteLine($"ETag: {response.ETag}"); 

For streaming downloads: 

// Before: direct Stream return 
using var stream = await transferUtility.OpenStreamAsync(streamRequest); 
 
// After: access ResponseStream from response object 
var response = await transferUtility.OpenStreamWithResponseAsync(streamRequest); 
using var stream = response.ResponseStream; 
Console.WriteLine($"Content-Type: {response.ContentType}"); 
Console.WriteLine($"Last Modified: {response.LastModified}"); 

Conclusion

The multipart download support in the AWS SDK for .NET Transfer Manager provides performance improvements for downloading large objects from Amazon S3. By using parallel byte-range or part-number fetches, you can reduce transfer times.

Key takeaways from this post:

• Use DownloadWithResponseAsync and OpenStreamWithResponseAsync for downloads with automatic multipart coordination
• Choose between PART and RANGE download strategies based on your object’s structure
• Customize configuration settings based on your specific environment (memory, network bandwidth, etc.)

Next steps: Try implementing multipart downloads in your applications and measure the performance improvements for your specific use cases.

To learn more about the AWS SDK for .NET Transfer Manager, visit the AWS SDK for .NET documentation. For questions or feedback about this feature, visit the GitHub issues page.