S3TransferManager for iOS – Part I: Asynchronous Uploads

Transferring files to and from Amazon Simple Storage Service (S3) is one of the most common operations we see for developers using the AWS SDK for iOS. Handling multiple files or larger files can sometimes be difficult, particularly in a way that is asynchronous and allows for accurate progress and retries. To help developers handle these kinds of operations, we added the S3TransferManager class.

S3TransferManager is an easy-to-use, high level component that is designed to efficiently upload lots of large files to Amazon S3. Underneath, it uses NSOperationQueue to handle multiple uploads efficiently, and automatically choose the right upload mode: multipart uploads for large files, and put requests for small files. Multipart uploads break up large files into multiple parts and individually upload these parts. When a part upload fails, it retries only the part that failed. S3TransferManager also aggregates the AmazonServiceRequestDelegate methods for multiple part uploads. Because of many added benefits, consider using S3TransferManager first, whenever you need to upload data to Amazon S3 before directly using AmazonS3Client to do so.

Using S3TransferManager

It is easy to get started with S3TransferManager. Here are the steps to make an asynchronous upload request:

1. Instantiate an AmazonS3Client object.
2. Instantiate an S3TransferManager object.
3. Pass the AmazonS3Client object to the S3TransferManager object.
4. Create and configure an S3PutObjectRequest object.
5. Invoke upload: on the S3TransferManager object with the S3PutObjectRequest object.

Here is a small code snippet to asynchronously upload a file to Amazon S3:

// Note that this is not the preferred way to create the AmazonS3Client object. Do not ship apps with your credentials in them.
AmazonS3Client *s3 = [[AmazonS3Client alloc] initWithAccessKey:@"your-access-key"
                                                 withSecretKey:@"your-secret-key"];

NSString *filePath = [[NSBundle mainBundle] pathForResource:@"testData"
                                                     ofType:@"txt"];
self.tm = [S3TransferManager new];
self.tm.s3 = s3;

S3PutObjectRequest *putObjectRequest = [[S3PutObjectRequest alloc] initWithKey:@"your-key"
                                                                      inBucket:@"your-bucket"];
putObjectRequest.filename = filePath;

[self.tm upload:putObjectRequest];

Please note that upload: is an asynchronous method and returns immediately. Since it doesn’t block the running thread, it is safe to call this method on the main thread.

S3TransferManager has the following three convenient methods for asynchronously uploading files:

- (void)uploadData:(NSData *)data bucket:(NSString *)bucket key:(NSString *)key;
- (void)uploadFile:(NSString *)filename bucket:(NSString *)bucket key:(NSString *)key;
- (void)uploadStream:(NSInputStream *)stream contentLength:(long)contentLength bucket:(NSString *)bucket key:(NSString *)key;

For instance, you can rewrite the first code snippet using the uploadFile:bucket:key: method:

// Note that this is not the preferred way to create the AmazonS3Client object. Do not ship apps with your credentials in them.
s3 = [[AmazonS3Client alloc] initWithAccessKey:@"your-access-key"
                                 withSecretKey:@"your-secret-key"];

NSString *filePath = [[NSBundle mainBundle] pathForResource:@"testData"
                                                     ofType:@"txt"];

self.tm = [S3TransferManager new];
self.tm.s3 = s3;

[self.tm uploadFile:filePath bucket:@"your-bucket" key:@"your-key"];

Configuring S3TransferManager

You can customize S3TransferManager behaviors. I’m going to highlight three of them.

Getting responses through AmazonServiceRequestDelegate

First, your delegate object needs to adopt the AmazonServiceRequestDelegate protocol:

#import <AWSiOSSDK/S3/AmazonS3Client.h>

@interface YourViewController : UIViewController <AmazonServiceRequestDelegate>
{
}

Then pass the delegate object to S3PutObjectRequest:

putObjectRequest.delegate = self;

Also, you can specify a default delegate by setting it to the S3TransferManager object:

self.tm.delegate = self;

Whenever the delegate property of the S3PutObjectRequest is not set, S3TransferManager‘s delegate is used. For the aforementioned convenient methods, which don’t take S3PutObjectRequest, S3TransferManager‘s delegate is always used.

Now you can receive callbacks through AmazonServiceRequestDelegate:

-(void)request:(AmazonServiceRequest *)request didReceiveResponse:(NSURLResponse *)response
{
    NSLog(@"didReceiveResponse called: %@", response);
}

-(void)request:(AmazonServiceRequest *)request didReceiveData:(NSData *)data
{
    NSLog(@"didReceiveData called");
}

-(void)request:(AmazonServiceRequest *)request didSendData:(NSInteger)bytesWritten totalBytesWritten:(NSInteger)totalBytesWritten totalBytesExpectedToWrite:(NSInteger)totalBytesExpectedToWrite
{
    NSLog(@"didSendData called: %d - %d / %d", bytesWritten, totalBytesWritten, totalBytesExpectedToWrite);
}

-(void)request:(AmazonServiceRequest *)request didCompleteWithResponse:(AmazonServiceResponse *)response
{
    NSLog(@"didCompleteWithResponse called: %@", response);
}

-(void)request:(AmazonServiceRequest *)request didFailWithError:(NSError *)error
{
    NSLog(@"didFailWithError called: %@", error);
}

These delegate methods are always called on the main thread so you can update UI elements in these methods. Also, be reminded not to make any synchronous network requests on the main thread.

The delegate of S3TransferManager is shared among multiple upload requests, so often it is beneficial to tag the request so that you can differentiate each request in the delegate methods.

putObjectRequest.requestTag = @"your-unique-tag";
[self.tm upload:putObjectRequest];

For example, if you tag your request like above, you can update a specific progress bar for the request in the delegate method:

-(void)request:(AmazonServiceRequest *)request didSendData:(NSInteger)bytesWritten totalBytesWritten:(NSInteger)totalBytesWritten totalBytesExpectedToWrite:(NSInteger)totalBytesExpectedToWrite
{
    if([request.requestTag isEqualToString:@"your-unique-tag"])
    {
        // Update the progress bar for "your-unique-tag".
    }
}

Customize AmazonS3Client Properties

You can customize properties of AmazonS3Client. Here is an example to update endpoint, maxRetries, and timeout:

s3.endpoint = [AmazonEndpoints s3Endpoint:US_WEST_2];
s3.maxRetries = 10;
s3.timeout = 240;

self.tm = [S3TransferManager new];
self.tm.s3 = s3;

Note that explicitly setting the endpoint to the region that the bucket is located in will improve performance.

Customize S3TransferManager Properties

You can also update the properties of S3TransferManager. Here is an example to limit the maximum number of concurrent upload operations allowed:

self.tm = [S3TransferManager new];
self.tm.s3 = s3;
self.tm.operationQueue.maxConcurrentOperationCount = 2;

Do not set maxConcurrentOperationCount too high. The default value of maxConcurrentOperationCount is 3, and in our test, setting it above the default may result in unreliable progress feedback on iOS 5 and above.

Putting everything together, the sample snippet will look like this:

// Note that this is not the preferred way to create the AmazonS3Client object. Do not ship apps with your credentials in them.
AmazonS3Client *s3 = [[AmazonS3Client alloc] initWithAccessKey:@"your-access-key"
                                                 withSecretKey:@"your-secret-key"];
s3.endpoint = [AmazonEndpoints s3Endpoint:US_WEST_2];
s3.maxRetries = 10;
s3.timeout = 240;

NSString *filePath = [[NSBundle mainBundle] pathForResource:@"testData"
                                                     ofType:@"txt"];

self.tm = [S3TransferManager new];
self.tm.operationQueue.maxConcurrentOperationCount = 2;
self.tm.s3 = s3;

S3PutObjectRequest *putObjectRequest = [[S3PutObjectRequest alloc] initWithKey:@"your-key"
                                                                      inBucket:@"your-bucket"];
putObjectRequest.delegate = self;
putObjectRequest.requestTag = @"your-unique-tag";
putObjectRequest.filename = filePath;

[self.tm upload:putObjectRequest];

I hope this post quickly get you started on the new S3TransferManager for iOS. If you have any questions regarding S3TransferManager, please leave a comment below!

Further reading

• S3TransferManager for iOS – Part II: Asynchronous Uploads Best Practices

We are hiring

If you like building mobile applications that use cloud services that our customers use on a daily basis, perhaps you would like to join the AWS Mobile SDK and Tools team. We are hiring Software Developers, Web Developers, and Product Managers.