Downloading Large Files from Amazon S3 with the AWS SDK for iOS

In a previous post, we discussed the S3TransferManager and how it can be used for uploading larger files to Amazon S3 with the AWS SDK for iOS. But what about downloading larger files? We are currently looking into extending the S3TransferManager to also handle downloads, but in the meantime, the SDK does offer ways for handling larger files with the lower level interface.

Streaming to a file

The most straightforward method to download larger files with the AWS SDK for iOS is to use the outputStream property of the S3GetObjectRequest. Setting this property to an already opened stream will prevent the SDK from caching the entire file in memory.

-(void)downloadSync 
{
    // create our stream
    NSOutputStream *outputStream = 
        [[NSOutputStream alloc] initToFileAtPath:FILE_PATH append:NO];
    [outputStream open];
    
    // create our request
    S3GetObjectRequest *getObjectRequest = 
        [S3GetObjectRequest alloc] initWithKey:FILE_NAME 
                                    withBucket:BUCKET_NAME];
    getObjectRequest.outputStream = outputStream;

    // start synchronous request
    [self.s3 getObject:getObjectRequest];

    // always make sure to close your streams when done
    [outputStream close];
}

This example code runs synchronously (see our previous post about synchronous vs. asynchronous requests). If you choose to also assign an AmazonServiceRequestDelegate to track progress of the download, be aware that the data is passed to the request:didReceiveData: method and should not be retained or appended to a buffer if you want to avoid an out of memory exception. Additionally, we will need to close our stream in the delegate callbacks on failure or success instead of the code block that initiated the request.

-(void)downloadAsync 
{
    // create our stream
    self.outputStream = 
       [[NSOutputStream alloc] initToFileAtPath:FILE_PATH 
                                         append:NO];
    [self.outputStream open];
    
    // create our request
    S3GetObjectRequest *getObjectRequest = 
         [[S3GetObjectRequest alloc] initWithKey:FILE_NAME 
                                      withBucket:BUCKET_NAME];
    getObjectRequest.outputStream = self.outputStream;
    getObjectRequest.delegate = self;

    // start asynchronous request
    [self.s3 getObject:getObjectRequest];
}

#pragma mark AmazonServiceRequestDelegate methods 

-(void)request:(AmazonServiceRequest*)request didReceiveData:(NSData*)data 
{
    // update our progress, but don't keep data around!
    self.totalTransfered += [data length];
}

-(void)request:(AmazonServiceRequest *)request didCompleteWithResponse:(AmazonServiceResponse *)response 
{
    // completed successfully, close our stream
    [self.outputStream close];
}

-(void)request:(AmazonServiceRequest *)request didFailWithError:(NSError *)error
{
    // did not complete, close and delete?
    [self.outputStream close];
}

So we now can get the contents of a large file from a single request and stream it to a file, but what happens if we aren’t on the most reliable connection? Thankfully, if we do timeout or otherwise fail to download the whole file, we don’t necessarily need to restart the whole request.

S3 Ranged Gets

S3GetObjectRequest supports fetching a portion of the file through the use of two properties, rangeStart and rangeEnd. These values are 0-indexed and inclusive, meaning that if we have a file that is 2000 bytes, the valid range values would be 0–1999. In order to make use of ranged gets, our app will need to know the size of the file before we start the download. We can easily get this value by using the S3GetObjectMetadataRequest object, which among other things will return a response with the file’s contentLength.

-(void) getFileSize 
{
    S3GetObjectMetadataRequest *getMetadataRequest = 
         [[S3GetObjectMetadataRequest alloc] initWithKey:FILE_NAME 
                                              withBucket:BUCKET_NAME];
    S3GetObjectMetadataResponse *metadataResponse = 
         [self.s3 getObjectMetadata:getMetadataRequest];
    self.fileSize = metadataResponse.contentLength;
}

Now that we know how large our file is, we can update our delegate code slightly to attempt to redownload the missing portion of our file.

#pragma mark AmazonServiceRequestDelegate methods 

-(void)request: (AmazonServiceRequest*)request  didReceiveData:(NSData*)data 
{
    // update our progress, but don't keep data around!
    self.totalTransfered += [data length];
}

-(void)request:(AmazonServiceRequest *)request didCompleteWithResponse:(AmazonServiceResponse *)response 
{
    // completed successfully, close our stream
    [self.outputStream close];
}

-(void)request:(AmazonServiceRequest *)request didFailWithError:(NSError *)error 
{
    // did not complete, start a new request
    S3GetObjectRequest *getObjectRequest  = 
        [[S3GetObjectRequest alloc] initWithKey:FILE_NAME 
                                     withBucket:BUCKET_NAME];

    // our start range will be the amount downloaded
    // and we want to download to the last byte
    [getObjectRequest setRangeStart:self.totalTransfered 
                           rangeEnd:self.fileSize-1];
    
    // reuse output stream and continue to use the same delegate
    getObjectRequest.outputStream = self.outputStream;
    getObjectRequest.delegate = self;

    // resume the download where we left off
    [self.s3 getObject:getObjectRequest];
}

The previous code is not quite complete; you will likely want to add a maximum number of retries and also break out of the retry loop if you detect there is no network connection at all. Additionally, you may want to verify the integrity of the download by calculating the MD5 checksum and comparing it to the value in S3. Hopefully, it gives you the building blocks you need to allow your app to download larger files, even on slower or unreliable connections. We are eager for feedback, and we want to know what other challenges developers face when building cloud-backed mobile apps. Please feel free to leave a comment below, or visit our forums to post feedback and questions.

We’re hiring

If you like building mobile applications that use cloud services that our customers use on a daily basis, perhaps you would like to join the AWS Mobile SDK and Tools team. We are hiring Software Developers, Web Developers, and Product Managers.