Why does an upload fail when I use the Amazon S3 console to upload a large file?

2 minute read
0

I want to use the Amazon Simple Storage Service (Amazon S3) console to upload a file that’s 1 GB or larger. However, the upload continually fails, and I get timeout errors.

Resolution

For large files, Amazon S3 might separate the file into multiple uploads to maximize the upload speed. During these uploads, the Amazon S3 console might time out because the session times out. Instead of using the Amazon S3 console to upload the file, use the AWS Command Line Interface (AWS CLI) or an AWS SDK.

Note: If you use the Amazon S3 console, then the maximum file size for uploads is 160 GB. To upload a file that's larger than 160 GB, use the AWS CLI, an AWS SDK, or the Amazon S3 REST API.

AWS CLI

Install the AWS CLI, and configure it with the credentials of an AWS Identity and Access Management (IAM) user or role. The IAM user or role must have the correct permissions to access Amazon S3.

Important: If you receive errors when running AWS CLI commands, make sure that you're using the most recent AWS CLI version.

To upload a large file, run the cp command:

aws s3 cp cat.png s3://docexamplebucket

Note: The file must be in the same directory that you're running the command from.

When you run a high-level aws s3 command such as aws s3 cp, Amazon S3 automatically performs a multipart upload for large objects. In a multipart upload, a large file splits into multiple parts that upload separately to Amazon S3. After all the parts upload, Amazon S3 combines the parts into a single file. A multipart upload can result in faster uploads and lower chances of failure with large files.

For more information on multipart uploads, see How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?

AWS SDK

For a programmable approach to uploading large files, use an AWS SDK, such as the AWS SDK for Java. For example operations, see Upload an object to an Amazon S3 bucket using an AWS SDK.

Note: For a full list of AWS SDKs and programming toolkits for developing and managing applications, see Tools to build on AWS.

Related information

Use Amazon S3 with the AWS CLI

AWS OFFICIAL
AWS OFFICIALUpdated 8 months ago