How do I troubleshoot slow downloads from or uploads to Amazon S3?
Last updated: 2020-01-14
How can I troubleshoot slow downloads from or uploads to Amazon Simple Storage Service (Amazon S3)? When I download from or upload to Amazon S3 from a specific network or machine, my requests might get higher latency. How can I diagnose the high latency?
Check the following to identify and mitigate what might be contributing to slow downloads or uploads:
- The location of the client making requests
- The client internet service provider (ISP)
- The client network
- The client resources
- Request rate to Amazon S3
- Amazon S3 server access logs to determine the Turn-Around Time
The location of the client making requests
Use Amazon S3 buckets that are as geographically close to your client as possible. Buckets are globally accessible, but they reside in a certain AWS Region. The geographical distance between the request and the bucket contributes to the time it takes for a response to be received.
You can test the impact of geographical distance between the client and the S3 bucket. For example, you can launch an Amazon Elastic Compute Cloud (Amazon EC2) instance in the same AWS Region as the bucket, and then launch another instance in another Region. Test an upload and download of the same file using both instances, and then compare the throughput between the two Regions.
To decrease the distance between the client and the S3 bucket, consider moving your data into a bucket in another Region that's closer to the client. You can configure cross-Region replication so that data in the source bucket is replicated into the destination bucket in the new Region. As another option, consider migrating the client closer to the bucket.
The client internet service provider (ISP)
Test your network path to check if any packet loss, high number of hops, or any other ISP-related issue affects how your connection traverses the internet.
Tools like mtr and traceroute can help provide clues for potential packet loss, as well as the latency of the packets that are sent to the remote host. Additionally, these tools can help identify if there's a network hop causing higher latency.
For example, the following Linux traceroute command uses TCP port 80 to test connectivity to the us-west-2 (Oregon) Regional endpoint:
sudo traceroute -P TCP -p 80 s3.us-west-2.amazonaws.com
On Windows operating systems, you can use a similar tool called tracert.
Note: Many network devices don't respond to ICMP. For a test that resembles requests to Amazon S3, be sure to run a TCP traceroute or mtr to the regional endpoint of the bucket.
If the internet route from the client to Amazon S3 isn't optimal, consider using Amazon S3 Transfer Acceleration, which leverages Amazon CloudFront's edge locations. Review the Amazon S3 Transfer Acceleration Speed Comparison tool to see if Transfer Acceleration can improve performance for your use case.
Note: If you enable Transfer Acceleration, additional data transfer charges might apply. Be sure to review Amazon S3 pricing for data transfers.
The client network
Confirm that your client network is healthy, such as reviewing internal packet inspections, anti-virus scanning, or network access management. Additionally, review how your client or application handles DNS resolution and caching.
To benefit from Amazon S3's distributed nature and availability, avoid caching the DNS resolution for long periods of time.
The client resources
There might be latency introduced in your application or how your host that's making the requests is handling the requests sent and responses received. As a best practice, confirm that there's no resource contention within the host (for example, CPU, memory, or network bandwidth) that might be contributing to the overall latency.
You can check the resource usage during data transfer on most client systems using tools such as Resource Monitor or the top command, depending on your operating system.
The client's storage device or system might also be a source of latency. If read or write operations to the client's storage device experience elevated latency, this can impact the performance of the download or upload to Amazon S3. You must troubleshoot the IOPS of the storage device from the client side, but Amazon S3 performance can be analyzed using the Amazon CloudWatch metric FirstByteLatency.
Note: Amazon S3 CloudWatch request metrics are billed at the same rate as custom metrics.
FirstByteLatency shows how long it took for Amazon S3 to process the request from the client and then begin sending the response to the client. The CloudWatch metric provides a bucket-level perspective of performance. You can narrow down your investigation using Amazon S3 server access logs. For more information, see the section Amazon S3 server access logs to determine the Turn-Around Time.
Request rate to Amazon S3
By default, S3 buckets can support thousands of requests per second per prefix. If a client is getting HTTP 5xx error responses from Amazon S3, this can indicate that the supported request rate per prefix is exceeded. To troubleshoot 5xx errors, see How do I troubleshoot HTTP 5xx errors from Amazon S3?
Amazon S3 server access logs to determine the Turn-Around Time
You can enable Amazon S3 server access logging to review Turn-Around Time metrics. Turn-Around Time is the time between when the last byte of the request is received and when the first byte of the response is sent. This is a Time To First Byte (TTFB) metric. You can use Turn-Around Time to determine the amount of time the operation is taking from an Amazon S3 perspective. Then, you can evaluate other latencies outside of Amazon S3 that might be contributing to an overall slow download or upload.
If you're seeing latency that's higher than usual, it's a best practice to retry the request instead of waiting for the request to complete. For more information on this guidance and other performance recommendations, see Performance Guidelines for Amazon S3.