How can I troubleshoot 503 Slow Down errors when I transfer data between Amazon S3 buckets in different AWS Regions?
Last updated: 2020-01-15
I'm trying to perform a copy operation between two Amazon Simple Storage Service (Amazon S3) buckets in different AWS Regions. However, Amazon S3 returns a response similar to the following:
"AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBEXAMPLE2C4D)"
How can I troubleshoot this error?
You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket. When Amazon S3 returns a 503 Slow Down response, this typically indicates that the requests exceed those request rates per prefix. In some cases though, Amazon S3 can return a Slow Down response if your requests exceed the amount of bandwidth available for cross-Region copying.
To troubleshoot the Slow Down error, consider the following approaches:
- Enable an exponential backoff and retry mechanism on the application that's sending the requests.
- Use the S3DistCp operation on Amazon EMR. For more information, see Seven Tips for Using S3DistCp on Amazon EMR to Move Data Efficiently Between HDFS and Amazon S3.
Note: Because this approach requires you use Amazon EMR, be sure to review Amazon EMR pricing.
For other ways to copy data between Regions, consider the following options:
- Try a GET operation from the source bucket, and then a PUT operation to the destination bucket.
- Enable cross-Region replication (CRR) on the source bucket. CRR automatically and asynchronously copies objects to the destination bucket.
Note: After you enable CRR, new objects are automatically copied to the destination bucket. Objects that were in the source bucket before enabling CRR are not automatically copied.