When my Amazon Simple Storage Service (Amazon S3) workload is very high (exceeding the request rates described in Request Rate and Performance Guidelines), Amazon S3 performance does not scale with the workload. When this occurs, I occasionally receive HTTP 500 or 503 errors. How can I optimize the performance of my Amazon S3 bucket to handle a large volume of requests?

Important: This information applies only to workloads that exceed the request rates described in Request Rate and Performance Guidelines. The following resolution is not required for smaller workloads.

Amazon S3 maintains an index of object key names in each AWS Region. Object keys are stored in UTF-8 binary ordering across multiple partitions in the index. The key name determines which partition the key is stored in. Using a sequential prefix, such as a timestamp or an alphabetical sequence, increases the likelihood that Amazon S3 will target a specific partition for a large number of your keys, which can overwhelm the I/O capacity of the partition.

If your workload is a mix of request types, introduce some randomness to key names by adding a hash string as a prefix to the key name. By introducing randomness to your key names, the I/O load is distributed across multiple index partitions. For example, you can compute an MD5 hash of the character sequence that you plan to assign as the key, and add three or four characters from the hash as a prefix to the key name. The following example shows key names with a four-character hexadecimal hash added as a prefix: 

exampleawsbucket/232a-2019-14-03-15-00-00/cust1234234/photo1.jpg
exampleawsbucket/7b54-2019-14-03-15-00-00/cust3857422/photo2.jpg
exampleawsbucket/921c-2019-14-03-15-00-00/cust1248473/photo2.jpg
exampleawsbucket/ba65-2019-14-03-15-00-00/cust8474937/photo2.jpg
exampleawsbucket/8761-2019-14-03-15-00-00/cust1248473/photo3.jpg
exampleawsbucket/2e4f-2019-14-03-15-00-01/cust1248473/photo4.jpg
exampleawsbucket/9810-2019-14-03-15-00-01/cust1248473/photo5.jpg
exampleawsbucket/7e34-2019-14-03-15-00-01/cust1248473/photo6.jpg
exampleawsbucket/c34a-2019-14-03-15-00-01/cust1248473/photo7.jpg
...

Without the four-character hash prefix, Amazon S3 might distribute all of this load to one or two index partitions because the name of each object begins with exampleawsbucket/2019-14-03-15-00-0, and all objects in the index are stored in alpha-numeric order. The four-character hash prefix ensures that the load is spread across multiple index partitions.

If your workload is sending mostly GET requests, you can add randomness to key names. You can also integrate Amazon CloudFront with Amazon S3 to distribute content to your users with low latency and a high data transfer rate.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2017-09-25

Updated: 2018-07-19