Dustin shows you how to
improve the performance
of your S3 bucket


When my Amazon S3 workload is very high, S3 performance does not scale with the workload. When this occurs I occasionally receive HTTP 500 or 503 errors. How can I optimize the performance of my Amazon S3 bucket for the following workloads?

  1. A mix of request types (GET, PUT, DELETE, or GET bucket) that exceed 100 requests per second.
  2. Primarily GET requests that exceed 300 requests per second.

Amazon S3 maintains an index of object key names in each AWS Region. Object keys are stored in UTF-8 binary ordering across multiple partitions in the index. The key name determines which partition the key is stored in. Using a sequential prefix, such as timestamp or an alphabetical sequence, increases the likelihood that Amazon S3 will target a specific partition for a large number of your keys, potentially overwhelming the I/O capacity of the partition.

When your workload is a mix of request types, introduce some randomness to key names by adding a hash string as a prefix to the key name. By introducing randomness to your key names the I/O load will be distributed across multiple index partitions. For example, you can compute an MD5 hash of the character sequence that you plan to assign as the key and add 3 or 4 characters from the hash as a prefix to the key name. The following example shows key names with a 4 character hexadecimal hash added as a prefix:


Without the 4 character hash prefix, S3 may distribute all of this load to 1 or 2 index partitions since the name of each object begins with examplebucket/2013-26-05-15-00-0 and all objects in the index are stored in alpha-numeric order. The 4 character hash prefix ensures that the load is spread across multiple index partitions.

When your workload is sending mostly GET requests, you can add randomness to key names. In addition, you can integrate Amazon CloudFront with Amazon S3 to distribute content to your users with low latency and a high data transfer rate.

For more information about introducing randomness to object key names and integrating Amazon CloudFront with Amazon S3 see Request Rate and Performance Considerations.

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2017-09-25