For Amazon S3 request rates, what's the difference between prefixes and nested folders? How many prefixes can I have in an S3 bucket?

3 minute read
0

I want to understand prefixes and nested folders for Amazon Simple Storage Service (Amazon S3) request rates.

Resolution

Prefixes

A key prefix is a string of characters that can be the complete path in front of the object name. This includes the bucket name. For example, if an object (123.txt) is stored as BucketName/Project/WordFiles/123.txt, then the prefix might be BucketName/Project/WordFiles/123.txt. The prefix can be any length, such as the entire object key name.

If the 123.txt file is in a bucket without a specified path, then Amazon S3 automatically adjusts the prefix value according to the request rate. Partitions can automatically form at any point in the prefix string.

A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second. There's no limit to the number of prefixes that you can have in a bucket. However, a spike in the request rate might cause throttling.

Note: In Amazon S3, there are no partitions for keys or objects. Partitions exist only at the prefix level, and not at the object level. For more information about how to use prefixes in Amazon S3, see Organizing objects using prefixes.

Folders

When you create a folder in Amazon S3, the service creates a 0-byte object with a key that's set to the provided folder name.

In Amazon S3, a folder groups objects and organizes files. Unlike a traditional file system, Amazon S3 doesn't use hierarchy to organize its objects and files. The Amazon S3 console supports the folder concept only as a way to group and display objects.

A folder is the value between the two slash (/) characters. For example, you store a file as BucketName/Project/WordFiles/123.txt. In this file path, there's a folder, Project, and subfolder, WordFiles. Both Project and WordFiles are folders. If you save the 123.txt file in a bucket without a specified path, then Amazon S3 doesn't use folders to store the file.

Note: The folder structure might not indicate any partitioned prefixes that support request rates.

Difference between prefixes and folders

The difference between a prefix and a folder is the significance of the slash character. For folders, this character signifies a subfolder or object name. For prefixes, a slash is simply a character and doesn't indicate a partition placement.

Note: The folder structure applies only to the Amazon S3 console. For more information, see Organizing objects in the Amazon S3 console by using folders.

Related information

AWS re:Invent 2018: Best practices for Amazon S3 and Amazon S3 Glacier

Organizing, listing, and working with your objects

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago
6 Comments

This is still quite confusing I think.

Amazon S3 automatically adjusts the prefix value according to the request rate.

So as an end-user I have no way of impacting prefixing at all, because it's all automatic?

A partitioned prefix in a bucket can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.

And by extension it's impossible to know what the actual limit is? Because this is a lower bound that is multiplied by a prefix value that is managed by AWS and invisible to us as the end user?

For prefixes, "/" is just another character. The "/" does not indicate a partition placement.

This also implies that the prefix partitioning can happen anywhere in the key name?

albgus
replied 8 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 8 months ago

In the video on youtube, you said the prefix is "BucketName/Project/WordFiles/", but in this post, you said the prefix is “BucketName/Project/WordFiles/123.txt”. Which is correct?

lin
replied 8 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 8 months ago

It's been 3 months and you still haven't updated the article or even properly acknowledged the questions. "Thank you, we will review, etc." is not an adequate response because it is generic. You should be answering the specific questions with specific answers if you expect folks to actually get help from this knowledge-center.

tb
replied 6 months ago

This post is quite unclear and confusing. It also points to the 2018 video which is out of date. Probably best to just delete this post and point to the current s3 performance whitepaper at: https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/introduction.html

profile picture
replied 5 months ago