AWS Storage Blog
Reducing petabyte-scale S3 storage costs with evidence-based lifecycle rules
Storing data you rarely read is one of the most common sources of avoidable cost at scale and one of the hardest to get right. When a dataset grows into hundreds of terabytes or petabytes, the team that owns it often can no longer say with confidence how often it is read. The safe but expensive instinct is to leave everything on the fastest, most expensive storage. The access pattern is rarely uniform: a security team might need logs available for the occasional investigation, while an analytics team queries the same data only at quarter-end. Without evidence, both end up paying for fast storage they almost never use. The hard part isn’t deciding to act. It’s knowing which data has gone cold, and where it can safely live.
Amazon S3 is built for exactly this problem. It offers a range of storage classes at different price and access latency points. For most workloads, S3 Intelligent-Tiering is the recommended default storage class. It moves objects that haven’t been accessed for 30 consecutive days to the Infrequent Access tier, saving 40%, and after 90 days to the Archive Instant Access tier, saving 68%. An object that is read moves back to the Frequent Access tier on its own, so a surprise scan or a seasonal spike does not wipe out the savings. When your workload has a predictable, evidence-backed access pattern, such as large objects that are written once and go cold at a known age, a lifecycle rule to S3 Glacier Instant Retrieval can cost even less because it avoids the per-object monitoring and automation charge. To make that determination with confidence, S3 provides S3 Storage Lens and S3 Storage Class Analysis, which replace assumptions about access patterns with measured evidence.
In this post, I show how to go beyond S3 Intelligent-Tiering when the evidence supports it. Using S3 Storage Lens and S3 Storage Class Analysis together, I walk through a repeatable method to prove that a petabyte-scale workload has a stable cold threshold, and then move that data to storage based on evidence such as object size and age. I then show how to extend that analysis toward S3 Glacier Instant Retrieval with a lifecycle rule for additional savings when the access pattern and a millisecond-latency requirement call for it. The result is a repeatable way to cut storage cost further for cold data while keeping it instantly available. This is an advanced (300-level) post that assumes working familiarity with S3 storage classes and S3 Lifecycle rules.
Start with S3 Intelligent-Tiering
S3 Intelligent-Tiering is the right starting point when you don’t yet know how your data is accessed. It monitors each object and moves it between the Frequent Access, Infrequent Access, and Archive Instant Access tiers for you. You don’t need to choose a transition age or plan for retrieval fees. This post covers the next step. Once you have evidence that a dataset goes cold at a predictable age, a lifecycle rule to S3 Glacier Instant Retrieval can cost less, because you no longer pay the per-object monitoring and automation charge.
Solution overview
One number recurs throughout this method: the 128 KB object size. S3 Standard-Infrequent Access (S3 Standard-IA), S3 Glacier Instant Retrieval, and S3 Intelligent-Tiering all treat 128 KB as a threshold: smaller objects are either billed as 128 KB, excluded from auto-tiering, or omitted from S3 Storage Class Analysis. S3 Standard-IA and S3 Glacier Instant Retrieval have a 128 KB minimum billable object size, and S3 Storage Class Analysis evaluates only objects larger than 128 KB. Since September 2024, S3 Lifecycle applies a default minimum object size that prevents objects smaller than 128 KB from being transitioned to most storage classes. You can override this default with ObjectSizeGreaterThan and ObjectSizeLessThan filters in your lifecycle rule if your use case requires it. Because of this default, you’re unlikely to raise costs by accidentally transitioning small objects. Object size still affects the savings, though. If most of your data is in objects under 128 KB, less of it can move. That’s why this method starts by measuring how much data sits above 128 KB.
The method rests on two ideas:
- Use S3 Storage Lens groups to find candidate buckets and prefixes. Break down stored volume and object count by object age (older than 30, 60, and 90 days) and by object size (larger than 128 KB) so you target the data a transition would move.
- Use S3 Storage Class Analysis to prove, by object age, when the data goes cold, observing long enough to catch seasonal access.
Treat the S3 Storage Class Analysis recommendation as an access-pattern data point, not the final storage-class decision. The analysis confirms when data goes cold. The class that best fits the real latency and cost requirements is a separate choice.
In a recent engagement, a customer stored roughly 5.8 PB of observability and search-index logs across two buckets, entirely in S3 Standard with no lifecycle rule, costing approximately $241,000 per month. Applying the following method reduced that bill by about 70 percent while preserving millisecond access. This is one example. Results vary with each customer’s storage footprint, object sizes, retention needs, and access patterns, so the roughly 70 percent reduction reflects this specific workload.
When this method delivers the biggest savings
This method fits a recognizable workload shape rather than a specific setup, and the savings are largest when all of the following are true:
- A large S3 footprint with hundreds of terabytes to multiple petabytes held predominantly in S3 Standard with no lifecycle rules.
- Write-once, rarely read data with a finite retention window. In the engagement, this was observability and search-index logs across two buckets, retained for about five years.
- A retained requirement for millisecond access even after the data goes cold for occasional queries, investigations, or compliance, which rules out asynchronous archive classes.
- A footprint made up mostly of objects larger than 128 KB, for the reasons covered in the Solution overview.
Find candidates and size the data with S3 Storage Lens
Before changing anything, start by finding where the opportunity is and understanding the shape of the data. S3 Storage Lens gives you an organization-wide view of your storage, so you can spot the buckets worth acting on and rule out the ones where the economics wouldn’t work.
Lead with S3 Storage Lens groups. An S3 Storage Lens group aggregates storage metrics using object metadata filters like prefix, suffix, object tag, object size, and object age so instead of a single bucket-wide number, you get object count and stored volume broken down by the dimensions that drive this decision.
Use groups to find candidate buckets and prefixes. Start from large buckets with no lifecycle rule (hundreds of terabytes to tens of petabytes) and attach groups that split each bucket two ways:
- By object size: objects larger than 128 KB versus smaller objects. As covered in the Solution overview, objects under 128 KB generally stay in S3 Standard. A bucket made up mostly of small objects has less data to move, so the savings are smaller. To analyze more than 50 prefixes, view object size distribution per prefix in the S3 Storage Lens advanced metrics instead, because a single S3 Storage Lens configuration supports up to 50 groups.
- By object age: stored volume and object count older than 30, 60, and 90 days. A strong candidate is a bucket where a large share of the volume sits in the older age bands and in objects larger than 128 KB. That is the data a lifecycle transition would actually move.
The following image shows an example of S3 Storage Lens groups with objects distributed by age and size. You can configure your own groups for your workload and needs. For how to set up and get the most from S3 Storage Lens groups, see the AWS Storage Blog post Gain workload-specific storage insights with S3 Storage Lens groups.
With candidate buckets and prefixes sized this way, enable S3 Storage Class Analysis on them (filtered by prefix where it helps) to answer the question the groups can’t: within those older age bands, is the data still being read?
Prove the access pattern with S3 Storage Class Analysis
Configure S3 Storage Class Analysis on a candidate bucket or prefix. Since S3 Storage Class Analysis can be scoped to a prefix, apply it at the prefix level when different prefixes within the same bucket have different access patterns. It groups objects by age and reports how much data is retrieved per age group. Let evidence, not assumption, set the transition age.
Give it time. The console shows first results in 24 to 48 hours, but resist acting on that early snapshot. S3 Storage Class Analysis observes access for 30 days or longer. I recommend longer still. In the engagement, I observed access for more than 45 days to capture seasonal or periodic reads that a shorter window would miss. This matters because retrieving data from an infrequent-access or archive class costs more per request and per GB than from S3 Standard. A seasonal read spike that arrives after you transition the data can erode the savings. In the engagement, objects older than 90 days showed an access ratio below 0.1 percent, a clear data-backed cold threshold.
It’s this boundary that keeps the “hot” (frequently accessed) layer protected. An S3 Lifecycle transition operates based on the object’s age (the number of days since its creation). A rule defined at the “cold” threshold moves only objects that exceed this milestone, keeping the most recent and still active objects in the S3 Standard class.
In the following example, this bucket has 17.6 GB of data in the 365–729 day age band with 647.3 MB of retrieval, an access ratio of about 3.6 percent. This makes it a good candidate to transition to a colder storage class.
Figure 2: S3 Storage Class Analysis retrieval by object age band. The numbers in this screenshot are illustrative, meant to show how to read the chart and compute the access ratio. In the engagement described earlier, the cold threshold was below 0.1% retrieval past 90 days.
Extend the analysis beyond the Standard-IA recommendation
S3 Storage Class Analysis focuses on the S3 Standard to S3 Standard-IA transition, which makes it an excellent first signal that your data has cooled and is ready to move to a lower-cost class. For data that is effectively never read but must stay instantly available, you can take that signal one step further: the same evidence that justifies a move to S3 Standard-IA can support a move to an archive class with millisecond retrieval, capturing additional savings. Use the recommendation as a confident starting point, and then match the final storage class to the real access and latency requirement.
Move cold data to S3 Glacier Instant Retrieval
For data with near-zero retrieval that still requires millisecond access, S3 Glacier Instant Retrieval is the better-fit class. It delivers the same millisecond latency and throughput as S3 Standard and S3 Standard-IA. For data accessed about once per quarter, it can cost up to 68% less than S3 Standard-IA. Confirm the workload fits its constraints before committing: a 90-day minimum storage duration, a 128 KB minimum billable object size, and per-GB retrieval plus higher per-request charges (see S3 pricing for current minimum storage duration and per-Region retrieval rates).
Set the transition age from the evidence
Choose the transition age conservatively. In the engagement, comparing a 60-day threshold against 90 days led to a 90-day rule because the marginal savings from the earlier transition didn’t justify the retrieval-fee risk if a brief access spike hit recently cooled data. Implement this as an S3 Lifecycle rule that transitions objects from S3 Standard to S3 Glacier Instant Retrieval at the chosen age. Budget for the one-time transition request charges: S3 Lifecycle charges per transition request (see S3 pricing for current rates). At petabyte scale with billions of objects, this is a meaningful upfront cost. Plan for it and consider staggering the rollout by prefix if needed.
Add a guardrail against retrieval-fee blowback
S3 Glacier Instant Retrieval charges per-GB retrieval fees and higher per-request costs than S3 Standard. At petabyte scale, a single full-prefix scan or a large recurring batch read can generate retrieval charges that erode the storage savings. (This is the exact risk that S3 Intelligent-Tiering eliminates. If you cannot enforce the guardrails below, consider staying with S3 Intelligent-Tiering instead.) To prevent this, restrict query engines and batch jobs from scanning archive-class prefixes directly. For example, configure your query layer to route full-table scans to a separate copy or summary in S3 Standard. Set S3 request metrics alarms on the archive prefixes so that unexpected retrieval spikes are caught early. Pair the lifecycle rule with this guardrail rather than shipping the transition on its own.
Add an expiration rule for data with a known retention window
If the workload has a finite retention period, five years in the engagement described here, add an expiration action to the same lifecycle rule. Expiring objects at the end of their retention window is easy, evidence-based savings that the transition alone does not capture. Without an expiration rule, data accumulates indefinitely in S3 Glacier Instant Retrieval, and you continue to pay storage costs for data you no longer need. A lifecycle configuration can combine a transition action and an expiration action in the same rule:
{
"Rules": [
{
"ID": "transition-and-expire-cold-logs",
"Status": "Enabled",
"Filter": {
"Prefix": "logs/"
},
"Transitions": [
{
"Days": 90,
"StorageClass": "GLACIER_IR"
}
],
"Expiration": {
"Days": 1825
}
}
]
}
Conclusion
For most workloads, S3 Intelligent-Tiering is the recommended starting point: it delivers automatic savings at the Infrequent Access and Archive Instant Access tiers with no retrieval fees and no need to choose a transition age. When your data shows a clearer pattern, such as large objects that are written once and go cold at a stable age, you can go further. In this post I showed how to use S3 Storage Lens and S3 Storage Class Analysis together to prove that pattern with evidence, and then move the data to S3 Glacier Instant Retrieval with a lifecycle rule for additional savings. In one customer environment, this reduced S3 storage cost by about 70 percent while preserving millisecond access.
To learn more, review the S3 storage classes overview and the documentation on managing your storage lifecycle, and explore S3 Storage Lens and S3 Storage Class Analysis in the S3 console. For help applying this to your own workloads, contact your AWS account team.
