As you grow from storing hundreds to millions or billions of objects, and from terabytes to petabytes of data on Amazon S3, it becomes increasingly valuable to gain visibility into your storage usage. Amazon S3 offers a number of features to help you better understand, analyze, and optimize your storage at scale.
Amazon S3 Storage Lens
S3 Storage Lens delivers organization-wide visibility into object storage usage, activity trends, and makes actionable recommendations to optimize costs and apply data protection best practices. S3 Storage Lens is the first cloud storage analytics solution to provide a single view of object storage usage and activity across hundreds, or even thousands, of accounts in an organization, with drill-downs to generate insights at multiple aggregation levels. Drawing from more than 16 years of experience helping customers optimize their storage, S3 Storage Lens analyzes organization-wide metrics to deliver contextual recommendations that make it easy to take immediate action.
Amazon S3 is used to store large shared datasets across tens to hundreds of accounts and buckets, multiple Regions, and thousands of prefixes. With S3 Storage Lens, you can easily understand your storage and analyze it to detect outliers. S3 Storage Lens delivers more than 60 metrics (free metrics and advanced metrics) on S3 storage usage and activity to an interactive dashboard in the S3 Console. Free metrics are provided to all customers at no charge, while advanced metrics can be enabled for a monthly per object monitoring charge to receive additional metrics, prefix-level insights, extended data retention, recommendations, and the option to publish metrics to Amazon CloudWatch.
How S3 Storage Lens works
All Amazon S3 customers can access an interactive S3 Storage Lens dashboard in the S3 Console containing pre-configured views to visualize storage usage and activity trends. With S3 Storage Lens free metrics, you receive 28 metrics across various categories at the bucket level, and 14 days of historical data in the dashboard. You can filter your dashboard by summary, cost optimization, data protection, access management, performance, or events to analyze specific metrics for your intended use cases. In addition to the dashboard in the S3 console, you can export metrics in CSV or Parquet format to an S3 bucket of your choice for further use. To learn more about S3 Storage Lens, read the documentation.
Advanced metrics for S3 Storage Lens
By upgrading to Storage Lens advanced metrics, you receive 35 additional metrics with 15 months of historical data. The advanced metrics provide insights related to activity (such as request counts), deeper cost optimization (such as S3 Lifecycle rule counts), additional data protection (such as S3 Replication rule counts), and detailed status codes (such as 403 authorization errors). You can also drill down to the prefix level to identify your largest and fastest growing prefixes within a bucket. Additionally, you can access S3 Storage Lens metrics through Amazon CloudWatch to create alarms, or the CloudWatch API to send metrics to integrated observability partners. Advanced insights can be used to expand Lifecycle rules usage for greater cost optimization, identify buckets with insufficient data protection policies, and improve the performance of your application workloads. For advanced metrics pricing details, visit the Amazon S3 pricing page.
S3 Storage Lens Deep Dive & Demo
S3 Storage Lens customers
Upstox is a leading Indian discount broker who provides financial education and a digital platform for making investments to more than 11 million customers.
“The path forward is to continue using Amazon S3 Storage Lens to maintain a closer pulse on our storage usage and further improve on cost efficiencies. With the S3 Storage Lens interactive dashboard, we can easily locate the S3 prefix hotspots where increases in cost happen and optimize them with the right retention policies and S3 storage class to further improve cost efficiencies.”
Indranil Chandra, Principal ML & Data Engineer - Upstox
Pinterest is a visual discovery engine more than 400 million people use each month to find inspiration for their lives.
“S3 Storage Lens will enhance our visibility into our storage usage, and help us continuously optimize our storage costs. We use the organization-wide view to see aggregated storage usage for our teams across multiple regions, and then drill-down to understand storage growth on a bucket or even prefix level.
Using S3 Storage Lens, we can then create different dashboards for different internal teams, so they self-serve to find ways to optimize storage costs by using a Lifecyle policy or transitioning data to a more cost-effective storage class.”
Yi Yin, Software Engineer - Pinterest
Snowflake delivers the Data Cloud - where organizations mobilize data with near-unlimited scale, concurrency, and performance.
"The ability to use see metrics based on prefixes will be invaluable to us. We will have better insights into usage for particular prefixes including total storage, object counts, average object sizes and break down by current and non-current versions, which can help us debug and see the trends of operations in an economical way. This wasn't previously achievable before S3 Storage Lens."
Ju-Yi Kuo, Senior Software Engineer - Snowflake
Veeva Systems Inc. is a leader in cloud solutions—including data, software, and services—for the global life sciences industry.
"S3 Storage Lens is very simple to setup and configure. We received good insights into our S3 usage, which allowed us to put into action plans to optimize costs in S3. We are now saving thousands of dollars a month by optimizing how we store objects in S3."
Dan Soble, Vice President of Technical Operations - Veeva
Amazon S3 Storage Class Analysis
S3 Storage Class Analysis enables you to monitor access patterns across objects to help you decide when to transition data to the right storage class to optimize costs. You can then use this information to configure an S3 Lifecycle policy that makes the data transfer. S3 Lifecycle policies can also be used to expire objects at the end of their lifecycles. If you observe that your access patterns are frequently changing, you can use S3 Intelligent-Tiering which automatically moves your data based on changing access patterns between 4 access tiers that include a frequent access tier, a lower-cost infrequent access tier, an archive access tier and a deep archive access tier for cost savings. Storage Class Analysis also provides daily visualizations of your storage usage on the AWS Management Console that you can export to an S3 bucket to analyze using business intelligence tools of your choice such as Amazon QuickSight. To learn more about Storage Class Analysis, visit the Storage Class Analysis documentation guide.
Amazon S3 Inventory
Amazon S3 Inventory is a feature that helps you manage your storage. S3 Inventory provides a report of your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or prefix. This report can be used to help meet business, compliance, and regulatory needs by verifying the encryption, and replication status of your objects. You can also use S3 Inventory reports to speed up business workflows and big data jobs.
S3 Inventory reports include corresponding metadata such as bucket names, key names, last modification dates, object size, storage class, replication or encryption status among other properties. S3 inventory reports can be delivered daily or weekly and can be encrypted to protect sensitive data. It can also be configured to deliver multiple reports that deliver different types of metadata that are relevant to your specific needs. It provides a list of your objects and their corresponding metadata for an S3 bucket or a shared prefix, which can be used to perform object-level analysis of your storage.
With Amazon S3 Inventory, you can audit encryption status of objects for security and compliance, track the replication status of objects to another S3 bucket, speed up business workflows and big data jobs, and identify target objects for S3 Batch Operations.
Ready to get started?
Pay only for what you use. There is no minimum fee.
Instantly get access to the AWS Free Tier and start experimenting with Amazon S3.
Get started building with Amazon S3 in the AWS Console.