AWS Storage Blog

Category: Analytics

Understand Amazon S3 data transfer costs by classifying requests with Amazon Athena

Cost is top of mind for many enterprises, and building awareness of different cost contributors is the first step toward managing costs and improving efficiency. Costs for transferring data may segregate into common but low cost and less frequent but higher cost groups. Data about these two groups is mixed together, and separating them enables […]

Managing duplicate objects in Amazon S3

When managing a large volume of data in a storage system, it is common for data duplication to happen. Data duplication in data management refers to the presence of multiple copies of the same data within your system, leading to additional storage usage as well as extra overhead when handling multiple copies of the same […]

Automatic monitoring of actions taken on objects in Amazon S3

Administrators may need to monitor and audit actions, like uploads, updates, and deletes, taken on files and other data to comply with regulations or company policies. A scalable and reliable method of tracking and saving actions taken on files can reduce manual work and operational overhead while helping to ensure compliance. An event-based fanout architectures […]

Amazon S3 Object Lambda

Automatically modify data you are querying with Amazon Athena using Amazon S3 Object Lambda

Enterprises may want to customize their data sets for different requesting applications. For example, if you run an e-commerce website, you may want to mask Personally Identifiable Information (PII) when querying your data for analytics. Although you can create and store multiple customized copies of your data, that can increase your storage cost. You can […]

Amazon S3 featured image - new

How to enforce Amazon S3 Access Grants with Immuta

Amazon Simple Storage Service (Amazon S3) is the most popular object storage platform for modern data lakes. Organizations today evolved to adopt a lake house architecture that combines the scalability and cost effectiveness of data lakes with the performance and ease-of-use of data warehouses. Likewise, Amazon S3 plays an increasingly important role as the foundational […]

Amazon S3 Archive Storage Classes

Simplify querying your archive data in Amazon S3 with Amazon Athena

Today, customers increasingly choose to store data for longer because they recognize its future value potential. Storing data longer, coupled with exponential data growth, has led to customers placing a greater emphasis on storage cost optimization and using cost-effective storage classes. However, a modern data archiving strategy not only calls for optimizing storage costs, but […]

Amazon S3 featured image - new

Getting visibility into storage usage in multi-tenant Amazon S3 buckets

SaaS providers with multi-tenant environments use cloud solutions to dynamically scale their workloads as customer demand increases. As their cloud footprint grows, having visibility into each end-customer’s storage consumption becomes important to distribute resources accordingly. An organization can use storage usage data per customer (tenant) to adjust its pricing model or better plan its budget. […]

Amazon S3 featured image - new

Consolidate and query Amazon S3 Inventory reports for Region-wide object-level visibility

Organizations around the world store billions of objects and files representing terabytes to petabytes of data. Data is often owned by different teams, departments, or business units, spanning multiple locations. As the amount of datastores, locations, and owners grow, you need a way to cost-effectively maintain visibility on important characteristics of your data, including based […]

Amazon S3 Archive Storage Classes

Identify cold objects for archiving to Amazon S3 Glacier storage classes

Update (02/13/2024): Consider Amazon S3 Lifecycle transition fees that are charged based on the total number of objects being transitioned, the destination storage class (listed on the Amazon S3 pricing page), as well as the additional metadata charges applied. You can use the S3 pricing calculator to estimate the total upfront and monthly costs by […]

AWS DataSync Featured Image 2020

Derive insights from AWS DataSync task reports using AWS Glue, Amazon Athena, and Amazon QuickSight

Update (9/22/2023): Step 6b updated to automatically detect and update the Amazon Athena table schema when crawler detects large data transfer values reported in bytes that would consume the table’s maximum integer value while storing data. As customers scale their migration of large datasets with millions of files across multiple data transfers, they are faced […]