Why is there a discrepancy in storage and size metrics between the Amazon S3 console and Amazon CloudWatch?

Last updated: 2021-12-03

I'm seeing a discrepancy between the "Calculate total size" number in the Amazon Simple Storage Service (Amazon S3) console and Amazon CloudWatch daily storage metrics. Why is there a discrepancy in the reported metrics between the two sources?

Short description

If there's a discrepancy between your CloudWatch storage metrics and Calculate total size number in the Amazon S3 console, then check whether the following is true:

  • Object versioning is enabled.
    Note: The object versioning feature in Amazon S3 retains multiple versions of an object in your bucket. By default, Amazon S3 object versioning is disabled on buckets, and you must explicitly enable the feature.
  • Incomplete multipart uploads exist.
    Note: Incomplete multipart uploads aren't visible in the Amazon S3 console, but are calculated as part of the storage in the CloudWatch metrics.

To identify the cause of the reporting discrepancy, check whether you enabled object versioning, and look for any multipart uploads in your bucket. These two factors can result in an increased value of the calculated bucket size in CloudWatch. For more information, see Amazon S3 CloudWatch daily storage metrics for buckets.

Tip: If you have incomplete multipart uploads in Amazon S3, then consider creating a lifecycle configuration rule. This lifecycle configuration rule can automatically clean up any incomplete parts, lowering the cost of data storage. Note that lifecycle rules operate asynchronously, so there might be a delay with the operation. However, as soon as the objects are marked for deletion, you are no longer billed for storage (even if the object isn't removed yet).

Additionally, the Amazon S3 monitoring metrics are recorded once per day, and therefore might not display the most updated information. Meanwhile, CloudWatch monitors your AWS resources and applications in real time.

Resolution

Daily storage metrics in CloudWatch

In CloudWatch, the BucketSizeBytes metric captures all Amazon S3 and Amazon S3 Glacier storage types, object versions, and any incomplete multipart uploads. This value is calculated by summing up all object sizes, metadata in your bucket (both current and non-current objects) and any incomplete multipart upload sizes. For example, the BucketSizeBytes metric calculates the amount of data (in bytes) that are stored in an Amazon S3 bucket in all these object storage classes:

  • S3 Standard
  • S3 Intelligent-Tiering
  • S3 Standard-IA
  • S3 One Zone-IA
  • S3 Reduced Redundancy Storage
  • S3 Glacier Deep Archive
  • S3 Glacier

Additionally, the NumberOfObjects metric in CloudWatch contains the total number of objects that are stored in a bucket for all storage classes. This value counts all objects in the bucket (both current and non-current), along with the total number of parts for any incomplete multipart uploads The NumberOfObjects metric also calculates the total number of objects for all versions of objects in your bucket. For example, if you have two versions of the same object, then the two versions are counted as two separate objects. For more information, see Metrics and dimensions.

"Calculate total size" in Amazon S3 console

To calculate the size of your bucket from the Amazon S3 console, you can use the Calculate total size action. Amazon S3 then calculates your bucket's storage size. However, note that multipart uploads, and previous or non-current versions aren't calculated in the total bucket size. Amazon S3 calculate only the total number of objects for the current or newest version of each object that is stored in the bucket. For example, if there are two versions of an object in your bucket, then Amazon S3's storage calculator counts them as only one object. As a result, the number that is calculated in the Amazon S3 console is smaller than the one reported by CloudWatch.

Incomplete multipart uploads

aws s3api list-multipart-uploads --bucket <bucket-example>

Then, list all the objects in the multipart upload, using the list-parts command and your UploadId value:

aws s3api list-parts --bucket <bucket-example> --key large_test_file --upload-id <examplevalue>

Creating a lifecycle rule

To automatically delete multipart uploads, you can create a lifecycle configuration rule. Follow these steps:

1.    Open the Amazon S3 console.

2.    Choose the Management tab.

3.    Choose Create new policy.

4.    Add the name of the policy.

5.    Choose Select - Delete expired delete markers or incomplete multipart uploads.

6.    (Optional) If your bucket isn't versioned, then choose Delete incomplete multipart uploads.

Object versioning

To review and audit your Amazon S3 bucket for different versions of objects, use the Amazon S3 inventory list. An Amazon S3 inventory list file contains a list of the objects in the source bucket and metadata for each object. The inventory list file capture metadatas such as bucket name, object size, storage class, and version ID.


Did this article help?


Do you need billing or technical support?