How can I check the integrity of an object uploaded to Amazon S3?

Last updated: 2021-02-22

I want to upload an object to an Amazon Simple Storage Service (Amazon S3) bucket. Additionally, I want to verify the integrity of the uploaded object. How can I do that?

Short description

Follow these steps to verify the integrity of the uploaded object using the MD5 checksum value:

Note: The entity tag (ETag) is a hash of the object that might not be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and encrypted. Because the ETag isn't always an MD5 digest, it can't always be used for verifying the integrity of uploaded files.

1.    Get the base64-encoded MD5 checksum value of the object.

2.    Verify the object's integrity during the upload.

Resolution

Get the base64-encoded MD5 checksum value of the object

If you're using a Windows operating system, follow these steps:

1.    Install the File Checksum Integrity Verifier (FCIV) utility.

2.    Run the FCIV utility with this command:

fciv.exe c:\S3\testfile

3.    The response contains the hexadecimal format of the checksum value, similar to the following:

fciv C:\Windows\explorer.exe
                //
                // File Checksum Integrity Verifier version 2.05.
                //
                example111aaa222bbb33cc44dd5e6f7 c:\\windows\\explorer.exe

4.    Convert the hexadecimal MD5 checksum value into its base64-encoded format.

If you're using a Linux operating system, run this Open SSL command:

openssl md5 -binary PATH/TO/FILE | base64

The response contains the base64-encoded MD5 checksum value, similar to the following:

user@example:/home$ openssl md5 -binary /bin/bash | base64
                examplemd5value1234567==

Verify the object's integrity during the upload

To verify the MD5 checksum value of the object during its upload to Amazon S3, use the aws s3api put-object command:

aws s3api put-object --bucket awsexamplebucket --key awsexampleobject.txt --body awsexampleobjectpath --content-md5 examplemd5value1234567==

Make sure to include the --content-md5 option, and enter the base64-encoded MD5 checksum value that you calculated.

Optionally, if you want to store the MD5 checksum value as metadata (custom HTTP header), you can also use the --metadata option, like this:

aws s3api put-object --bucket awsexamplebucket --key awsexampleobject.txt --body awsexampleobjectpath --content-md5 examplemd5value1234567== --metadata md5checksum=examplemd5value1234567==

If the checksum that Amazon S3 calculates during the upload doesn't match the value that you entered for --content-md5, S3 won't store the object. Instead, you receive an error message in response. For more information, see Does the AWS CLI validate checksums?

Note: When you run your AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.