How can I check the integrity of an object uploaded to Amazon S3?

Last updated: 2019-09-23

I want to upload an object to an Amazon Simple Storage Service (Amazon S3) bucket. Additionally, I want to verify the integrity of the uploaded object. How can I do that? 

Short Description

Follow these steps to verify the integrity of the uploaded object using the MD5 checksum value:

Note: The entity tag (ETag) is a hash of the object that might not be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and encrypted. Because the ETag isn't always an MD5 digest, it can't always be used for verifying the integrity of uploaded files.

1.    Get the base64-encoded MD5 checksum value of the object.

2.    Verify the object's integrity during the upload.

Resolution

Important: This resolution verifies the integrity of objects using the Content-MD5 header. If your upload is signed with AWS Signature Version 4, you need to use the x-amz-content-sha256 header instead. For more information, see Does the AWS CLI validate checksums?

Get the base64-encoded MD5 checksum value of the object

If you're using a Windows operating system, follow these steps:

1.    Install the File Checksum Integrity Verifier (FCIV) utility.

2.    Run the FCIV utility with this command:

fciv.exe c:\S3\testfile

3.    The response contains the hexademical format of the checksum value, similar to the following:

fciv C:\Windows\explorer.exe
                //
                // File Checksum Integrity Verifier version 2.05.
                //
                example111aaa222bbb33cc44dd5e6f7 c:\\windows\\explorer.exe

4.    Convert the hexadecimal MD5 checksum value into its base64-encoded format. As one option for getting the base64-encoded format, see Database storage format using the FCIV utility.

If you're using a Linux operating system, run this Open SSL command:

openssl md5 -binary PATH/TO/FILE | base64

The response contains the base64-encoded MD5 checksum value, similar to the following:

user@example:/home$ openssl md5 -binary /bin/bash | base64
                examplemd5value1234567==

Verify the object's integrity during the upload

To verify the MD5 checksum value of the object during its upload to Amazon S3, use the AWS Command Line Interface (AWS CLI) command aws s3api put-object and include the --content-md5 option. For the value of --content-md5, enter the base64-encoded MD5 checksum value that you calculated, similar to the following:

aws s3api put-object --bucket awsexamplebucket --key awsexampleobject.txt --body awsexampleobjectpath --content-md5 examplemd5value1234567==

Optionally, if you want to store the MD5 checksum value as metadata (custom HTTP header), you can also add the --metadata option in the command, similar to the following:

aws s3api put-object --bucket awsexamplebucket --key awsexampleobject.txt --body awsexampleobjectpath --content-md5 examplemd5value1234567== --metadata md5checksum=examplemd5value1234567==

If the checksum that Amazon S3 calculates during the upload doesn't match the value that you entered for --content-md5, Amazon S3 won't store the object. Instead, you receive an error message in response. For more information, see Does the AWS CLI validate checksums?