Posted On: Feb 25, 2022
Amazon S3 accelerates integrity checking of requests by up to 90%. For the first time in the cloud, you can choose from four supported checksum algorithms for data integrity checking on your upload and download requests. In addition, enhancements to the AWS SDK and S3 API significantly improve checksum efficiency, increasing the performance and reducing the cost of data validation. You can automatically calculate and verify checksums as you store or retrieve data from S3, and can access the checksum information at any time using a new S3 API or an S3 Inventory report. Using checksums for data validation is a best practice for data durability, and these capabilities increase the performance and reduce the cost to do so.
Amazon S3 can automatically check the integrity of your data when you store or retrieve objects using standard S3 API calls. You can choose a SHA-1, SHA-256, CRC32, or CRC32C checksum algorithm, depending on your application needs. To verify the integrity of your data, you can either provide a precalculated checksum with your request, or have the AWS SDK automatically calculate a checksum as it streams data into S3. Calculating a checksum as you stream data into S3 saves you time as you’re able to both verify and transmit your data in a single pass, instead of as two sequential operations.
In addition, S3 can now calculate and store part-level checksum information for objects uploaded through S3 multipart upload. This means that large objects can be broken up into many parts so that checksum operations can be parallelized. S3 will independently calculate a checksum when your data is durably stored to validate that it stored the same bits that you transmitted from the client. Then, S3 will store the checksum that it calculated in the metadata for your objects. These checksums (including checksums for individual parts for an S3 multipart upload) can be accessed at any time using S3’s new GetObjectAttributes API.
These features are available at no additional cost, and can be accessed through the S3 console, Amazon S3 API, SDKs, and CLI, in all AWS Regions including the AWS GovCloud (US) Regions, the Amazon Web Services China (Beijing) Region, operated by Sinnet, and the Amazon Web Services China (Ningxia) Region, operated by NWCD. To learn more about Amazon S3 accelerated integrity checking, read the launch blog, visit the S3 features page, and view the user guide.