New – Range Retrieval for Amazon Glacier
Update (October 2019) – The range retrieval function described in this post is part of Amazon S3 Glacier’s
InitiateJob function. You cannot use the S3 API to initiate a range retrieval on an object that has the Glacier storage class.
Amazon Glacier is designed for storing data that is infrequently accessed. Once you have stored your data, you can retrieve up to 5% of it (prorated daily) each month at no charge.
Today we are making it easier for you to remain within the 5% retrieval band by introducing Range Retrievals. You can use this new feature to fetch only data you need from a larger file or to spread the retrieval of a large archive over a longer period of time.
Glacier’s existing archive retrieval function now accepts an optional RetrievalByteRange parameter. If you don’t provide this header, Glacier will retrieve the entire archive.
If you choose to provide this parameter, it must be in the form StartByte-EndByte. The value provided for StartByte must be megabyte aligned (a multiple of 1,048,576). The value provided for EndByte + 1 must be megabyte aligned if you are retrieving data from somewhere within the archive. If you want to retrieve data from StartByte up to the end of the archive, simply specify a value that is one less than the archive size.
When you upload data to Glacier, you must also compute and supply a tree hash. Glacier checks the hash against the data to ensure that it has not been altered en route. A tree hash is generated by computing a hash for each megabyte-sized segment of the data, and then combining the hashes in tree fashion to represent ever-growing adjacent segments of the data.
If you would like to use tree hashes to confirm the integrity of the data that you download from Glacier (and you definitely should), then the range that you specify must also be tree-hash aligned. In other words, a tree hash must exist (at some level of the tree of hashes) for the exact range of bytes retrieved. If you specify such a range, Glacier will provide you with the corresponding tree hash when the retrieval job completes.
For More Information
Here are some quick links that you can use to learn more about Range Retrievals in Glacier: