Data Transfer from Amazon S3 Glacier Vaults to Amazon S3 restores, copies, and transfers archives stored in an Amazon Simple Storage Service Glacier (Amazon S3 Glacier) vault to an S3 bucket and storage class of your choice, including the S3 Glacier storage classes. This AWS Solution simplifies the use of your data by automating the transfer process, making archived data more accessible and cost-effective.
Amazon S3 Glacier storage classes, including Glacier Deep Archive, Glacier Flexible Retrieval, and Glacier Instant Retrieval, are different from the S3 storage classes. Visit this webpage to learn more about these storage classes.
Automation saves time and minimizes the likelihood of human error during the data transfer process, helping ensure a more reliable and consistent operation.
Transferring data from Amazon S3 Glacier vaults to S3 buckets facilitates easier data analysis and utilization. Data is more readily accessible for applications and analytics tools, without extended restore times.
Amazon S3 storage classes allow for tagging and quicker access to your data. Tagging benefits include data classification, fine-grained access control, lifecycle management, and cost allocation.
For data that is rarely accessed, the Amazon S3 Glacier Deep Archive storage class can save almost 75% on storage costs in the AWS US East (Ohio) Region compared to an S3 Glacier vault.
The Step Functions Orchestrator workflow initiates a nested Step Functions Get Inventory workflow to retrieve the inventory file.
Upon completion of the inventory retrieval, the solution invokes the Initiate Retrieval nested Step Functions workflow.
The solution stores all job completion notifications in the Amazon Simple Queue Service (Amazon SQS) Notifications queue.
When an archive job is ready, the Amazon SQS Notifications queue invokes the AWS Lambda Notifications Processor function. This Lambda function prepares the initial steps for archive retrieval.
The Lambda Notifications Processor function places chunks retrieval messages in the Amazon SQS Chunks Retrieval queue for chunk processing.
The Amazon SQS Chunks Retrieval queue invokes the Lambda Chunk Retrieval function to process each chunk.
The Lambda Chunk Retrieval function downloads the chunk from the Amazon S3 Glacier vault.
After a new chunk is downloaded, the solution stores chunk metadata in Amazon DynamoDB (etag, checksum_sha_256, tree_checksum).
The Lambda Chunk Retrieval function verifies whether all chunks for that archive have been processed. If yes, it inserts an event into the Amazon SQS Validation queue to invoke the Lambda Validate function.
The Lambda Validate function performs an integrity check and then closes the Amazon S3 multipart upload.
A DynamoDB stream invokes the Lambda Metrics Processor function to update the transfer process metrics in DynamoDB.
The Step Functions Orchestrator workflow enters an async wait, pausing until the archive retrieval workflow concludes before initiating the Step Functions Cleanup workflow.
The DynamoDB stream invokes the Lambda Async Facilitator function, which unlocks asynchronous waits in Step Functions.
Monitor the transfer progress by using the CloudWatch dashboard.