Skip to main content

AWS Solutions Library

Guidance for Automated Restore and Copy for Amazon S3 Glacier Objects

Go to sample code

Overview

This Guidance enables you to leverage Amazon S3 Batch Operations with enhanced flexibility in manifest generation. You can either use an automated manifest generator that filters objects based on criteria like prefix, suffix, and last modified date, or manually upload a CSV manifest for precise control. The Guidance presents two architectural approaches: the first demonstrates how to request restoration of archived objects using S3 Batch Operations and perform copying through Batch Operations Invoke Lambda, while the second showcases an event-driven method for copying archived objects after being restored by S3 Batch Operations. These approaches provide efficient, scalable solutions for managing large-scale Amazon S3 operations, particularly when dealing with archived content.

How it works

Overview

This architecture shows how to request restoration of archived items listed in a CSV manifest. For details on how to automate the creation of the CSV manifest, open the Automated CSV Manifest Generator tab.

Download the architecture diagram
Architecture diagram illustrating an automated restore and copy solution using Amazon S3 Glacier, S3 Batch Operations, AWS Lambda functions, DynamoDB, EventBridge, CloudFormation, and SNS for managing archival, restoration, and copy workflows in AWS Cloud.

Automated CSV Manifest Generator

This architecture shows how to automate the creation of a CSV manifest using an AWS Step Functions workflow. For an overview of the full architecture, open the Overview tab.

Download the architecture diagram
Diagram illustrating an AWS cloud workflow for S3 bucket operations, including Lambda functions, DynamoDB tables, SNS email topics, dead-letter queues, and Glacier archive processes.

Well-Architected Pillars

The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.

This Guidance can be re-deployed using the AWS CloudFormation template. The solution includes an SNS notification function that provides job status and failures.

Read the Operational Excellence whitepaper

Data stored in DynamoDB and Amazon S3 are protected by default through AWS encryption. By default, S3 buckets have access control lists (ACLs) disabled and public block access enabled.

Read the Security whitepaper

Step Functions has retry and exponential back-off enabled to retry the Lambda functions it invokes. Step Functions states in the Guidance implement a retry and back-off mechanisms. Amazon S3 Batch Operations also retries Lambda service-related errors. Additionally, AWS Software Development Kits (SDKs) used in the Lambda functions have default retry and back-off configuration.

Read the Reliability whitepaper

Amazon S3 Batch Operations is designed to manage large-scale operations. Lambda functions automatically scale to handle the number of concurrent invocations. You can enable provisioned capacity for DynamoDB which will reserve sufficient system resources to meet your requirements. 

Read the Performance Efficiency whitepaper

S3 Glacier provides multiple options for archive retrieval, including bulk retrieval, the lowest cost option that allows you to retrieve petabytes of data within 5-12 hours. S3 Glacier Flexible Retrieval provides free bulk retrieval for archived items that you’d want to retrieve infrequently, such as 1-2 times a year. Additionally, Amazon S3 Batch Operations allows you to manage billions of objects at scale without the need to provision costly and complex compute.

Read the Cost Optimization whitepaper

Amazon S3 Lifecycle rule is applied to the guidance S3 bucket to have objects expire after 180 days. The solution DynamoDB items are set to expire 60 days after restore and copy job completion. Automating expiration helps you avoid unnecessarily using storage resources for items that you no longer need.

Read the Sustainability whitepaper

Implementation resources

The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Open sample code on GitHub

Disclaimer

The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.