Automatically archive Amazon EBS Snapshots with Amazon Data Lifecycle Manager

Customers use snapshots to capture a point-in-time copy of their data, which can be used to enable disaster recovery, migrate data across regions and accounts, and improve backup compliance. For enterprises using snapshots, it is a common requirement to keep snapshots (which are seldom accessed) for more than 90 days, for compliance purposes. However, the cost of storing these full snapshots for extended periods presents a challenge to IT budget and customers risk non-compliance by deleting snapshots due to budget pressures.

On November 29 2021, Amazon Elastic Block Store (EBS) announced the availability of Amazon EBS Snapshots Archive, a new storage tier for the long-term retention of Amazon Elastic Block Store (EBS) snapshots of EBS volumes. Today, we are excited to announce that customers can now leverage Amazon Data Lifecycle Manager – a policy-based lifecycle management solution for EBS Snapshots and EBS-backed Amazon Machines Images (AMIs) available at no additional cost – to create custom policies that automate the archival and retention of EBS Snapshots.

In a previous blog, we took you through the steps to automate Amazon EBS Snapshot management using Amazon Data Lifecycle Manager. In this post, we dive deeper into using Data Lifecycle Manager to automate the creation, archival, and deletion of Amazon EBS Snapshots. Customers that need to keep full, point-in-time snapshots for backup and compliance can now do so by creating ‘set-and-forget’ policies instead of having to manage and maintain complex custom scripts that may lead to data loss because of human errors.

Overview

In this blog, we will showcase two examples of how you can automate snapshot archival and management with Data Lifecycle Manager, followed by guidance on how you can use the EC2 Console to view the snapshots managed and archived by Data Lifecycle Manager. In the first example, we will create an age-based policy to automate snapshot creation and archival with a single policy schedule. In the second example, we will create a policy to automate snapshot creation and archival using multiple policy schedules. One schedule will keep daily/weekly snapshots in the Amazon EBS Snapshot Standard tier (standard tier), while the other moves snapshots created on a monthly cadence to the EBS Snapshots Archive tier.

In both examples, we complete the following steps:

1. Create a Data Lifecycle Manager EBS Snapshot policy with EBS volumes as the target resource type
2. Configure policy target tags, description, and IAM role
3. Create Data Lifecycle Manager policy schedule(s) to create and retain snapshots
4. Enable ‘Snapshot archiving’ to the policy schedule

Example 1: Automate EBS Snapshot creation and archival using a single policy schedule

Customers often have compliance requirements to keep full, point-in-time snapshots for an extended period of time, ranging from months to years. In some cases, they initially keep incremental snapshots in the standard tier so they can be quickly restored into volumes. After a set period, these snapshots are often moved to the archive tier for long-term retention. Rather than having to manage complex custom scripts for this workflow, customers can now create a single policy to help manage the entire process of creating, archiving and deleting snapshots.

1. On the Amazon EC2 console, choose Lifecycle Manager located in the left side panel under Elastic Block Store. Under Create new lifecycle policy, choose EBS snapshot policy from the drop-down menu, and then select Next step.

2. Select the resource type as Volume. In Target with these tags, enter the tags of the Amazon EBS volumes from which you want to create EBS Snapshots. Data Lifecycle Manager uses resource tags to identify the volumes. Then, enter a brief Description for the policy.

3. For the IAM role, you may choose to use the Default role (created as part of this policy if one does not exist), or you may choose another role. If you do not use the Default role, make sure that the role has the necessary permissions.

4. Define the Policy Schedule to create the snapshots. For each policy, you may create up to four separate schedules. In this example we will only create one schedule. Choose Age as the Retention type here and specify the standard tier expiration period. Once you enable Snapshot Archiving, the time specified here will be the retention period of snapshots in the standard tier. In this example, we are creating snapshots every year on the 1^st day of January at 9:00 UTC, and retaining it in the standard tier for 3 months.

5. Click Snapshot Archiving under Advanced Settings – optional and check the box to enable snapshots archival. You will now need to specify the retention period of snapshots in the archive tier, with a minimum of 90 days (or 3 months). When a snapshot expires in the standard tier, Data Lifecycle Manager will automatically move it to the archive tier. When a snapshot expires in the archive tier, Data Lifecycle Manager will then automatically delete it.

In this example, once the snapshot has been kept in the standard tier for 3 months, Data Lifecycle Manager will automatically move it to the archive tier for another year before the final deletion of the snapshot. As a result, the total retention time of the snapshot across both tiers is 1 year and 3 months.

6. Click Review Policy at the bottom of the page to review all policy configurations and then select Create Policy.

Example 2: Automate snapshot creation and archival using multiple policy schedules within a single policy

Customers also want to create and keep daily/weekly snapshots in the standard tier to quickly restore snapshots into volumes, while moving monthly snapshots to the archive tier as soon as they are created. They can achieve this by creating multiple schedules within a single policy, which helps reduce the number of policies that customers will have to manage.

As with Example 1, you will need to complete Steps 1-3 to create the policy, set target resource type, target resource tags, policy description, and IAM role. Given those steps are exactly the same, we will skip directly to the schedule creation part.

1. Define the first Policy Schedule to create the Amazon EBS Snapshots. Choose Age as the Retention type here and use Weekly as the Frequency. You also need to specify the standard tier expiration date. For this schedule, we will not configure any archive settings.

In this example, we are creating snapshots every Friday at 9:00 UTC, and retaining it in the standard tier for 1 month.

2. Click Add another schedule button on the top of the page to create another schedule for this policy. For the second schedule, the Retention type is automatically propagated as Age and we will use Monthly as the Frequency and set retention to zero days. This means the snapshot created by this policy schedule will be moved to the archive tier immediately after creation. Please note that if you set the same Starting at time (ie. 09:00 UTC) for multiple schedules within the same policy, then only one snapshot (per volume) will be created for all schedules, and this snapshot will be archived if a policy schedule has this enabled. If you want to keep a separate copy of the snapshot in standard tier, then simply set a different starting time (ie. 09:01 UTC) and a separate snapshot will be created by each policy schedule.

3. Click Snapshot Archiving under Advanced Settings – optional and check the box to enable snapshots archival. As with Example 1, you will now need to specify the retention period of snapshots in the archive tier, with a minimum of 90 days (or 3 months). In this example, the snapshot will be archived immediately after creation. Data Lifecycle Manager will keep it in archive tier for 90 days and then delete it. As a result, the total retention time of the snapshot is 90 days.

4. Click Review Policy at the bottom of the page to review all policy configurations and then select Create Policy.

In this example, you have created a single policy with two schedules. The first schedule creates snapshots every week and retains them in the standard tier for 1 month. The second schedule creates snapshots every month from the same targeted volumes, moves them to the archive tier immediately upon creation, and retains them in the archive tier for 90 days.

Example 3: Monitoring snapshots archived by Amazon Data Lifecycle Manager

Now that you have completed the earlier steps and your Amazon Data Lifecycle Manager policy has automated the creation of snapshots in the standard tier and copy of the snapshots to the archive tier. Next you may want to monitor the snapshots using the EC2 Console.

1. On the Amazon EC2 console, choose Snapshots located in the left side panel under Elastic Block Store. You will now see a list of snapshots in your account in that region and you can apply filters based on Tags to view only snapshots created and managed by Data Lifecycle Manager (dlm:managed = true) as well as snapshots created by specific policies (aws:dlm:lifecycle-policy-id = policy-abc)

2. You can also filter based on Volume ID (Volume ID = vol-abc) to identify all the snapshots that were created from a specific volume, including those that are managed by Data Lifecycle Manager (dlm:managed = true).

3. Finally, you can sort by Started time to identify snapshots that reside across both standard and archive tiers to help decide which snapshot to use when you need to create a volume. Keep in mind that snapshots can take up to 72 hours to restore to the standard tier, depending on the size of the snapshot. If you permanently restore an archived snapshot before the minimum archive period of 90 days, you will be billed for the remaining days in the archive tier, rounded to the nearest hour.

Cleaning up

Any snapshots created or archived as part of this example will incur future charges. To avoid those costs, delete the snapshots and also the associated Amazon Data Lifecycle Manager policies. Please note that the minimum archive period of snapshots is 90 days. If you delete or permanently restore an archived snapshot before the minimum archive period of 90 days, you will be billed for the remaining days in the archive tier, rounded to the nearest hour.

Conclusion

In this blog, we covered how to automate the archival of Amazon EBS Snapshots created through Amazon Data Lifecycle Manager. We went through two examples of using different policy configurations and also discussed how to view and monitor snapshots using the EC2 Console. By automating the archival and the deletion of snapshots with Data Lifecycle Manager, you no longer have to worry about managing snapshots manually or through complex custom scripts.

Thank you for reading this blog! For more information on automating snapshots archival wth Amazon Data Lifecycle Manager, visit the EC2 user guide. If you have any comments or questions, please don’t hesitate to leave them in the comments section.