Automating backups and optimizing backup costs for Amazon EFS using AWS Backup

Automating backups allows you to meet rigorous regulatory compliance regulations and business continuity goals. It also enables you to avoid time-consuming configuration of your backups on a regular basis. AWS Backup enables you to centralize and automate data protection across AWS services by scheduling backups using backup plans. With backup plans, you can configure different requirements for different resources and implement lifecycle rules to delete or transition backups to colder storage, which can help you optimize your backup costs.

In this blog post, we demonstrate configuring scheduled backups in a backup plan by modifying an existing backup rule and taking backups of Amazon EFS file systems. Amazon EFS supports transitioning backups from warm to cold storage using lifecycle rules, so we also discuss optimizing the cost of EFS file system backups using lifecycle rules. By leveraging automation for your backups, you can save time and money while ensuring that you meet all of your regulatory compliance regulations and business continuity goals.

Scheduling backup jobs

If you do not have a backup plan already, you can create one by following the steps mentioned in the “Automate and centrally manage your backups” blog post.

AWS Backup provides a scheduling feature that enables customers to to automatically create recovery points at regular intervals to back up Amazon EFS file systems, or any of the other services supported by AWS Backup. With the scheduling feature, you can configure a schedule to run backup jobs periodically at fixed times, dates, or intervals. A scheduled backup job runs automatically at a frequency you set, eliminating the need to write and run scripts to back up your AWS resources.

Using AWS Backup, you can edit an already existing backup rule or add a new rule in a backup plan to set the frequency using a custom schedule and lifecycle. The backup frequency determines how often AWS Backup creates a snapshot backup. Using the AWS Backup console, you can choose a frequency of every 12 hours, daily, weekly, or monthly. You can also create a cron expression that creates backups as frequently as hourly.

Before we review the detailed steps, let’s review how custom cron expressions and AWS Backup lifecycle work.

Custom cron expressions

Cron expressions have six required fields, which are separated by white space. The six fields represent Minutes, Hours, Day-of-month, Month, Day-of-week, and Year, respectively. Allowed values and wildcards for the fields can be found in the cron expression documentation.

Here are a few examples:

• Run at 7:00 am (UTC) every Monday through Friday:
  • 0 7 ? * MON-FRI *

• Run at 9:00 pm (UTC) every 14th day of the month:

• • 0 21 14 * ? *

You can create a new backup plan using create-backup-plan API. You can also update an existing backup plan identified by its backupPlanId with the input document in JSON format using the AWS CLI. Provide the following input into the update-backup-plan CLI command.

Updating an AWS Backup backup plan using the AWS CLI

To update your AWS Backup backup plan using the AWS CLI, use the following command:

  update-backup-plan
--backup-plan-id <value>
--backup-plan <value>
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]

Here are few examples of custom cron expressions using AWS CLI commands:

• Run at 3:45 pm (UTC) between day 15 and 21 of the month:

aws backup update-backup-plan --backup-plan-id "dcfXXXXX-bXXX-4XXX-8XXX-62XXXXXXXXXX" \
--backup-plan "{\"BackupPlanName\":\"Example-Backup-Plan\",\"Rules\":[{\"RuleName\":\"
myrule1\",\"ScheduleExpression\":\"cron(45 15 15-21 * ? *)\",\"TargetBackupVaultName\":\"
Default\",\"Lifecycle\":{\"DeleteAfterDays\":90}}]}"

• Run at 1:00 am (UTC) on every second day-of-month:

aws backup update-backup-plan --backup-plan-id "dcfXXXXX-bXXX-4XXX-8XXX-62XXXXXXXXXX" \
--backup-plan "{\"BackupPlanName\":\"Example-Backup-Plan\",\"Rules\":[{\"RuleName\":\"
myrule1\",\"ScheduleExpression\":\"cron(30 7 ? * 7#3 *)\",\"TargetBackupVaultName\":\"
Default\",\"Lifecycle\":{\"DeleteAfterDays\":90}}]}"

Clarification on symbols

The “*” (asterisk) wildcard includes all values in the field. In the Hours field, “*” would include every hour. You cannot use “*” in both the Day-of-month and Day-of-week fields. If you use the asterisk in one field, you must use “?” in the other field.

The “?” (question mark) wildcard specifies one or another. In the Day-of-month field, you could enter “7,” and if you didn’t care what day of the week the seventh was, you could enter “?” in the Day-of-week field.

How AWS Backup helps you save on costs

When you make use of AWS Backup to take backups, there is no additional charge to use the AWS Backup centralized backup features, like monitoring beyond the existing backup storage pricing charged by AWS services, such as Amazon EBS snapshot storage fees. There is no additional charge for Amazon EC2 instance backups.

For services that introduce AWS Backup-native capabilities, such as Amazon EFS, backup usage is metered and billed by AWS Backup. You can learn more about AWS Backup pricing on the AWS Backup pricing page.

Optimizing Amazon EFS backups using Lifecycle rules

There are two pricing points to consider: backup storage pricing and restore pricing. When possible, implement lifecycle rules to ensure you gain maximum benefit of cheaper storage where possible. AWS Backup provides a lifecycle feature that allows you to automatically transition your EFS recovery points from a warm storage tier to lower-cost cold storage. You can also automatically expire the backups of tall the supported resource as per the lifecycle rule.

Each lifecycle rule contains an array of transition objects specifying how long in days before a recovery point transitions to cold storage, or is deleted. As of now, the transition to cold storage is ignored for all resources except for Amazon EFS.

When you transition Amazon EFS to cold storage, you pay $0.01 per GB-Month if your backups are stored in the us-east-1 Region. When stored in warm storage, backups are charged at a rate of $0.05. The restore amount billed in a month is based on the amount of data restored for the month. The data restored in a month is measured in GB, and represents the sum of the data across all the restores performed in the month. When you restore an Amazon EFS File System Backup in warm storage, you are charged $0.02 per GB, while the restore price is $0.03 per GB if the data is restored from cold storage.

Working example

Assume that your Amazon EFS backups are stored in the us-east-1 (N. Virginia) Region. You use 400 GB of Amazon EFS warm storage backup during 15 days in April. For the other 15 days in April, you use cold storage. At the end of April, you would have the following backup storage usage, in GB-Hours:

• Total usage (GB-Hours) in warm storage = [400 GB x 15 days x (24 hours / day)] = 144000 GB-Hours = 200 GB-Month
• Total usage in (GB-Hours) in cold storage = [400 GB x 15 days x (24 hours / day)] = 144000 GB-Hours = 200 GB-Month

At the end of April, you would have the following backup storage charges:

• Total monthly storage charge in warm storage = 200 GB-Month x $0.05 = $10
• Total monthly storage charge in cold storage = 200 GB-Month x $0.01 = $2

We add up GB-Hours and convert to GB-Month to calculate monthly charges:

• Total monthly storage charge = 200 GB-Month x $0.05 + 200 GB-Month * $0.01= $12

Let’s also assume that you restored 10 backups that were 1 GB each and were stored in warm storage. At the end of April, you would have the following backup restore usage, in GB:

• Total usage 10 restores x 1 GB = 10 GB
• Total monthly restore charge = 10 GB x $0.02 = $0.20

Adding both your backup storage and backup restore charges together gives you your total monthly AWS Backup bill:

• Total monthly AWS Backup bill = $12 + $0.20 = $12.20

Backups transitioned to cold storage must be stored in cold storage for a minimum of 90 days. Therefore, in the AWS Backup console, the Retention period setting must be 90 days greater than the transition to cold storage setting, as shown in the following screenshot. This is because backups deleted before 90 days incur a pro-rated charge equal to the storage charge for the remaining days. The transition to cold storage setting cannot be changed after AWS Backup transitions a recovery point to cold storage.

In the next section, we demonstrate how to add a schedule to an existing backup plan by adding to new backup rules for a custom schedule. We’ll also demonstrate how to use lifecycle to optimize costs.

Deploying the solution

There are two ways to create a new backup plan; you can build a backup plan from scratch or you can build a new backup plan based on an existing backup plan. In this section, we go over deploying the solution, which consists of setting up an AWS Backup rule, configuring a lifecycle rule, and assigning resources.

Step 1: Setting up a backup rule with a custom cron expression and lifecycle

1. In the AWS Management Console, navigate to the AWS Backup service console under the Storage category. Make sure you are in the correct Region, as AWS Backup is a Region-specific service.

2. As we already have a backup plan created, in the navigation pane on the left, choose Backup plans.

3. Choose the backup plan in which you want to add a new backup rule, and select the Add Backup rule button. You can also modify an already existing rule in the backup plan by selecting the Edit button.

4. On the Backup rule configuration page:

• Under the Backup frequency section, we have selected Custom cron expression. We are taking backups every hour starting at 00:00 UTC by entering the cron expression cron(0 * ? * * *) in the cron expression box.
• You can also customize the backup windows as per your requirements. Backup windows consist of the time that the backup window begins and the duration of the window in hours. Backup jobs are started within this window. If you are unsure what backup window to use, you can choose to use the default backup window that AWS Backup recommends. The default backup window starts at 5 AM UTC (Coordinated Universal Time) and lasts 8 hours. It has a completion window of 7 days.
• For our blog post, we are selecting the Start within as 1 hour. This is the shortest backup window. AWS Backup will start the backup within the window (within 1 hour of the start time) and will complete within 7 days, as you can see in the following screenshot. As this does not affect the completion window, there is no risk in selecting a small start window.
• While setting up the backup rule, you must make sure that you select the correct backup vault where you want to store your backups. You can select the already existing backup vault, default vault, or you can also create a new backup vault.
• We also add a lifecycle, which helps us optimize our backups. Here, we set Transition to cold storage to one week, and expiring them after 97 days (Retention period). Note: The Transition to cold storage setting cannot be changed after a backup has been transitioned.

Step 2: Configuring cross-Region or cross-account copy with lifecycle for copied backups in your backup rule

In this step, we set up a cross-Region copy action in our backup rule. This cross-region copy action will help you to copy your backups to another Region for DR purposes.

• To generate a cross-region copy or cross-account copy, enter the destination Region and backup vault and select Add copy. If you want to create and share encrypted backups, refer to this blog.
• In this blog, we are copying the backups to another Region in the same account and specifying a lifecycle policy for Amazon EFS to transition them to cold storage after 7 days and retain them for 97 days.
• You can add tags to the recovery point created by the rule by selecting Tags added to recovery point.
• Once you have filled in all the details, as we can see in the following screenshot, click on Add Backup rule.

Once you have created the rule, you can view the details of the rule by selecting the rule on the backup plan summary page as we can see in the following screenshot. If necessary, you can also edit the rule by selecting the Edit button.

Any changes that you make to a backup plan or rule have no effect on existing backups created by the backup plan/rule. The changes apply only to backups that are created in the future.

For example, when you update the retention period in a backup rule, the retention period of backups created before you made the update remain the same. Any backups that are created by that rule going forward will reflect the updated retention period.

Step 3: Assigning resources to a backup plan

In this section, we are assigning resources to a backup plan. When you assign a resource to a backup plan, that resource is backed up automatically according to the backup plan. The backups for that resource are managed according to the backup plan. You can assign resources using tags or resource IDs. We review how to complete this task in the following steps.

1. Under the Resource assignments section, select the Assign resources button to add the resources that you want to back up. You can assign the resources that you would like to back up on the basis of the frequency.

2. Now, we have added the resource assignment based on the Amazon EFS resource ID. You can also make use of tags to back up all off the resources with a particular tag key value. If you are protecting more than 100 resources in a plan, we recommend that you use tag-based management because using tags to assign resources is a simple and scalable way to back up multiple resources. Any resources with the tags that you specify in the resource assignment are assigned to the backup plan. In order to perform backups per the schedule, select the Assign resources button to finalize your resource to back up, as per the schedule that we configured in the preceding backup rule.

Congratulations! You have now successfully configured a customized schedule for your backup job.

Cleaning up

If you were following the steps in the blog for testing purposes, ensure you delete the backup rule. To perform this action, go to the Backup plans page and choose the backup plan on which you have added or edited the backup rule that you want to delete. To delete the backup rule for your plan, choose the radio button next to the backup rule name, and then choose Delete, as shown in the following screenshot.

After following the cleaning up steps, your backup job will no longer run periodically.

Conclusion

In this blog post, we demonstrated how to use AWS Backup scheduling and lifecycle features to optimize Amazon EFS backups. This included covering different examples of creating a custom schedule using cron expressions via the AWS Backup console and the AWS CLI. You can use AWS Backup to manually set your own backup plans, in which you can specify the desired backup frequency, when to back up, how long to retain backups, and lifecycle policies. You can then assign Amazon EFS file systems, natively integrated with AWS Backup, or other AWS resources, to that backup plan.

Having a schedule and lifecycle for your backups helps you to automate and consolidate backup tasks that were previously performed for the resources, service-by-service. AWS Backup also removes the need to create custom scripts and manual processes, enabling you to automate your backups. With just a few clicks in the AWS Backup console, you can create backup policies that automate backup schedules and retention policies. These policies can help you meet your company’s backup and retention requirements with no extra cost for using an AWS managed Backup service.

Thanks for reading this post. We look forward to your feedback and questions in the comments section.