DevOps automation for backup compliance in AWS using AWS Backup Audit Manager

Backup compliance in AWS includes defining and enforcing backup policies to encrypt your backups, protect them from manual deletion, prevent changes to your backup lifecycle settings, and audit and report on backup activity from a centralized console. AWS Backup Audit Manager, a feature within the AWS Backup service, provides built-in compliance controls for these areas. Furthermore, it lets you customize those controls to define your data protection policies. It’s also designed to automatically detect violations of your defined data protection policies based on those controls.

Shifting left – moving security or compliance sooner in your development process – not only speeds up the overall time to release, but also results in improved security. This requires automated integration of security (and compliance) throughout the entire development lifecycle. In this post, I provide a solution that integrates Backup Audit Manager with AWS CodePipeline, thereby enabling developers to embed automated backup controls for AWS resources in their development workflows and shift left with backup compliance in AWS.

Solution overview

This solution is deployed in a multi-account organization in AWS Organizations. The AWS developer adds/removes/updates additional backup compliance controls from a developer account. I refer to all of the other accounts where Backup Audit Manager framework controls get deployed as managed accounts.

In this post, I leverage the recently announced support for AWS CloudFormation in Backup Audit Manager to provision a Backup Audit Manager framework with multiple backup-related data protection controls. I use CodePipeline to build and deploy this framework as an AWS CloudFormation StackSet in the developer account. When a developer performs a check-in or update of Backup Audit Manager controls using standard git commands, then these updates flow via CodePipeline to the managed accounts. The solution uses the Backup Audit Manager framework stackset in the developer account as the basis for provisioning stacks into selected managed accounts across specified AWS Regions. The following figure provides the architecture for this end-to-end flow within the solution:

Prerequisites

1. Enable AWS Config in the developer account, as well as all of your managed accounts in the organization. Conduct Step 1 from the Automate configuration compliance at scale blog post to utilize AWS Systems Manager Quick Setup, so that you can accomplish it with just a few clicks from your console.
2. Integrate AWS Cloud9 local Git repository with AWS CodeCommit remote Git repository:
   1. Complete Step 1 from this AWS CodeCommit tutorial to create a CodeCommit Git repository. Provide a name for your CodeCommit Git repository (for example “BackupDevOps”). Note the branch name (master, main, etc.) that you will use to check-in source code.
   2. AWS Cloud9 is one of the easiest AWS methods for setting up a local Git repository and integrating with CodeCommit as the remote Git repository. Follow these steps to set up AWS Cloud9 and integrate with a CodeCommit repository.
3. Download the following files from the GitHub repository from this post, and upload them to the root folder of your local cloned AWS Cloud9 Git repository.
   1. aws-backupcompliance-framework.yaml. This template provisions your Backup Audit Manager framework
   2. buildspec.yml
   3. buildspec-update.yml
4. In the following files that are available for download from the solution, substitute the accountID parameter with the AWS Account ID of the shared services account. Substitute the region parameter with your shared services account AWS Region. Substitute the managedaccount and managedregion parameters each with comma-separated AWS Account IDs and comma-separated AWS Regions of the managed accounts where the solution will be deployed.
   1. buildspec.yml
   2. buildspec-update.yml
5. Use standard git commands from your AWS Cloud9 cloned repository’s root folder (where the buildspec.yml file resides) in the AWS Cloud9 console to check-in the source code from your local AWS Cloud9 repository to the remote CodeCommit Git repository:
   1. git add .
   2. git commit -m "initial commit"
   3. git push origin
6. Create an Amazon Simple Storage Service (Amazon S3) staging bucket with this naming convention: s3-backupdevops-accountid-region.

Setup

The solution is available here for download with a detailed README, and it’s installed in the following single step:

1. Launch the aws-backupcompliance-codepipeline.yml CloudFormation Template. This template provisions the CodePipeline based DevOps automation with AWS CodeCommit and AWS CodeBuild stages. It enables the build and deployment of your backup in the specified accounts and regions of your organization. Furthermore, it takes the following parameters:
   1. RepositoryName: Name of the CodeCommit repository for the backup compliance templates. Obtain this value from Step 2a of the prerequisites section.
   2. BranchName: Branch in the CodeCommit repository for the backup compliance templates. Obtain this value from Step 2a of the prerequisites section.
   3. BackupComplianceStagingBucket: Backup Audit Manager framework template staging bucket. Obtain this value from Step 6 of the prerequisites section.

Test and run

Validate DevOps

1. From the CodePipeline console, navigate to Pipelines on the left panel, select the backupcompliance-pipeline pipeline in the main console, and validate that the DevOps code pipeline gets initiated and that the CodeCommit and CodeBuild stages of the pipeline execute successfully.
2. From the AWS Config console, navigate to Rules on the left panel, and from the main console you can validate that Backup-related AWS Config rules are provisioned for various AWS Backup resources, specifically backup selection, backup vaults, backup plans, and recovery points.

Validate compliance

Let’s test our solution by triggering a sample backup compliance finding. We’ll test the evaluation of the BACKUP_RESOURCES_PROTECTED_BY_BACKUP_PLAN control that checks if AWS resources are protected by a backup plan. We will launch an Amazon Aurora MySQL cluster that isn’t protected by a backup plan. Furthermore, we’ll validate that a backup compliance related finding is created for this now, because our DevOps pipeline has provisioned a Backup Audit Manager framework in your AWS environment. Here are the steps to do this:

1. Log in to the Amazon RDS console of your managed AWS account, and follow the steps here to launch an Amazon Aurora MySQL cluster. In the Choose a database creation method option, choose Easy create, and accept all of the default settings.
2. Navigate to the AWS Backup console of your managed AWS account, and select Backup plans from the left panel. Verify that no existing backup plans are already configured in your account for your newly-provisioned Aurora database by selecting Resource assignments for each configured backup plan.
3. After a few minutes, navigate to the AWS Config console of your managed AWS account, and you should see an AWS Config rule with a prefix “AURORA-RESOURCES_PROTECTED_BY_BACKUP_PLAN-“ that has been provisioned in your environment. Select the rule, scroll down to Resources in scope, and select Noncompliant from the toggle bar. You will see your Aurora database listed there.

The following figure shows the noncompliant evaluation based on this Backup Audit Manager generated AWS Config rule for your Aurora database cluster:

Perform updates

1. Let’s remediate the compliance finding by protecting our Aurora MySQL cluster with a Backup Plan. Follow the steps here to create a backup plan from the console and assign the Aurora MySQL cluster to it.
2. In the backup rule of your backup plan, set the Backup frequency to monthly and the Retention period to eight days.
3. In the root folder of your cloned AWS Cloud9 Git repository, make a copy of your current buildspec.yml and aws-backupcompliance-framework.yml files.
4. Rename buildspec-update.yml to buildspec.yml. Download the aws-backupcompliance-framework-v1.yaml file, rename it as aws-backupcompliance-framework.yaml, and upload it to the root folder of your cloned AWS Cloud9 Git repository.
5. Use standard git commands from your AWS Cloud9 cloned repository’s root folder (where the buildspec.yml file resides) in the AWS Cloud9 console:
   1. git add .
   2. git commit -m "updates"
   3. git push origin

Validate compliance after development updates

You remediated your previous finding by protecting your Aurora MySQL cluster with a Backup Plan. Let’s assume now that you have a requirement for daily backups with a retention period greater than 35 days for your Aurora MySQL cluster. To do this, you update your backup compliance CloudFormation template by adding a BACKUP_PLAN_MIN_FREQUENCY_AND_MIN_RETENTION_CHECK control, set the minimum required frequency (24 hrs) and minimum retention period (35 days) in the template and check in this template in your development workflow. This was done in Steps 4-5 of the Perform Updates section. However, based on Step 2 of the Perform Updates section, the backup rule of your current backup plan protects the Aurora MySQL cluster only with a Backup frequency of monthly and a Retention period that’s set to eight days. Therefore, this triggers a backup compliance finding again as described below:

1. Navigate to the AWS Config console of your managed AWS account, and you should see an AWS Config rule with the prefix “BACKUP_PLAN_MIN_FREQUENCY_AND_MIN_RETENTION_CHECK“ that has been provisioned in your environment. Select the rule, scroll down to Resources in scope, and select Noncompliant from the toggle bar. You will see your backup plan associated with the Aurora MySQL cluster resource there.

Cleanup

To avoid recurring charges, and to clean up your account after trying the solution outlined in this post, perform the following:

1. Delete the stacks in the following sequence for the two templates:
   1. aws-backupcompliance-codepipeline.yml
   2. aws-backupcompliance-framework.yml
2. Empty and delete the s3-backupdevops-accountid-region Amazon S3 Bucket that was created for this solution.

Conclusion

In this post, I have provided a solution that integrates Backup Audit Manager with CodePipeline. The solution enables developers to embed automated backup controls for AWS resources in their development workflows, as well as shift left with backup compliance in AWS.

About the Author