Creating compliance insights across Regions and accounts with AWS Backup Audit Manager reports

Customers use AWS Backup Audit Manager to automate continuous monitoring of backup activities such as changes to a backup plan or backup vault and generate daily reports. AWS Backup Audit Manager also provides auditing and reporting of data protection compliance across your backup estate. Previously, these compliance and backup activity reports were generated in and limited to single account, single Region or single account, cross Region format. You would have to use your own tooling to get the reports from several different accounts and multiple Regions into a central location to have a full view of compliance in your backup estate.

With AWS Backup Audit Manager support for cross account cross Region reporting, you will be able to monitor backup activities and evaluate the compliance status of backups from a central location using your AWS Organizations management account. You will no longer have to build tools to ingest backup reports from different accounts. Instead, these generated reports from AWS Backup will be delivered to the AWS Organizations management account for reviewing and analytics purposes. You will not lose the ability to store the reports in each account. You will also be able to select multiple accounts in your AWS Organizations to generate a report if the report template is created from the management account.

In this blog post, we will show how to create a cross Region, cross account report using AWS Backup console. Figure 1 architecture diagram shows at a high-level how AWS Backup Audit Manager reports are generated using the new capability.

Figure 1: Architecture diagram showing how AWS Backup Audit Manager reports are generated

NOTE: Your compliance responsibility when using AWS Backup is determined by the sensitivity of your data, your organization’s compliance objectives, and applicable laws and regulation. This blog is not meant to be a comprehensive approach to satisfying any regulation or certification, but rather highlighting a new feature which may help in simplify management of compliance.

Walkthrough deploying a sample reporting solution to analyze AWS Backup Audit Manager reports

The sample solution automates the creation of Amazon Athena tables required to analyze the AWS Backup Audit manager cross account cross Region reports using AWS CloudFormation. The CloudFormation template contains the resources required to create a reporting solution using the report files generated. After the reports are generated in Amazon S3, we will run Amazon Athena queries to analyze the data.

The CloudFormation template will create the following resources:

• Amazon EventBridge rule to get events when a new file is inserted in the reports destination bucket. This rule will send the event to the AWS Lambda function.
• AWS Lambda function that will process the event and create a new partition in the AWS Glue Table.
• AWS Glue Data Catalog and Tables, one table per type of report (Backup jobs, Copy jobs, Restore jobs, Control compliance and Resource compliance).
• Amazon Athena to run queries against the Glue Tables generated.

Prerequisites

• To create the cross-account reports, you need to be logged into the management account in an AWS Organization.
• To perform cross-account reports, you need to enable “Cross-Account Monitoring” on the Settings page of AWS Backup
• An Amazon S3 bucket that complies to your organization’s security standards with EventBridge notifications turned on. This Configuration can be found at S3 bucket > Properties > Amazon EventBridge.

Figure 2: Amazon EventBridge configuration

Step 1: Create an AWS Backup Report Plan using AWS Console

1. Log in to AWS Console, search for AWS Backup, and select AWS Backup from the drop down. In the left pane, under Backup Audit Manager, select on Reports for creating Report plans. Then, select on Create report plan.

Figure 3: AWS Backup with report plans

2. When creating a new report plan, 5 different reports types are available.

• Backup jobs report
• Restore jobs report
• Copy jobs report
• Control compliance report
• Resource compliance report

3. You are required to select a report template and report plan name as mandatory fields. Also, optional fields are included such as report plan description and adding tags.

Figure 4: AWS Backup Audit Manager Create report plan

4. There are 3 new sections included in the reports.

• Accounts: where you can select if the report plan applies only to the account where you are launching the report, or one or more accounts in my organization.
• Organizational units (OUs) & accounts: where you can select the OUs or accounts to be included in the scope of the report plan. This only show up if one or more accounts in my organization is selected.
• Regions: where you can select one or more regions to be included in the scope of the report plan.

Figure 5: Account selection in the report plan

Organizational units (OUs) & accounts: when selecting the root, every OU and account included will be on-boarded to the scope of the report plan. Each OU will be identified by the OU Id, and the accounts will be identified by the Account Id and the Account email.

Figure 6: AWS Organizations OUs or accounts selection

Regions: when selecting the regions, a drop down menu appears with checkboxes to include specific regions or to include All available regions.

Figure 7: AWS Regions selection

5. When selecting All available regions option, an optional feature will be available for Include new Regions when they are incorporated into Backup Audit Manager.

Figure 8: AWS Regions selection in the reports plan

6. In the Report delivery section, you are allowed to define the file format for the reports. For cross account cross Region, feature only CSV format is supported. You will need to select an existing Amazon S3 bucket, and include an optional prefix if needed.

You will need to edit the bucket policy in order to be able to deliver the reports to the destination bucket.

Figure 9: Report delivery section in the reports plan

7. A bucket policy will be provided to copy-paste in the bucket permissions section. As an example of the provided policy:

Figure 10: Amazon S3 Bucket policy in the report plan

Step 2: Deploy the CloudFormation template by launching the stack directly

1. Log in to the AWS Management Console and account where you want to deploy the template. Then, select the Launch Stack button.

2. From top right-hand corner of the AWS Management Console, select your Region. The default Region for sample Cloudformation template is us-east-1. Change the Region to where you want to deploy the solution.

3. Choose Next.

4. Provide a stack name and the following parameters for deployment. Select Next twice when you are done. The rest of the parameters must remain by default and not be modified.

Parameters have an input validation that follows the naming rules in AWS Documentation.

5. Read and accept the Capabilities items listed, select Create stack.

6. When you see the CREATE_COMPLETE status, that means you have successfully deployed the template.

Different use cases related to AWS Backup Audit Manager report template setup

The report templates can be configured from different account types within AWS Organizations. Depending on where the report template is setup, the scope of the reports generated can change. The differences are explained in the next sections.

Management account

• When creating a report plan from a management account, it’s possible to launch report plans including all OUs and accounts inside the OU. When using the console, you must select an Amazon S3 bucket on the management account, but using CloudFormation, you can use an Amazon S3 bucket in the management account or in other account, e.g. in Audit account. By doing this, you can centralize all the backup reports in your audit or security account.

Non-management account, part of AWS Organizations

• When creating a report plan from a non-management account, it’s not possible to launch cross account report plans including all accounts in the OU, only the account where the report plan is launched.

Non-member account, standalone account

• As these accounts are not part of an organization, it’s not possible to launch cross account report plans including other accounts, only the account where the report plan is launched.

Other points of interest

• The report will split into smaller sized files once the 50Mb threshold is reached.
• Only management account can create cross Region cross account reports.
• The new flow of cross account cross Region includes all the frameworks existing in the accounts and regions, by default.

Cross account cross Region report structure

Below are few snippets showing how exactly compliance reports look, be advised that not all fields are shown due to row sizes. The org job report will have the same columns as regular job report, except for additional columns PathToRoot and AccountId, which are available for management account’s report plan only.

More details on the AWS Backup Audit Manager reports can be found in the AWS Documentation.

Control compliance report sample

Resource compliance report sample

Backup job report sample

Considerations

• You must ensure that AWS Backup service is activated at the organization level. This setting can be verified in AWS Organizations > Services > AWS Backup > Enable Trusted Access
• Regarding file formats, the new and old format can’t be in the same folder because it will crash in the Athena query resulting in creating different tables, one table for each report. Therefore, it is recommended to create a new Amazon S3 bucket for the reports created with these new formats.
• Backup plan prefix. If a prefix is defined during the report plan creation, the parameter must be modified in the CloudFormation template.

Running queries on Amazon Athena

Here are some example queries to run against the aggregated reports data to identify things such as jobs failures across an OU or in a simple Region and non-compliant controls.

• Example queries
  • Select all non-compliant results from the control compliance report on a specific Region (in this example, the Region is ‘us-east-1’)
    • select * from control_compliance_report where "aws region" = 'us-east-1' and "control compliance status" = 'NON_COMPLIANT';

• • Select all non-compliant results from the control compliance report on a specific Region and with an specific pathtoroot (in this example, the Region is ‘us-east-1’)
    • select * from control_compliance_report where "aws region" = 'us-east-1' and "control compliance status" = 'NON_COMPLIANT' and "pathtoroot" = 'OrganizationId/RootId/OuId';

• • Select all non-compliant results from the resource compliance report where the Control Name is ‘LockVault’.
    • select * from resource_compliance_report where "Control Name" = 'LockVault' and "Resource Compliance" = 'Non-Compliance';

• • Select all backup jobs that has failed on a specific Region
    • select * from backup_job_report where “job_status” = 'FAILED' and "aws region" = 'eu-west-1'

• • Select all restore jobs that has failed on a specific Region
    • select * from restore_job_report where “job_status” = 'FAILED' and "aws region" = 'eu-west-1'

• • Select all copy jobs that has been aborted on a specificRegion
    • select * from backup_job_report where “job_status” = 'ABORTED' and "aws region" = 'eu-west-1'

Cleaning up

Step 1: Delete S3 bucket content

• In order to delete the CloudFormation stack, you must delete the content of the S3 bucket.

Step 2: Delete the CloudFormation stack

• In the AWS Console go to ‘CloudFormation > Stacks > Stack details’ on the Stack that you want to delete and select ‘Delete’.

Conclusion

In this blog post, we introduced the new cross account cross Region reporting feature for AWS Backup Audit Manager, provided an overview of why auditing backup estate is vital, and showed how AWS Backup helps maintain data protection compliance across your global organization. We, then, detailed how cross account cross Region reporting can be configured and what the report structure looks like. We have also included sample CloudFormation template to deploy the solution rapidly and a few Athena queries that you can run against the aggregated reports to analyze the data directly in Amazon S3. You can customize the sample CloudFormation template and Athena queries to meet your specific requirements.

Thanks for reading this blog post. To learn more about AWS Backup, visit the AWS Backup Developer’s Guide. If you have any questions or comments, please leave them in the comment section.