AWS Cloud Operations Blog

Gain visibility of AWS backup activities using Amazon Managed Grafana

AWS Backup is a comprehensive service that simplifies the process of centralizing and automating data protection across various AWS services, both in the cloud and on-premises, all managed seamlessly. Organizations have different requirements and want to track their backup, copy and restore activities across AWS cloud resources. Currently, in order to view status of resource backup operations, it requires you to have AWS Identity and Access Management (IAM) user access privileges on the AWS Management Console. The ability to centrally monitor backup activities and generate insights makes it easier for you to identify any failures while still maintaining the principle of least privilege You can visualize backup activities across multiple accounts using Amazon Managed Grafana that retrieves and refreshes the data periodically.

This post explains how Amazon Managed Grafana dashboards help you to visualize and track AWS Backup Metrics.

Architecture Overview

The following architecture diagram showcases AWS Backup Job events that are triggered to Amazon Event Bridge in real-time. This functionality enables the collection of AWS Backup Job statuses, which are then saved in Amazon Simple Storage Service (Amazon S3) buckets using Amazon Kinesis. Subsequently, AWS Glue retrieves this data from an Amazon S3 bucket, extracts the metadata, and establishes table definitions within the AWS Glue Data Catalog. After the data is stored in the database. Amazon Athena generates a structured view of the information. Finally, the Amazon Managed Grafana team utilizes the Athena data source to create a comprehensive AWS Backup report dashboard on Amazon Managed Grafana.

Figure 1. Architecture Diagram

Prerequisites:

  1. Enable and configure AWS Backup Service.
  2. Configure cross-account management feature in AWS Backup to manage and monitor your backup, restore, and copy jobs across AWS Regions and AWS accounts that you configure with AWS Organizations.
  3. Set up Amazon Athena workgroups.
  4. Set up Amazon Managed Grafana workspace. For information, and steps for creating the Amazon Managed Grafana workspace, see Creating a Workspace.
  • For user authentication and authorization, Amazon Managed Grafana can integrate with identity providers (IdPs) that support SAML 2.0 and also can integrate with AWS IAM Identity Center. Review the Amazon Managed Grafana supports direct SAML integration with identity providers.
  • To use AWS data source configuration, first use the Amazon Managed Grafana console to enable service-managed Identity and Access Management (IAM) roles that grant the workspace with IAM policies necessary to read resources in your account/Organization. Then, use the Amazon Managed Grafana workspace console to add Athena Data Source.

Step 1: Launch the AWS CloudFormation template

Download and launch the following AWS CloudFormation template to deploy Kinesis Firehose, Glue Crawler, Glue Database and its related components.

Template Link

Note: Some of the resources that this stack deploys incur costs when in use.

To create your resources using AWS CloudFormation template, complete the following steps:

  • Sign in to the AWS Management Console
  • Navigate to the AWS CloudFormation console > Create Stack > “With new resources”
  • Specify a “Stack name” and choose Next
  • Leave the “Configure stack options” at default values and choose Next
  • Review the details on the final screen and under “Capabilities” check the box for “I acknowledge that AWS CloudFormation might create IAM resources with custom names”
  • Choose Submit

Figure 2. Acknowledgement

Note: You can review the progress of your new stack under AWS CloudFormation > Stacks > [StackName] > Events tab

Once the Stack is created successfully, you will see the following resources deployed:
Amazon EventBridge Scheduler, AWS Kinesis Firehose Delivery Stream, Amazon S3 Bucket, AWS Glue Crawler, AWS Glue Database and the corresponding AWS IAM Roles and Policies are created successfully.

Step 2: Create View in Amazon Athena using the below queries:

  • Go to Amazon Athena > Query editor > Saved queries tab and choose the query name “AWS-Backup-Events”

Figure 3. Saved Athena Query

  • On the Query editor, verify the Data source, Database and Table names and replace <account-id> with your account id while running the query. Upon successful execution, the query creates a View named “grafana_view”.
    Note: backupsizeinbytes attribute would only be available for tasks with the status COMPLETED or RUNNING.

Figure 4. Run Athena Query in Query Editor

Step 3: Configure Amazon Athena Data Source in Amazon Managed Grafana

  • Launch the Amazon Managed Grafana console using the Grafana workspace URL and login using the user credentials you configured
  • Under Administration > Data sources > choose Amazon Athena
  • Configure the Amazon Athena settings by choosing Default Region (us-east-1), Data source (AWSDataCatalog), Database (aws-backup-event-records), Workgroup (primary) and the Output Location of your Athena query
  • Choose Save & test to verify that the data source is working. Start querying and visualizing the metrics from the AWS environment

Note: In case you receive a permission denied error, verify the Grafana service role permissions.

Figure 5. Amazon Athena Datasource Configuration

Figure 6. Load the JSON Code

Figure 7. Import the dashboard using JSON Code

Step 4: Create an Amazon Managed Grafana Dashboard

Amazon Managed Grafana is a fully managed service designed to simplify the process of creating, configuring, and sharing interactive dashboards and charts for monitoring your data. It offers the ability to establish alerts and notifications based on specific conditions or thresholds, enabling swift identification and response to issues.
In this next step, we will utilize Amazon Managed Grafana to generate a new AWS Backup dashboard.

  • Retrieve the AWS Backup dashboard JSON file from this GitHub Repository.
  • Import the dashboard by navigating to Dashboards > New and selecting Import in the Amazon Managed Grafana console. For additional information on exporting and importing dashboards, refer to the documentation.
    Note: You have the option to upload a dashboard JSON file, paste a dashboard URL, or directly input dashboard JSON text into the designated text area.

Figure 8. Amazon Managed Grafana Dashboard

Figure 9. Amazon Managed Grafana Dashboard

Finally, AWS Backup Report is integrated into Amazon Managed Grafana. This centralized backup console offers a consolidated view of your backups and backup activity logs, making it easier to audit your backups and ensure compliance. Furthermore, Amazon Managed Grafana’s alerting system delivers actionable alerts, enabling us to swiftly identify system issues near real time For further insights into Amazon Managed Grafana alerting, please visit the “Alerts in Grafana” section.

Note: For Cross Account or Cross Region Monitoring, by default you will not get the event in the Management account, you can only see the backup jobs that you create. In order for cross account/region you need to push the event to the target bus (Management Account). Refer Sending and receiving Amazon EventBridge events between AWS accounts.

Clean up

You will continue to incur cost until you clean up the infrastructure that you created for this post:

Conclusion

In this blog post, we introduced the AWS Backup monitoring solution using Amazon Managed Grafana. You can obtain enriched cross-account, multi-Region daily reports on your AWS Backup activities, and visualize the data using Amazon Managed Grafana dashboards. The aggregated reports and visualization dashboards can help you quickly identify and report on items and trends related to your data protection activities across your AWS accounts. You can customize the sample CloudFormation templates provided in this blog to meet your organization’s monitoring requirements, and gain the insights and visibility into your AWS Backup operations as needed.

To get started and learn more, visit Getting started with AWS Backup and Amazon Managed Grafana Dashboards. You can get hands-on experience with the AWS Observability services at One Observability Workshop. Visit the AWS Observability guide to learn more about best practices.

About the authors:

Yash Bindlish

Yash Bindlish

Yash is an Enterprise Support Manager at Amazon Web Services. He has more than 17 years of industry experience including roles in cloud architecture, systems engineering, and infrastructure. He works with Global Enterprise customers to help them build scalable, modern, and cost-effective solutions on their growth journey with AWS. He loves solving complex problems with his solution-oriented approach.

Anjali Sharma

Anjali Sharma

Anjali Sharma is a Technical Account Manager (TAM) at AWS with more than 7 years of IT experience. Her diverse career includes roles such as Cloud Consultant and Operations Engineer at AWS Managed Services. In her current position, she collaborates with global customers to develop sustainable software solutions. She has a passion for troubleshooting and enhancing operational excellence for her customers.

Dheeraj Kumar

Dheeraj Kumar

Dheeraj is a Technical Account Manager at Amazon Web Services, with over 11 years of technical expertise. He works closely with Global Enterprise customers, leveraging his deep skills in Databases, Migrations, System Engineering, and AIML to deliver scalable, modern, and cost-effective solutions that drive their growth on the AWS Cloud.