AWS Partner Network (APN) Blog
Managing an Enterprise Backup Strategy with AWS Backup and CloudQuery
By Sabith Venkitachalapathy, Solutions Architect – AWS
By Yevgeny Pats , Co-Founder & CEO – CloudQuery
 |
| CloudQuery |
 |
Businesses consider data as a valuable asset and prioritize safeguarding it through various data protection mechanisms. They highly value the capability to effortlessly track and act upon data protection activities, ensuring compliance with their policies, strategies, and relevant regulatory mandates.
Observability holds significant importance in an AWS Backup implementation. By offering vital insights and a clear view of the backup process, it facilitates proactive monitoring, swift issue detection, effective troubleshooting, and ongoing optimization. This, in turn, guarantees the dependability, efficiency, and security of the backup solution.
Consequently, organizations can uphold data integrity and accessibility while optimizing cost and resource management.
AWS Backup offers a cost-effective, fully managed, policy-based managed service that simplifies data protection at scale. AWS Backup leverages AWS Organizations to centrally automate backup policies to implement, configure, manage, and govern backup activity across supported AWS resources.
CloudQuery is an open-source, high-performance extract, load, transform (ELT) framework that can be used to ingest more than 30 infrastructure connectors (including AWS) to any destination such as PostgreSQL, Amazon S3, Amazon Athena, BigQuery, Snowflake, and others.
CloudQuery is an AWS Partner that can be used to create full asset inventory across multiple Amazon Web Services (AWS) accounts, multi-cloud environments, and other infrastructures with your own modern data stack.
Combining CloudQuery and AWS Backup enables holistic management and visualization of AWS Backup, including backup job health, configuration management, compliance across resources, and protected resources. CloudQuery’s solution can be customized for unique customer needs and requirements.
Importance of Compliance in Data Protection
Ensuring adherence to industry standards and regulatory mandates can be a significant undertaking, demanding considerable time and effort. Organizations must diligently validate their compliance with diverse security frameworks and regulations, including but not limited to CIS, NIST, PCI DSS, ISO, and others.
A considerable amount of organizational hours is often devoted to manual preparations for compliance-related activities, such as rigorous preparation for a PCI audit. Post-preparation, additional time is invested in crafting comprehensive reports that effectively illustrate compliance.
It’s important to acknowledge this process merely marks the inception of an ongoing and iterative compliance endeavor. As outlined in NIST document SP 800-209 Security Guidelines for Storage Infrastructure, organizations are obligated to routinely and proactively evaluate configuration compliance with storage security policies.
CloudQuery Helps Build Compliance Posture
CloudQuery only requires access to configuration data and metadata and does not access the underlying data stored in AWS Backup or within AWS data stores such as Amazon Simple Storage Service (Amazon S3) and Amazon Relational Database Service (Amazon RDS).
AWS customers control access given to CloudQuery via AWS Identity and Access Management (IAM) policies and can customize the resource metadata scanned by CloudQuery, which has multiple architecture options including a self-managed solution within the customer AWS environment.
Customers can leverage CloudQuery to build visualizations and manage their AWS Backup strategies across multiple accounts. The following examples will utilize an AWS, CloudQuery, PostgreSQL, and Grafana stack.
Once we have loaded our AWS infrastructure data via CloudQuery to PostgreSQL, we can run analytical queries to draw insights and manage AWS Backup across resources and accounts.
Common Use Cases
One of the common operational requirements is to find any failed jobs that would have hindered a data protection strategy implementation. In order to find all failed backup jobs, we can run the following query:
SELECT * FROM aws_backup_jobs WHERE state='FAILED';
Figure 1 – AWS Backup job status.
Beyond daily operational monitoring, as per compliance reporting you’ll have to find the resources that are currently protected by AWS Backup. To find all protected (backed up resources), we can run the following query:
SELECT * FROM aws_backup_protected_resources;
Figure 2 – AWS resources protected by AWS Backup.
To determine the scope of a data protection implementation, we can run analytics and reporting on the number of protected resources grouped by resource type (examples include Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EC2) and by account using the below query:
SELECT resource_type, COUNT(arn), account_id FROM aws_backup_protected_resources GROUP BY resource_type, account_id;
Figure 3 – AWS resources protected by AWS Backup across an organization.
Figuring out what resources currently are not being backed up by AWS Backup is a common compliance ask. We can use the below query to determine resources that aren’t backed up, let’s say across S3, DynamoDB, EC2 instances, and Amazon EBS volumes:
SELECT resources.arn
FROM (
SELECT arn
from aws_s3_buckets
UNION
SELECT arn
from aws_dynamodb_tables
UNION
SELECT arn
from aws_ec2_ebs_volumes
UNION
SELECT arn
from aws_ec2_instances
) as resources
LEFT JOIN aws_backup_protected_resources ON resource_arn = resources.arn
WHERE last_backup_time is NULL;To ensure a smooth operation, we can also check for resources that didn’t have a successful backup within the last seven days but had previous backups using the below query:
SELECT * FROM aws_backup_protected_resources WHERE last_backup_time < (NOW() - interval '7 days');Additionally, we can check our AWS Backup Global Settings across all of the AWS accounts and regions we’re syncing from with CloudQuery:
SELECT account_id, region, global_settings, last_update_time FROM aws_backup_global_settings;The examples mentioned above can be extended further based on your business requirements using the AWS tables described in the CloudQuery documentation.
Running CloudQuery
There are multiple options for running CloudQuery, including within your AWS environment. One option is to run CloudQuery on Amazon Elastic Container Service (Amazon ECS) as highlighted in the CloudQuery documentation.
CloudQuery can be integrated with dashboards for reporting and visibility into infrastructure data. Here, we’ll use PostgreSQL and Grafana to visualize data synced by CloudQuery.
This pre-built dashboard includes information about:
- Resource compliance with example requirements across Amazon S3, EC2, and DynamoDB.
- Resources backup and reporting on numbers of resources backed up by AWS Backup.
- Health of backup jobs including completed jobs, failed jobs, and jobs with issues.
Figure 4 – AWS Backup health dashboard powered by CloudQuery and Grafana.
Conclusion
In this walkthrough, we have shown a few ways in which you can started with CloudQuery’s open-source solution to query AWS Backup’s operational data. This foundation can be extended to implement more robust observability patterns for your data protection strategies.
See CloudQuery’s website for additional technical documentation.
.
.
CloudQuery – AWS Partner Spotlight
CloudQuery is an AWS Partner that can be used to create full asset inventory across multiple AWS accounts, multi-cloud environments, and other infrastructures with your own modern data stack.