Optimize AWS Resource Management with Tag Inventory Reports leveraging AWS Resource Explorer

Customers are increasingly seeking an efficient solution to manage their expanding AWS resources, spanning AWS accounts and Regions, amidst changes like mergers, acquisitions, and cloud migrations. AWS Tags offer an effective solution for organizing, identifying, and filtering resources by categorizing them based on criteria such as purpose, owner, or environment. AWS customers would like to optimize AWS resource management with tag inventory reports, which provide detailed listings of the tags associated with various AWS resources. AWS introduced multi-account search using AWS Resource Explorer in 2023 to help improve resource management experience. However, while implementing a multi-account strategy customers often struggle with developing a tag policy in the first place because they don’t have good information on what tags already exist, what resources they’re being applied to, and what resources aren’t tagged.

To address this problem, we have developed an AWS Cloud Development Kit (CDK) solution leveraging AWS Resource Explorer, AWS Step Functions, AWS Lambda, AWS Glue, Amazon Athena, and Amazon Simple Storage Service (Amazon S3). It provides insights into tag usage – reporting on existing tags, what resources they are applied to, as well as what resources are untagged across a customer’s AWS Organizations. The data generated by this solution can be used to assist customers developing organizational tagging strategies, monitor their effectiveness over time, and make data-driven decisions on enforcement mechanisms. With greater visibility into resource tagging, companies can craft tagging policies that are optimized for their business needs including cost allocation, financial management, operations, security, and governance.

In this blog post, you will learn how you can deploy the solution in your own AWS accounts to generate reports that provide data on tagged resources, applied tags, and untagged resources across your AWS Organizations.

Solution Overview

The solution architecture (depicted in Figure 1 below) comprises a Central AWS account, designated as the primary hub, and one or more Spoke AWS accounts connected to it. In this setup, tag inventory data is gathered from each Spoke account and transmitted to an S3 bucket residing within the Central account. Subsequently, the collected data is aggregated to generate an inventory report in CSV format, which is then dispatched to the output S3 bucket located within the Central AWS account.

                                                                           Figure 1: Solution Architecture

Generating a consolidated tag inventory report

The processing happens at both levels (Spoke and Central accounts) and comprise the following 8 steps:

1. Spoke accounts have an Amazon EventBridge Scheduler, which allows for scheduling tasks that can invoke multiple AWS services and API operations. This EventBridge Scheduler would periodically trigger an AWS Step Functions state machine (SpokeAccountStateMachine in Figure 1). A Step Function state machine refers to a series of event-driven steps that may call upon different AWS services to perform specific tasks.
2. In this solution, the state machine will query all resources across all AWS Regions within the Spoke account using AWS Resource Explorer.
3. The state machine will then process and transform the results and write them to an Amazon S3 bucket (org-tag-inventory bucket in Figure 1) in the Central account.
4. Within the Central account there is an AWS Glue Crawler that will periodically crawl the Amazon S3 bucket containing the tag inventory data from the Spoke accounts. The crawler will generate a table in the AWS Glue Data Catalog. This will form the input for step 5.
5. An Amazon EventBridge Scheduler in the Central account will periodically trigger the GenerateCsvReportFunction AWS Lambda function.

6. GenerateCsvReportFunction run the following SQL statement using Amazon Athena, an analytics services that makes it easy to analyze data directly in Amazon S3 using standard SQL.

   CREATE TABLE "<DATABASE>"."tag_inventory_csv"
   WITH (
   format = 'TEXTFILE',
   field_delimiter = ',',
   external_location = 's3://<REPORT_BUCKET>/<YESTERDAYS_DATE>',
   bucketed_by = ARRAY['d'],
   bucket_count = 1)
   AS ( SELECT d,tagname,tagvalue,r.owningAccountId,r.region,r.service,r.resourceType,r.arn FROM "<DATABASE>"."<TAG_INVENTORY_TABLE>", unnest("resources") as t ("r") where d='<YESTERDAYS_DATE>')

7. Amazon Athena creates the table in the AWS Glue Data Catalog which also generates a csv file in the reporting org-tag-inventory-reporting bucket (see Figure 1) in Amazon S3.
8. Once complete, the GenerateCsvReportFunction will rename the report file and delete the table from the AWS Glue Data Catalog.

Design of AWS Spoke Account State Machine

The Figure 2 below, shows what happens in each AWS Spoke account to gather and process tag inventory (corresponding to steps 1, 2, and 3 in the Generating a consolidated tag inventory report section). This state machine is run periodically based on the schedule chosen at deployment. It gathers tag inventory data for resources within the account by querying the AWS Resource Explorer API, transforming the results, and sending them to the S3 bucket in the Central account for reporting.

                                                                   Figure 2: State Machine Workflow

Prerequisites

• All accounts (Central and Spoke) need to be part of the same AWS Organizations.
• Node.js is installed.
• AWS CLI is installed.
• Install the AWS Cloud Development Kit (AWS CDK) Tooklkit npm install -g aws-cdk.
• Make sure to have AWS credentials available on your terminal.

Note that the easiest way to deploy the solution is using the supplied command line interface (CLI) supplied as part of the solution.

Walkthrough

We recommend that you download the solution code from the GitHub repository.

Deploying in the Central stack

1. Open a terminal on your local machine.
2. Ensure there are AWS credentials available on your terminal for the account that you want to deploy the central stack to.
3. Checkout the project from GitHub by running git clone https://github.com/aws-samples/aws-organizations-tag-inventory.git.
4. Change into the project directory cd aws-organizations-tag-inventory.
5. Install dependencies npm install.
6. Run the CLI tool npm run cli, select Central and follow the prompts.

7. Select the AWS Region you want to deploy the resources to.

8. Select the frequency you want to generate the report.

9. Choose whether you want to deploy the Amazon QuickSight dashboard to visualize your tag data.

If you choose to deploy the Amazon QuickSight dashboard, you’ll be asked to select the users and groups that can access the dashboard.

10. Select whether you want to Deploy or Destroy this stack.

11. Confirm your choice.

12. Copy the output values for CentralStackNotificationTopicArnOutput , CentralStackPutTagInventoryRoleOutput , and OrganizationsTagInventoryBucketNameOutput , you’ll need these values later.

Once the Central stack is deployed, you can subscribe to notifications delivered through the deployed Amazon Simple Notification Service (Amazon SNS) topic.

Deploying a single Spoke stack

If you’d like to deploy just a single Spoke stack, follow the directions below. However, if you would like to deploy the Spoke stack to multiple account across your AWS Organizations, go to the Deploying multiple Spoke stacks using AWS CloudFormation StackSets section.

1. Open a terminal on your local machine.
2. Ensure there are AWS credentials available on your terminal for the account that you want to deploy the Spoke stack to.
3. Checkout the project from GitHub by running git clonehttps://github.com/aws-samples/aws-organizations-tag-inventory.git.
4. Change into the project directory cd aws-organizations-tag-inventory.
5. Install dependencies npm install
6. Run the CLI tool npm run cli, select Spoke and follow the prompts.

7. Select the AWS Region you want to deploy the resources to.

8. Select the frequency you want to gather tag inventory for.

9. Enter the central bucket name from Deploying in the Central stack Step 12.

10. Enter the central topic arn from Deploying in the Central stack Step 12.

11. Enter the arn of the central cross account role from Deploying in the Central stack Step 12.

12. Select the AWS Region that you want to setup AWS Resource Explorer indexes in.

13. Select the AWS Region that you want to setup the AWS Resource Explorer aggregator index in.

14. Specify an AWS Resource Explorer query for example tag:none. The default value will query all resources. See Query string syntax in the AWS Resource Explorer User Guide for more details.

15. Select whether you want to Deploy or Destroy this stack.

16. Confirm your choice.

Deploying multiple Spoke stacks using AWS CloudFormation StackSets

A prerequisite to deploy multiple spoke stacks using StackSets is the activation of trusted access with AWS Organizations.

1. Open a terminal on your local machine.
2. Ensure there are AWS credentials available on your terminal for the management account for your AWS Organizations.
3. Checkout the project from GitHub by running git clonehttps://github.com/aws-samples/aws-organizations-tag-inventory.git.
4. Change into the project directory cd aws-organizations-tag-inventory.
5. Install dependencies npm install.
6. Run the CLI tool npm run cli, select Organization and follow the prompts.

7. Select the AWS Region you want to deploy the resources to.

8. Select the frequency you want to gather tag inventory.

9. Enter the central bucket name from Deploying in the Central stack Step 12.

10. Enter the central topic arn from Deploying in the Central stack Step 12.

11. Enter the arn of the central cross account role from Deploying in the Central stack Step 12.

12. Select the AWS Region that you want to setup AWS Resource Explorer indexes in.

13. Select the AWS Region that you want to setup the AWS Resource Explorer aggregator index in.

14. Specify an AWS Resource Explorer query for example tag:none. The default value will query all resources. See Query string syntax in the AWS Resource Explorer User Guide for more details.

15. Select the organizational units (OUs). The Spoke stack will be deployed to accounts within these OUs. Choose the Root if you want to gather tag inventory from all accounts in the organization.

16. Select whether you want to Deploy or Destroy this stack.

17. Confirm your choice.

Review the generated report

The inventory report will be generated in the reporting bucket in the Central account. This bucket will begin with the prefix tag-inventory-reports. The report will be generated based on the schedule frequency you chose in Step 8 of Deploying in the Central stack.

                                                                                Figure 3: Tag Inventory Report

The following fields are captured in the report output:

date – the date the report was generated.

tagname – key assigned to the tag.

tagvalue – value assigned to the tag.

owningaccountid – the AWS account id where the resource is deployed in.

region – the AWS region the resource is deployed in.

service – the AWS service the tag is associated to.

resourcetype – the AWS resource type the tag is associated to.

arn – the arn of the AWS resource the tag is associated to.

Amazon QuickSight Dashboard

The optionally deployed Amazon QuickSight dashboard currently shows several example visualizations, such as the number of distinct tag names and the percentage of tagged vs untagged resources. For an example, refer to the different visualizations in this chart (Figure 4).

                                                                 Figure 4: QuickSight Dashboard

Clean up

To remove the deployed resources, simply re-run the command line tool for Central, Spoke, and Organizations stacks, select Destroy to delete the stacks via CDK.

Other considerations

Removal policies

For the purpose of this example, the removal policy for all Amazon S3 buckets has been configured as DESTROY. This means that when the resource is removed from the app, it will be physically destroyed.

Service quotas

If you have many of resources in an account or run the reports daily, you may exceed the Aggregator Region search monthly quota limit depending on the AWS Resource Explorer query you specified during the deployment.

Cost

The cost to run this solution will depend on the number of accounts it is deployed to, the number of resources and tags in each of those accounts, and the frequency with which the data is generated.

Conclusion

In this blog post, we detailed an AWS Cloud Development Kit (CDK) solution for gaining visibility into resource tagging across an AWS Organizations. The solution utilizes AWS services including Resource Explorer, Step Functions, Lambda, Glue, Athena, and S3 to report on existing tags, what resources they are applied to, as well as what resources are untagged. With the data supplied by this solution you can take control of your resource tagging strategy by utilizing AWS Resource Explorer to gain comprehensive visibility, enabling data-driven decisions aligned with cost, financial, operational, security, and governance objectives across your AWS Organizations. For more information on developing a tagging strategy, check out the Best Practices for Tagging AWS Resources whitepaper.