Report and visualize your AWS Service Catalog estate

AWS Service Catalog allows organizations to create and manage catalog of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. In addition, organizations can centrally manage deployed IT services, applications, resources, and metadata. This helps you achieve consistent governance to meet your compliance requirements while enabling users to quickly deploy approved IT services.

To maximize value to their end-users, customers are interested in understanding how the catalog of IT Services (delivered via AWS Service Catalog) are consumed across their AWS accounts. Tracking who is consuming them and how efficiently these are being adopted will help provide insight into business and investment decisions. Hence gaining insights and reporting capabilities will allow customers to determine the impact when driving adoption of delivering services through Service Catalog.

For example, a customer can use AWS Service Catalog as a vending machine for an application. Consumers within their organisation deploy multiple instances of these application through a self-serve mechanism. In this case, it is important for the customer to understand who is using the application, if an outdated/out of support version of the application is being used, and the adoption rate of the application.

In this article we will show you how to leverage AWS services to extract, visualize, and report on Service Catalog usage across your AWS accounts.

Solution Overview

Figure 1: Proposed Architecture

Prerequisites

Before deploying the solution explained in this post, kindly follow the guide below:

• Prior to starting, we recommend reviewing the up-to-date pricing for Amazon S3, AWS Config, AWS Lambda and Amazon QuickSight.
• Setup AWS Config in one or more accounts. For setting up AWS Config you can refer to Getting Started with AWS Config. If you use AWS Organizations, you can enable AWS Config for all accounts using StackSets with all features enabled.
• Setup Aggregator in the Management Account using Setting Up Aggregator
  • AWS recommends delegating administration of AWS Config to a delegated administrator account rather than using the management account, for more information, review the AWS Config documentation
• Sign up for an Amazon QuickSight subscription in the same AWS account where AWS Config Aggregator is enabled

Walkthrough

As depicted in Figure 1, we are leveraging AWS Config, AWS Lambda, Amazon S3, Amazon QuickSight to deliver a reporting workflow, where the results of each component are used to deliver the overall reporting of Service Catalog usage.

• Aggregator from AWS Config consolidates the configuration state of AWS resources across the accounts
• A Lambda function executes SQL queries on AWS Config to extract the Service Catalog configuration state
• Lambda function stores the usage artifacts to S3 Bucket
• A QuickSight dataset is set up using a S3 Bucket where the artifacts of the Service Catalog usage are stored
• A QuickSight analyses is created based on the dataset

Now let’s go into a bit more detail on each component from the workflow and discuss how the components are tied together.

AWS Config provides an overview of the configuration of resources in your AWS account (see supported resource types in AWS Config). AWS Config has a feature called advanced queries which enables users to query the current state of resource configuration. In this article we are using queries to extract the state of a Service Catalog portfolio, products, and provisioned products.

E.g. the following query extracts the list of portfolios:

SELECT accountId, configuration.providerName, resourceName, resourceId, supplementaryConfiguration.portfolioAccess.accountIds WHERE resourceType = 'AWS::ServiceCatalog::Portfolio'

For complete list of properties that can be queried from AWS Config for Service Catalog, refer to:

• Portfolios
• Products
• Provisioned Products

You may have a multi-account setup and the adoption of Service Catalog could be across your AWS estate, within an organization or across multiple standalone AWS accounts.

AWS Config aggregator can collect configuration and compliance data from multiple AWS accounts and Regions into a single account and Region to get a centralized view of the resources. Above mentioned advanced queries can be executed on the aggregator.

Figure 2: AWS Config Aggregator

In the proposed solution (Figure 1), we are using a Lambda function which executes a set of SQL queries on an aggregator and extracts the information on Service Catalog Usage across the accounts. The extracted information is stored as an artifact into an S3 bucket in JSON format.

To visualize and analyze the extracted information, we are using Amazon QuickSight. The stored artifacts in S3 Bucket will be used to create a dataset on QuickSight and the analyses will be performed on the created datasets. The solution described in this article deploys the resources which are shown in the Figure 1, an analyses and a dashboard in QuickSight.

Step 1: Pre-deployment Steps

• Download the AWS CloudFormation template from here
• Create a New S3 bucket in the same region where the solution will be deployed (to store usage reports/artifacts) and enable permissions for QuickSight to access this S3 bucket as described here
• Create a User Group on QuickSight (this group can be used to manage permissions to the QuickSight analyses) as described here

Step 2: Deploy the CloudFormation template

Creating the Stack using the CloudFormation template requires the following inputs:

1. AWSConfigAggregator – Name of the Config aggregator to be used as the source for information
2. ReportingBucketName – S3 Bucket name where reports will be stored; QuickSight must have permissions to this bucket
3. QuickSightIdentityRegion – Region where the QuickSight is setup
4. QuickSightGroup – Group name of QuickSight author/admin from default namespace to which the dashboards/analyses will be shared
5. Suffix (optional) – Use a numeric suffix if you need to create multiple instances of this sample on same AWS account

Open CloudFormation from AWS Command Console and select “Create stack” → “with new resources”. Upload the “qs-analyses-template-Release.yaml” template from your local machine and provide the inputs to the stack (all inputs are mandatory):

Figure 3: Create Stack in CloudFormation

Figure 4: Create Stack – Parameters

Deploy the stack after providing the above inputs and once the deployment is deployed successfully, open QuickSight and search for “QSProductAnalysis” in Analysis section. Open analyses to visualize the results.

Similarly, open Dashboards section in QuickSight and search for “ProductUsage-Dashboard” which allows you to print or download the usage reports.

The results from the deployment from a test account are as below:

Figure 5: QuickSight Dashboard Part 1

Figure 6: QuickSight Dashboard Part 2

Step 3: Additional steps

• If you are interested to build additional analysis or dashboards depending on your use case, you can leverage the datasets created by the above stack to build your own analysis. You can refer to the AWS documentation on how the develop the analysis.
• For any additional properties or queries which needs to be included in your dataset, refer to the properties document in the intro of this blog and modify/add the queries to the Lambda function
• If you do not have a multi-account setup and are interested to capture the Service Catalog usage from a single account, you could leverage the similar approach but query from AWS Config instead of the AWS Config Aggregator
• To setup an ongoing data refresh, the Lambda function is enabled with a CloudWatch event trigger which updates the usage artifacts every 30 minutes. However, the dataset update has to be enabled manually. To enable data refresh on QuickSight datasets see https://docs.aws.amazon.com/quicksight/latest/user/refreshing-imported-data.html

Cleanup

• To delete the resources created, you can delete the CloudFormation stack using either the AWS Console or AWS CLI
• Ensure to empty the ReportingBucket S3 bucket before deleting it

Summary

In this article we discussed how we can use AWS Config aggregators and advanced SQL queries to consolidate the state of Service Catalog resources in a multi-account setup. We further discussed how QuickSight can be used to analyze and visualize the usage results. We deployed a Lambda function which executes the queries and stores the information on S3. The queried information is integrated into QuickSight through Datasets.

The above approach enables you to keep track of the number of products (catalog of IT services) being used with in your AWS environment. Additionally, it enables you to gain a deeper understanding of the versions of these and consumption of them. All this will assist you in providing the right solutions to your end-users, ensure latest versions with features and compliance are being used and more. We look forward to seeing the additional data and reports you generate using the described approach!