AWS Cloud Financial Management

Visualizing Your Eligible On-Demand Compute Expense for AWS Savings Plan

Voiced by Amazon Polly

One of the most common questions we get from customers is how to manage compute costs for resources like Amazon EC2, AWS Lambda, and AWS Fargate.  Amazon Web Services (AWS) has many offerings to help you optimize spending, one of which is AWS Savings Plans. You can receive up to 72% discount off your On-Demand Instances usage, like EC2 Reserved Instances, in exchange for a commitment to use a specific amount of compute power (measured in $/hour) for a 1 or 3-year period.

Companies of all sizes face the challenge of gaining insights into organizational wide cloud cost. Typically, based on your annual estimated compute growth plans, you can often sign up for AWS Savings Plans. It happens sometimes that if you underestimate your compute needs and over time your On-Demand compute cost can grow beyond your original estimated Savings Plans.  You can purchase additional Savings Plans anytime during the year. AWS Savings plan recommendations help you to understand what you can save, how the commitment will be used, and more. However, it does not dive deep into your daily On-Demand compute spend across individual member (linked) accounts. In this blog post, we will walk you through how you can build an actionable Amazon QuickSight dashboard on ‘AWS On-Demand compute spent’ which is Savings Plan eligible.

Solution Overview

AWS services like AWS Cost and Usage report (CUR), Amazon Simple Storage Service (S3), AWS Glue, Amazon Athena, and Amazon QuickSight are leveraged to build this solution. At a high level, we create the CUR with details such as Linked Account ID, On-Demand compute spend hourly granularity and choose Amazon Athena as the report data integration. Next, we integrate the CUR data with Amazon Athena using AWS Glue Crawler. Finally, we use Amazon QuickSight to build the dashboard using a dataset created via querying the Athena views.

Customers might incur minimal expense to run and maintain the AWS resources mentioned in this blog. AWS Pricing Calculator lets you explore these AWS resources, and create an estimate for the cost of the use case on AWS. The following image shows the solution we build in this blog post.

Walkthrough

For this walkthrough, you should have access to the following prerequisites:

  • An AWS account
  • AWS Organizations set up manually or via AWS Control Tower
  • Admin user or role in your AWS Organizations  account
  • Amazon Athena Tables and Views
  • Amazon Quicksight Enterprise edition
  • AWS CloudFormation
  • AWS Well-Architected Labs

Step 1: Enabling CUR Data for Amazon Athena Integration

Cost and Usage Report data contains granular details on your resource usage of all the member accounts in your AWS Organization. It contains cost breakdown of your resources based on the hour, day, or month, and by AWS service or by user-defined cost allocation tags. CUR data is configured to deliver the granular data point per resource to the Amazon S3 bucket following the instructions here. For time granularity, choose “hourly” as we want the line items in the report to be aggregated by the hour.  It can take up to 24 hours for AWS to start delivering reports to your Amazon S3 bucket. AWS Cost and Usage reports are delivered once a day in a parquet format which enables integration with Amazon Athena.

Step 2: Creating a CUR and Amazon Athena integration pipeline

To streamline and automate integration of your Cost and Usage Reports with Athena, use this AWS CloudFormation template which creates resources along with the reports that you set up for Athena integration. This automated data ingestion workflow (referred in the following architecture diagram) consists of AWS Glue crawler, AWS Glue Database, AWS Lambda functions and S3 event notification resource. This pipeline delivers up-to-date CUR data to Athena.

After the CloudFormation stack creation is completed, make sure that the following resources are created successfully using AWS Management Console or AWS CLI.

  • Three IAM roles
  • An AWS Glue database
  • An AWS Glue crawler
  • Two Lambda functions
  • An Amazon S3 notification

Step 3: Validating from Amazon Athena console

Next, navigate to Amazon Athena console. The ‘cost and usage report’ table created by the AWS Glue Crawler appears in the left navigation pane under Database.

The following screenshot shows the ‘cost and usage report’ Athena Table.

Step 4: Creating Amazon Athena View for Amazon QuickSight Integration

A view in Amazon Athena is a logical, not a physical, table. Now, we create a view with a subset of columns from the original CUR table to simplify querying data.

Create a new Amazon Athena view by selecting the appropriate query in the AWS Well-Architected labs based on your use-case.

The following screenshot shows the Athena view after the query is run.

Step 5: Building the Amazon QuickSight Insights

Create a QuickSight dashboard to discover ‘Compute Savings plan Eligible’ spend.

1.      On the AWS Management Console, navigate to QuickSight service and then select your AWS Region from top right corner. Note that QuickSight Dataset should be in the same AWS Region as that of Amazon Athena table.

2.  Make sure that you authorize Amazon QuickSight to use S3 and Athena

3.     Create QuickSight Dataset by selecting the data-source as the Amazon Athena view that you created.  You can choose to automatically refresh the Dataset.

The following screenshot shows the QuickSight Dataset after the creation.

3.      Create QuickSight Analysis.

4.      Create QuickSight Visual by selecting the visual type ‘stacked area line chart’

5.      Select the following Fields

x axis – usage_date

y axis – unblended_cost(Sum)

color – linked_account_id

6.      Format the Visual to meet your needs.  In our dashboard, we used a reference line to show the ‘daily average on-demand spend’. Refer the following screenshot to create one for yourself.

7.      Now, you can publish your dashboard  from the Analysis.

8.      You can send the dashboard in report form to individuals or groups, either once or on a schedule.

With the preceding dashboard, you are now able to identify ‘Compute Savings plan Eligible’ spend across all the ‘linked accounts’ in an Organization. You can create different visual types in Amazon QuickSight to fit your needs. For example, you can create a pivot table and export it as an excel or csv file.

Export Amazon QuickSight dashboard from an Organization/Account to another Organization/Account

The QuickSight dashboard that you created is a reusable resource that you can export from an Organization/Account to another Organization/Account. Follow the next steps to share the template across the constituent OU’s/Accounts.

Steps in Organization/Account – A

Follow the next steps to create a template and grant cross-account READ permission.

Step 1

Run the following command to get the DatasetID of CUR-specific Dataset

aws quicksight list-data-sets --aws-account-id <Account_A> --region <region_1>

Step 2

Run the following command to get the AnalysisID of CUR-specific Analysis

aws quicksight list-analyses --aws-account-id <Account_A> --region <region_1>

Step 3

To create a template, run the command using the following AccountAtemplate.json (sample) file.

aws quicksight create-template --aws-account-id <Account_A> --region <region_1> --template-id firsttemplate --source-entity file://AccountAtemplate.json
 Sample AccountAtemplate.json 
       
{ 
   "SourceAnalysis": { 
     "Arn": "arn:aws:quicksight:<Region of analysis>:<Account_A>:analysis/<Analysis ID>", 
     "DataSetReferences": [ 
       { 
         "DataSetPlaceholder": "<Any name>", 
         "DataSetArn": "arn:aws:quicksight:<dataset region>:<Account_A>:dataset/<DatasetID>" 
       } 
     ] 
   } 
}
 
       
Run the following command to verify the newly created template since we can’t see it in the AWS Management Console. Save the ARN of template 'firsttemplate' as you would need this in Account B.
 
       
aws quicksight describe-template --aws-account-id <Account_A> --region <region_1> --template-id firsttemplate
 
       
Step 4
 
       
To grant cross-account READ permission from the source template, run the command using the following TemplateSharingConfig.json (sample) file.
 
       
aws quicksight update-template-permissions --aws-account-id <Account_A> --region <region_1> --template-id firsttemplate --grant-permissions file://TemplateSharingConfig.json
 
       
Sample TemplateSharingConfig.json (before you proceed, please make sure you have the Account-ID of Account B ready)
 
       
[ 
   { 
     "Principal": "arn:aws:iam::<Account_B>:root", 
     "Actions": [ 
       "quicksight:UpdateTemplatePermissions", 
       "quicksight:DescribeTemplate" 
     ] 
   } 
]
 
       
Steps in Organization/Account – B
 
       
Follow the next steps to create a dashboard in Account B based on the source template from Account A.
 
       
Before you proceed
 
       
     
        
  • Account B IAM user should have appropriate QuickSight permissions
    •  
        
  • CUR-specific Dataset must be created
    •  
        
  • ARN of template 'firsttemplate' from Account A
    •  
       
 
       
Step 1
 
       
To create a template, run the command using the following TemplateReplicationConfig.json (sample) file.
 
       
aws quicksight create-template --aws-account-id <Account_B> --region <region_1> --template-id firsttemplate --source-entity file://TemplateReplicationConfig.json
 
       
Sample TemplateReplicationConfig.json
 
       
{
   "SourceTemplate": {
     "Arn": "arn:aws:quicksight:<region_1>:<Account A ID>:template/firsttemplate"
   }
}

 
       
Run either of the following commands to verify the newly created template since you can’t see it in the AWS Management Console. Save the template ARN.
 
       
aws quicksight list-templates --aws-account-id <Account_B> --region <region_1>
 
       
aws quicksight describe-template --aws-account-id <Account_B> --region <region_1> --template-id firsttemplate
 
       
Step 2
 
       
To create a dashboard, run the command using the following DashboardFromTemplate.json (sample) file.
 
       
aws quicksight create-dashboard --aws-account-id <Account_B> --region <region_1> --dashboard-id <dashboard_id> --name <dashboard_name> --source-entity file://DashboardFromTemplate.json
 
       
(use "aws quicksight list-data-sets --aws-account-id <Account_B> --region <region_1>" to get CUR Dataset ARN)
 
       
Sample DashboardFromTemplate.json
 
       
{
    "SourceTemplate": {
    "DataSetReferences": [
         {
                "DataSetPlaceholder": "<Any name>",
                "DataSetArn": "arn:aws:quicksight:<region_1>:<Account_B>:dataset/<Dataset ID>"
          }
     ],
     "Arn": "arn:aws:quicksight:<region_1>:<Account_B>:template/<Template ID>"
     }
  }
 
       
Step 3
 
       
To grant IAM permission to the IAM role in Account B, run the command using the following DashboardPermission.json (sample) file.
 
       
aws quicksight update-dashboard-permissions --aws-account-id <Account_B> --region <region_1> --dashboard-id <dashboard_id> --grant-permissions file://DashboardPermission.json
 
       
Sample DashboardPermission.json
 
       
[ 
   { 
     "Principal":"arn:aws:quicksight:<region_1>:<Account_B>:user/default/<Quicksight user name>", // ARN of the QuickSight user you want to assign ownership 
     "Actions":[ 
      "quicksight:DescribeDashboard", 
      "quicksight:ListDashboardVersions", 
      "quicksight:UpdateDashboardPermissions", 
      "quicksight:QueryDashboard", 
      "quicksight:UpdateDashboard", 
      "quicksight:DeleteDashboard", 
      "quicksight:DescribeDashboardPermissions", 
      "quicksight:UpdateDashboardPublishedVersion" 
     ] 
   } 
]
 
       
 Step 4 Run either of the following commands to verify the newly created Dashboard List the Dashboard 
       
aws quicksight list-dashboards --aws-account-id <Account_B> --region <region_1>
 
       
Describe the Dashboard
 
       
aws quicksight describe-dashboard --aws-account-id <Account_B> --region <region_1> --dashboard-id <dashboard_id>
 
       
Step 5
 
       
On AWS Management Console, navigate to QuickSight dashboards and you should be able to see the dashboard created in step 2 of this section.
 
       
Cleaning up
 
       
To avoid ongoing charges in your AWS account, you can delete the resources created in this blog post.
 
       
     
        
  • Login to AWS Management Console
    •  
        
  • Empty your S3 bucket provisioned for CUR data
    •  
        
  • Go to the AWS CloudFormation console
    •  
        
  • Choose the stack you created and select ‘Delete’ to clean up the AWS resources that were created by the CloudFormation template
    •  
        
  • Turn off the CUR report data integration by editing the Cost and Usage Reports configuration
    •  
        
  • Unsubscribe your Amazon QuickSight account
    •  
       
 
       
Conclusion
 
       
The Amazon QuickSight dashboard created in this blog post will help Customers gain insights into On-Demand compute spent per AWS Account, which is Savings Plans eligible. We also recommend to track your current Savings Plan usage using the Savings Plans utilization report from the AWS Cost Management console to make additional Savings Plans purchase decision to augment your cloud cost optimization strategy. The solution outlined in this blog can be complimented with the additional recommendations outlined in the AWS Well-Architected cost optimization lab. Companies of all sizes can adopt this proactive approach to cloud cost optimization strategy and improve business efficiency through cloud cost analytics and data-driven decisions.
 
       
For further reading, please refer AWS Well-Architected Framework, Architecture Best Practices for Cost Optimization and Cloud Financial Management. We are here to help you and if you need further assistance in cost optimization of your AWS environment, please reach out to AWS Support and your AWS account team.
 
        
      
 
      
 
       
 
        
 
         Arun Chandapillai 
        
 
        
Arun Chandapillai
 
        
Arun Chandapillai is a Senior Cloud Architect who is a diversity and inclusion champion. He is passionate about helping his Customers accelerate IT modernization through business-first Cloud adoption strategies and successfully build, deploy, and manage applications and infrastructure in the Cloud. Arun is an automotive enthusiast, an avid speaker, and a philanthropist who believes in ‘you get (back) what you give’.
 
       
 
       
 
        
 
         Shak Kathir 
        
 
        
Shak Kathir
 
        
Shak Kathirvel is Senior Cloud Application Architect with AWS ProServe. He enjoys working with customers and helping them with Application Modernization and Optimization efforts, guide their Enterprise Cloud management and Governance strategies and migrate their workloads to the cloud. He is passionate about Enterprise architecture, Serverless technologies and AWS cost and usage optimization. He loves his job for its challenges and the opportunity to work with inspiring customers and colleagues.