The AWS Cost & Usage Report: The Next Step on Your Cost Management Journey

by Erin Carlson, AWS Product Marketing Manager

Introduction by Mark Schwartz, AWS Enterprise Strategist

One of the advantages of the cloud is that it allows you to right-size your infrastructure continuously—to add infrastructure when you need it and release infrastructure and stop paying for it when you don’t. This gives enterprises new degrees of freedom in managing their IT capabilities and their budgets, and the freedom the cloud brings increases business agility.

Often, over time,  enterprises develop a good sense of how much infrastructure they will need, or at least a minimum they expect to use. They are then able to reduce their infrastructure costs substantially with Reserved Instances (RIs). The enterprise still retains the flexibility to add more instances if necessary, but gains in predictability and cost savings. In this guest post, Erin Carlson, the Product Marketing Manager for the AWS Billing and Cost Management experience, discusses RIs and the tools AWS offers for gaining transparency into RI usage and spending, and how to use that information to manage costs.

– Mark

The AWS Cost & Usage Report: The Next Step on Your Cost Management Journey

by Erin Carlson, AWS Product Marketing Manager

As an executive, there are a multitude of options available as you ideate and execute on your cloud management strategy. One of the most important things to plan for is how to make your cloud management solution scalable for your future business needs. Having a well-iterated plan is especially important when it comes to tracking your cloud costs, as the cloud has led to a paradigm shift in cost planning, where most of the resources that you use are metered at an hourly rate rather than on a monthly or quarterly basis. This has led to a significant increase in the number of line items being examined by your DevOps and finance teams.

Using AWS Cost Management products, such as AWS Cost Explorer and AWS Budgets, you can gain greater visibility into your usage patterns and underlying cost drivers, as well as take action on any issues that you might see. However, if you are looking to build an enterprise-grade cost management solution in-house, you should strongly consider using the AWS Cost & Usage Reports as your foundation. To learn more about the Cost & Usage Reports, you can also go here.

This post will walk you through some of the benefits of using the Cost & Usage Report, and will provide an overview of how this product stacks up against some of AWS’s legacy billing reports.

AWS Billing Reports Overview

AWS Billing Reports (which include the Cost & Usage Report and the legacy Detailed Billing Reports) are delivered at least once per day. The most comprehensive source of information available regarding your AWS usage is contained within the Cost & Usage Report. At a high level, these reports contain information about the following: accounts, billing, line item details, reservations, pricing, products, and tags. (For more details, please refer to the data dictionary.) The Cost & Usage Reports surface the AWS usage associated with each service category used by an account and its IAM users, allowing you and your team to dive deep into your organization’s usage patterns and cost trends. The reports can be ingested directly into Amazon Athena or Amazon Redshift, or uploaded to Amazon QuickSight for further analysis.

So, why should your business adopt the AWS Cost & Usage Report today?

Benefit #1: Comprehensive Reservation Related Information

Reservations (also known as Reserved Instances or RI) offer you a discounted hourly rate (sometimes up to 75%) compared to on-demand usage in exchange for committing to a one- or three-year term of service, which can translate into significant savings. In order to help you monitor and manage your reservation portfolio, the AWS Cost & Usage Report provides comprehensive information, including the Reservation Amazon Resource Number (ARN), the number of reservations, and the total reserved units. Using this information, you can track the application of your reservation related discounts to specific resources, helping you to better understand your savings. The Detailed Billing Reports provide a subset of this metadata, and significant work is required to transform the required columns.

 The Cost & Usage Report also provides a number of additional columns not available in the Detailed Billing Reports, including information regarding your amortized reservation costs. When reservations are purchased, upfront costs are recorded at the time they are charged, rather than at the time the associated usage was incurred. As such, it can be difficult to apportion these costs across workloads covered by a reservation. To address this issue, amortized costs distribute the upfront fees evenly across the duration of the RI lease as they are applied to the resource usage covered by the reservation, eliminating the need to calculate the amortized costs manually.

Benefit #2: Availability of On-Demand Pricing Information

The Cost & Usage Report provides information regarding the on-demand rates for each individual line item of usage. This information makes it easy to quantify your savings (e.g., pertaining to reservation related discounts) by simply subtracting the amount you paid from the on-demand rate, allowing you to quickly and easily compute your savings as compared to on-demand prices. This also gives you the flexibility of allocating your costs using public on-demand rates should you wish to do so. In contrast, the Detailed Billing Reports do not contain information about on-demand rates, only the billed amount, making it difficult to calculate your overall savings or perform cost allocation using on-demand rates.

Benefit #3: Granular Breakdown of Discounts

 AWS supports a number of usage-based discounts. A granular view of these discounts can be accessed from the Cost & Usage Report. In cases where discounts have been applied, you can use this report to view the cost prior to being discounted, the amount of the applicable discount, and the total cost after the discount was applied at the line item level. Only a subset of this information is available via the Detailed Billing Reports.

Benefit #4: Automated Data Ingestion at Scale

When the Cost & Usage Report is delivered, you can easily configure an event to fire to trigger an automated data ingestion process, streamlining the process of refreshing the billing data in your in-house systems. Please note that unlike the Detailed Billing Reports the Cost & Usage Report data can be automatically refreshed when charges are detected relating to previous months (e.g., refunds). Additionally, the AWS Cost & Usage Report is generated as multiple files, providing the added benefit of segmenting the data into smaller chunks. This provides an easy structure for parallelizing the data ingestion process using multiple ingestion worker processes, as well as the ability to, in cases of failure, retry data downloads in smaller chunks.

Additionally, the Cost & Usage Report is formatted in a way that allows for easy data location and extraction. This report is modeled via a manifest file that contains information regarding the overall structure of the data, including a list of every column that is contained within the report. This makes the report easily extensible in cases where new information regarding your usage becomes available. Because the Detailed Billing Report was architected at a time where fewer services and features existed, it became architecturally necessary to surface additional information within existing columns as appended values. These overloaded columns can cause additional complexity during post-processing, and require extra steps to extract values and load data.

Benefit #5: Cross-Product Integration

 The Cost & Usage Report supports out of the box integration with Amazon Redshift and Amazon QuickSight, making it easy to quickly build out an AWS-based cost management solution. Additionally, the Cost & Usage Report recently added support for Amazon Athena as well as support for providing data in Parquet format, broadening your options when it comes to building out your own cost and usage reporting system.

Summary

In summary, the Cost & Usage Report is the most comprehensive source of information with respect to your AWS costs and usage, allowing you to quickly access, ingest, and analyze your usage patterns and underlying cost drivers, especially when operating at enterprise scale. The Cost & Usage Report is best suited for organizations with complex cost management requirements, especially those who wish to establish dedicated query- or analytical-based systems in-house for cost reporting and analysis purposes. The Cost & Usage Report is also the best source of RI-related information, especially for customers who wish to view their costs in an amortized fashion.

To get started using the Cost & Usage Report, you can refer to the AWS Cost & Usage Reporting webpage and the Understanding Your Usage with Billing Reports user guide. To learn more managing and monitoring your AWS costs and usage at scale, you can view the AWS Cost Management webpages.

*****

Erin Carlson is a Product Marketing Manager for the AWS Billing & Cost Management experience. She works with customers to provide helpful guidance and resources around accessing, analyzing, and optimizing their AWS costs and usage.