The Amazon Web Services (AWS) Cloud offers flexible services and pricing options that allow you to pay only for the services you need, yet still meet your performance and capacity requirements. To help manage costs, AWS provides monitoring and reporting tools such as Cost Explorer, budgets, and  detailed billing reports, which are accessible in the Billing and Cost Management console. Customers can use this information to help monitor and forecast their service usage and charges in an effort to optimize costs and maximize their return on investment.

To support customers in their cost optimization efforts, AWS developed the Cost Optimization Monitor. This solution automatically processes detailed billing reports to provide granular metrics that you can search, analyze, and visualize in a customizable dashboard. The solution uses Amazon Elasticsearch Service (Amazon ES) and leverages its built-in support for Kibana, enabling customers to visualize their first batch of data as soon as it's processed.

This webpage gives high-level best practices for cost management as well as an overview of the Cost Optimization Monitor solution architecture and features.

When implementing a cost management strategy, remember that you can only control what you can measure. Take time to identify the most important metrics for your business, and develop processes for calculating and tracking that data. Leverage AWS billing reports to help monitor and forecast monthly service usage charges. AWS Cost and Usage reports and detailed billing reports offer granular data, giving customers insight into how specific projects or teams are consuming resources. With this in mind, consider the following AWS best practices for cost monitoring:

  • Consider establishing a team to support your cost management initiatives. It’s common to have people informally working on cost optimization, but a dedicated cloud management team can help promote best practices that are designed to accelerate optimization.
  • For a quick, high-level analysis use Cost Explorer, which is a free tool that you can use to view graphs of your AWS spend data. It includes a variety of filters and preconfigured views, as well as forecasting capabilities. Cost Explorer displays data from the last 13 months, the current month, and the forecasted costs for the next three months, and it updates this data daily.
  • Consider using budgets if you have a defined spending plan for a project or service and you want to track how close your usage and costs are to exceeding your budgeted amount. Budgets use data from Cost Explorer to provide you with a quick way to see your usage-to-date and current estimated charges from AWS. You can also set up notifications that warn you if you exceed or are about to exceed your budgeted amount.
  • Visit the AWS Trusted Advisor console regularly. Trusted Advisor works like a customized cloud expert, analyzing your AWS environment and providing best practice recommendations to help you save money, improve system performance and reliability, and close security gaps.

AWS offers a solution that automatically uploads detailed billing report data to Amazon ES for analysis and visualization in the Kibana user interface. The diagram below presents the Cost Optimization Monitor architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation template.

  1. The solution deploys an Amazon ES domain and an Amazon Elastic Compute Cloud (Amazon EC2) instance configured with an Nginx proxy to restrict access to the domain endpoint.
  2. The instance also hosts the solution's custom Python scripts, which retrieve and parse detailed billing report data and upload it daily to Amazon ES.
  3. The solution automatically loads a set of default Kibana dashboards as a starting point for data visualization.
  4. User requests from approved IP addresses can access the Kibana UI using customer-defined credentials, and start working with its search, visualization, and reporting capabilities to manipulate data from your domain.
Deploy Solution
Implementation Guide

What you'll accomplish:

Deploy the Cost Optimization Monitor using AWS CloudFormation. The CloudFormation template will automatically launch and configure the components necessary to send detailed billing report data to Amazon ES for analysis and visualization in a customizable, user-friendly dashboard.

Control access to your domain dashboards using an Nginx proxy to simplify authentication to Amazon ES, as well as user credentials for an extra layer of protection.

Simplify data visualization using built-in Amazon ES support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana 4.

What you'll need before starting:

An AWS account: You will need an AWS account to begin provisioning resources. Sign up for AWS.

Skill level: This solution is intended for IT infrastructure and networking professionals who have practical experience architecting on the AWS Cloud.

Q: Does the Kibana dashboard display historical data?

The dashboard will visualize data from all detailed billing reports that are uploaded to the solution's Amazon S3 bucket. If you enable detailed billing reports for the first time when you deploy the solution, the dashboard will only display metrics using data from that point forward. To check high-level historical data for your account, use Cost Explorer in the Billing and Cost Management console.

Q: Can I use the Cost Optimization Monitor to analyze billing data in multiple AWS accounts?

Customers who use Consolidated Billing to manage payments for multiple accounts can use this solution to aggregate and analyze billing data from all linked accounts. Note that you must deploy the solution in the payer account.  

Q: Can I deploy the Cost Optimization Monitor in any AWS Region?

Customers can deploy the solution's CloudFormation template only in AWS Regions where AWS Lambda is available because the service is required to configure the initial infrastructure. However, once deployed, the Amazon ES domain will receive billing data from all resources in your account.

Tell us what you think