AWS Partner Network (APN) Blog
Validating Cost Data for Amazon EKS with Kubecost and AWS Cost Explorer
By Isha Dua, Sr. Solutions Architect – AWS
By Jason Janiak, Partner Solutions Architect – AWS
By Thomas Nguyen, Sr. Software Engineer – Kubecost
 |
| Kubecost |
 |
Deploying Kubecost on a single cluster makes cost data easy to read and understand. However, complexity increases as infrastructure grows to include multiple Amazon Elastic Kubernetes Services (Amazon EKS) clusters across various regions and Amazon Web Services (AWS) accounts.
As such, retrieving and gathering cost data from multiple endpoints/Amazon EKS clusters to generate reports becomes time-consuming and complicated, especially for chargeback purposes.
This post acts as a guide for comparing/validating costs presented by the Kubecost dashboard with those via AWS Cost Explorer, helping to give better visibility into where data discrepancies may be occurring. We will also provide recommendations as to how these discrepancies can be remediated.
We’ll be looking at integrating Kubecost to get granular visibility into your Amazon EKS cluster costs, letting you aggregate costs by the majority of Kubernetes contexts, starting from the cluster level down to the container level.
Kubecost is an AWS Specialization Partner and AWS Marketplace Seller with the Amazon EKS Service Ready designation. It provides real-time cost visibility and insights for teams using Kubernetes, helping you continuously reduce cloud costs.
Solution Overview
Before comparing costs between Kubecost and AWS Cost Explorer, ensure your Kubecost deployment has configured Cloud Billing Integrations. Once you have the dashboard up and running, visit the Kubecost Assets page and ensure you’re aggregating by Service.
The Kubecost Assets dashboard provides a breakdown of Kubernetes cluster costs by the specific assets that make up the cluster, such as nodes, disks, and other resources. This allows you to analyze cost trends over time and validate allocation data. It can also (optionally) include costs for assets outside the cluster, categorized by service, tags, and labels. The goal is to give visibility into what’s driving spend in the cluster and provide auditing capabilities for cost allocation.
Kubecost Assets Dashboard
In the Kubecost Assets dashboard, you can aggregate cost by native Kubernetes concepts. While selecting Single Aggregation, you’ll only be able to select one concept at a time. While selecting Multi Aggregation, you can filter for multiple concepts at the same time. Assets will be by default aggregated by Service.
Figure 1 – Aggregating on service.
Select the date range of the report by setting specific start and end dates, or using one of the preset options. Select a start and end date which covers a window of one day, and is beyond 48 hours ago. You want to target a small and specific date range when doing this audit. You also want to select a date you know has reconciled with the AWS Cost and Usage Reports (CUR).
Kubecost is granted access to view cloud billing data, and initially it estimates asset prices using public cloud API data or custom pricing sheets. Once cloud billing data is available, usually within 1-2 hours for Amazon EC2 Spot instances and up to a day for Reserved instances, Kubecost adjusts its asset price estimates to be more accurate based on the actual billing data.
Though most adjustments happen within a day, some billing adjustments like credits may come in throughout the month or even at the very end. Kubecost continues reconciling adjustments over time as new billing data comes in. This allows Kubecost to provide highly precise real-time cost estimates that become even more accurate as complete billing data becomes available.
Figure 2 – Selecting date range.
Next, select the Kubernetes service and filter on Node. You’ll arrive at a page like this.
Figure 3 – Filtering on node.
Click on a Node to view its details. When you drill down at this level, you’ll be able to see a breakdown of CPU, RAM, and GPU usage.
Figure 4 – Drilling down on node.
AWS Cost Explorer Dashboard
AWS Cost Explorer allows you to view your costs in more detail by turning on hourly and resource-level granularity. To get a deeper insight into costs at the finest granularity, it’s recommended to enable hourly resource-level granularity to get visibility into specific resources, so you can pinpoint which assets are driving costs.
Using AWS Cost Explorer’s capabilities for time selection, filtering, and grouping, you can isolate the cost and usage data for your workloads in order to gain more insights. Compare this instance’s costs with what’s found in AWS Cost Explorer (Amortized). The most accessible way of doing so is Group by > Resource, and then Filters > Resource > <instance-id>.
Figure 5 – Filtering on the instance.
If you are unable to Group by > Resource (because it requires enabling hourly/resource granularity in AWS Cost Explorer), try Group by > Tag > aws:eks:cluster-name, and then Filters > Tag > aws:eks:cluster-name > <cluster-name>.
Note that these costs only account for the nodes and not the cluster management, control plane, disk, load balancer, or network costs.
Figure 6 – Total costs.
You can drill down at a node level in both AWS Cost Explorer and the Kubecost dashboard. We filtered on a single day (February 14) and the cluster name (kc-demo-prod) to see a cost comparison. In this example, both the numbers match to ~ $13.91. At this point, we have checked data at the finest granularity and now have the ability to drill up to the cluster and account levels for further validation.
When auditing other resources, note that assets whose costs in Kubecost get adjusted once reconciled with the AWS Cost and Usage Reports. Kubecost should be presenting its cost data within a 3% margin of what AWS Cost Explorer is presenting. When Kubecost reports costs on non-Kubernetes assets, those numbers should be exact as they are directly derived from the CUR.
Troubleshooting
In instances where a discrepancy is occurring between the cost data reported by Kubecost and AWS costs, the following are options which may help to remediate:
- Verify you’re using the latest version of Kubecost. There are often improvements and bug fixes that help align the numbers better. Upgrade if you are on an older version.
- Check that you have correctly configured your AWS credentials and permissions in Kubecost. Credentials need read-only access to AWS billing data and service usage data.
- Verify that Kubecost is collecting cost data from all relevant AWS services used by your Amazon EKS cluster. You can check this in the Kubecost user interface (UI).
- Compare the time periods for the Kubecost and AWS Cost and Usage Reports. If they’re different, the costs may not match up. Use the same time range.
- Check that auto-discovery of AWS resources is working properly in Kubecost. It should detect resources like load balancers and persistent volumes.
- Make sure Kubecost has usage metrics for all pod/services. Check for missing metrics which could skew costs. Filter by namespace and pod in Kubecost to isolate services where there may be incorrect cost allocations. Focus debugging there. For big discrepancies, export the Kubecost CSV data and compare with detailed CUR to find the source.
In addition, there are some programmatic methods you can use to validate the numerical data seen on the Kubecost dashboard:
- You can get the AWS Cost Explorer CSV export for one day filtered by account and service in AWS and compare that to the Kubecost /model/asset API request for the same day. This allows you to cross-check the numbers between the two sources.
- Set up AWS Cost and Usage Reports to export as a CSV file and then investigate the CSV file with Amazon Athena or other tools. You can run Athena queries against the data from the CUR reports to further validate the numbers.
- Investigate Prometheus to see if the underlying metrics about the node are sporadic or missing. If the metrics are inconsistent, that could indicate an issue with the Kubecost data. By programmatically validating against external sources like CUR, and checking the underlying Prometheus data, you can thoroughly confirm the accuracy of the Kubecost dashboard numbers.
Conclusion
This post focused on comparing the cost data between Kubecost and AWS Cost Explorer to validate for accuracy. In addition, guidance for troubleshooting steps that may assist in remediation of cost discrepancies.
If you’re having installation issues, troubleshooting documentation does highlight the various steps. If you experience a non-200 HTTP response code while using Kubecost, consult this post for a possible fix. Kubecost diagnostics run a series of tests to determine if resources necessary for accurate cost reporting are available.
If you have Kubecost’s enterprise license, additional features are enabled, such as governance features that allow you to set budget rules for different projects or audit the costly deployments on your Amazon EKS cluster. The enterprise licenses are available from Kubecost or through AWS Marketplace.
.
.
Kubecost – AWS Partner Spotlight
Kubecost is an AWS Specialization Partner that provides real-time cost visibility and insights for teams using Kubernetes, helping you continuously reduce your cloud costs.