Gain visibility into your Kubernetes spend with CloudZero and Amazon CloudWatch Container Insights

Container adoption has been increasing rapidly in the past few years. Customers are deploying workloads of all sizes on Amazon Elastic Kubernetes Service (Amazon EKS). Typically, cluster administrators deploy several business applications and workloads on a cluster to achieve more efficient deployment density.

On large clusters in a shared infrastructure where workloads of different sizes are deployed, it can be challenging to identify the operational costs for individual applications.

Here is what one of our customers, Matthew Jackson, an Engineering Lead (DevOps and SRE) at Drift had to say about the issue:

In order to make our K8s clusters more efficient, we needed to know which services were driving most of the resource consumption, so that we knew where to prioritize our engineering efforts.

In this blog post, let’s discuss how CloudZero uses Amazon CloudWatch Container Insights to create a solution for this problem.

Addressing the challenge

Solving this problem starts with collecting the necessary data. This can be challenging because many billing and performance data points have to come together.

• In Amazon Elastic Compute Cloud (Amazon EC2), we need to know how many Amazon EC2 instances are in use in a cluster. We also need to know their instance types (On-Demand, Reserved, Spot, or a combination), number of vCPUs, memory and storage size, and compute type.
• On the Amazon EKS cluster, we need to know the number of pods or namespaces, node types, CPU, and memory allocation by pod and namespace.
• Finally, we need to know which other AWS services and resources are being used. Customers might ask themselves: What am I paying for those services? How much am I paying for network traffic flowing to or from my cluster? How are those resources tagged? If some of those resources are shared by multiple systems, what percentage is allocated to my containerized features?

Solution

CloudZero developed a 100% serverless platform for AWS Cloud intelligence to continuously track your resource consumption and spend. Customers have everything they need to understand their Amazon EC2 costs and operational details. But they were missing the data required to understand what was happening inside the Amazon EKS. To solve that challenge, CloudZero turned to AWS and Amazon CloudWatch Container Insights.

Amazon CloudWatch Container Insights has been generally available since August 2019. This feature brings all of the operational and performance details inside a cluster directly into Amazon CloudWatch metrics and logs. It provides new dashboards that are easily accessible from the CloudWatch Container Insights console.

More importantly to CloudZero and AWS customers, Container Insights feature provides a secure and automated solution for sharing data collected from the container environment. For those customers already using Container Insights, CloudZero offers the added benefit of detecting and reporting on container costs immediately, without any configuration.

EKS Service level performance monitoring dashboard screen from CloudWatch Container Insights page on AWS console. Dashboard contains Widgets including CPU Utilization, Memory Utilization, Network traffic and Number of Pods.

By using Amazon CloudWatch Container Insights, CloudZero delivers one of the most secure, and easy to deploy solutions for container cost monitoring. This solves our customers’ challenges with managing the cost of containers on AWS.

As Matthew Jackson says,

“CloudZero and Amazon CloudWatch Container Insights made it possible for us to go further and easily track cluster costs based on actual consumption, which let us identify the applications consuming the most resources and prioritize our engineering resources where we could leverage them the most.”

Setting up CloudZero in your AWS account

Prerequisites

In order to implement the instructions in the following sections, you need to have access to an AWS account with administrative permissions.

If you’re new to Container Insights and CloudZero, follow these steps to get set up.

Step 1: Connect CloudZero to your AWS account

If you don’t already have CloudZero configured for your environment, go to https://app.cloudzero.com to create your account. Your first month is free, and if your spend is under $1000 a month, CloudZero is free to use.

CloudZero automated account connection and onboarding webpage for connecting AWS accounts, Slack integration and adding team members found at https://app.cloudzero.com

You need access to your primary AWS payer account and an account that has a Kubernetes or Amazon EKS cluster deployed. After you are connected through CloudZero’s fully automated AWS CloudFormation template, and you have AWS billing data flowing, you are ready to install Container Insights.

Step 2: Deploy Container Insights to your cluster

Follow the steps in the Amazon CloudWatch user guide to set up Container Insights.

After it’s deployed, Container Insights gathers performance metric data including CPU, memory, and network usage. It also writes this information to the following Amazon CloudWatch log groups:

Container Insights uses the embedded metric format (EMF) to create custom CloudWatch metrics that also appear in the Container Insights dashboards. These custom metrics are available for you to use in your own CloudWatch dashboards and alarms.

Step 3: Connect CloudZero to Container Insights

CloudZero automatically identifies when Container Insights is enabled and uses the data from the /aws/containerinsights/<cluster-name>/performance log to calculate and display your container costs. No additional configuration is required. CloudZero performs an efficient CloudWatch Logs Insights query every hour to aggregate metrics with minimal impact on your AWS costs.

CloudZero also ingests past data automatically. So if you already have a month’s worth of Container Insights data, you can analyze that months’ worth of costs after you connect CloudZero to your account.

CloudZero and Container Insights flow diagram

Fully automated container cost allocation

By using CloudZero to combine the metrics from Container Insights, and AWS cost and usage information, customers can automatically allocate costs to workloads that are being orchestrated by Kubernetes. CloudZero calculates the cost of each pod within the architecture. It then reaggregates them to attribute costs to other hierarchical Kubernetes concepts, like namespaces and clusters.

Maps Containers cost to your business. Identify cost of a single business application which is hosted on a shared EKS cluster.

Cost is calculated based on the cost of the EC2 instance, which represents the cost of a node in the Kubernetes system, combined with pod-level CPU and memory utilization. This makes it possible to assign a portion of the node’s total cost to the pod. The CloudZero platform handles this automatically. There is no need for manual allocation rules.

Generally speaking, this proportional algorithm works across a range of EC2 instance types.

Using CloudZero, you now have a new way to explore your container costs over time by cluster, pod, or namespace. For example, we can use CloudZero to check one of our clusters and find how its costs decrease as we scale down the cluster.

We can also observe the cost to run Container Insights on the cluster, which in this example, is only about $0.01 a day.

A view into EC2 and Kubernetes spends with CloudZero

Drift is now empowered to go beyond their cost savings goals and start to think proactively about optimizing their cloud computing spend. In the words of Matthew Jackson,

Now, in addition to having hit our most recent cost goals, we’re much more consistently able to identify when applications are consuming more of the shared cluster resources in advance, and can give timely feedback to their developers before it becomes a problem.

Cleaning up

You can easily remove CloudWatch Container Insights setup from your Amazon EKS or Kubernetes on EC2 cluster by following the instructions from the docs Amazon CloudWatch Container Insights documentation.

Conclusion

In this blog post, we described the solution CloudZero developed using metrics and logs collected by Amazon CloudWatch Container Insights. This helps cloud operations teams identify costs for individual workloads deployed on Amazon EKS clusters based on their usage. Amazon CloudWatch Container Insights helps you get performance and health insights into your Amazon EKS, Kubernetes on EC2, Amazon ECS, and Fargate cluster environments by automatically collecting metrics and logs from these environments. For more information, visit CloudZero.com and the Amazon CloudWatch Container Insights documentation. The One Observability Workshop is a great way to get familiar with observability features offered by Amazon CloudWatch, including Container Insights.

Authors

Erik Peterson  is the founder and CTO of CloudZero . Erik has been building on AWS for over a decade and is a frequent speaker on cloud economics, DevOps and security. Previous to CloudZero Erik was Director of Technology Strategy for Veracode and has 20 years of software industry experience, including senior leadership and technology roles at HP, SPI Dynamics, GuardedNet and Sanctum.

Imaya Kumar Jagannathan is a Senior Solution Architect focused on Amazon CloudWatch and AWS X-Ray. He is passionate about Monitoring and Observability and has a strong application development and architecture background. He likes working on distributed systems and is excited to talk about microservice architecture design. He loves programming on C#, working with Containers and Serverless technologies.

Kevin Mueller is Senior Solution Architect on the SaaS Factory team helping AWS Partners accelerate adoption of SaaS application. Kevin has over 30 years’ experience developing distributive software applications in the financial exchange, high frequency trading, simulation, and virtual venality space. He is a co-author of the AWS Well-Architected SaaS Lens.He is a US Army veteran holding master degree in computer science. He spent 18 year in the financial industry helping disrupting this space from manual trading to be fully electronic trading while working for New York Stock Exchange Group and MIAX Exchange Group. He is the inventor for the patent “Systems and Methods for Testing a Financial Trading System.”