AWS Cloud Operations Blog
Observing Kubernetes workloads on AWS Fargate with AWS managed open-source services
AWS constantly innovates on its customers’ behalf and strives to improve customer experience by reducing complexity. With AWS, customers look to spend their time solving business problems without worrying about operating their infrastructure. Amazon Elastic Kubernetes Service (Amazon EKS) on AWS Fargate allows customers to run Kubernetes pods without worrying about creating and managing the lifecycle of underlying worker nodes. With Fargate, you pay only for computing resources, with no upfront expenses. If you can run your containers on Fargate, you can avoid having to size EC2 instances for your workload. Fargate allows you to allocate the required resources for the application and only pay for that. EKS enables a cluster where some pods run on EC2 while others run on Fargate. You can run all the pods in a namespace on Fargate or specify a label for the pods you want to run on Fargate.
Customers are now prioritizing workload gravity and turning to managed services like Amazon Managed Service for Prometheus, and Amazon Managed Grafana to monitor their workloads. Customers know that establishing a monitoring system helps them visualize essential performance metrics for the cluster and workload. Using metrics such as vCPU utilization
, memory utilization
, and network usage
can offer valuable insights into how resources are being used and help identify potential problems ahead of time.
We recently announced the release of the AWS CDK Observability Accelerator, a set of opinionated modules to help customers set up observability for Amazon EKS clusters. It simplifies observability provisioning and maintenance for Amazon EKS clusters using AWS Cloud Development Kit (AWS CDK) and CDK EKS Blueprints. The AWS CDK Observability Accelerator and AWS Observability Accelerator for Terraform are solutions organized around patterns and reusable units for deploying multiple resources.
In this post, we’ll explore leveraging the AWS CDK Observability Accelerator to build observability quickly for monitoring Amazon EKS on AWS Fargate with AWS-managed open-source services. We will be demonstrating infrastructure monitoring along with monitoring a java workload and Nginx ingress on AWS Fargate.
Solution Overview
The pattern we are going to deploy will provision the following components:
- An Amazon EKS cluster powered by Fargate providing on-demand compute capacity for our container pods
- AWS Distro For OpenTelemetry (ADOT) Operator and Collector for collecting metrics and traces from applications
- AWS for FluentBit to capture and ingest logs into Amazon CloudWatch
- Amazon Managed Prometheus configured with rules to collect observability data generated by EKS Fargate Cluster
- External Secrets Operator to retrieve and sync the Grafana API keys from AWS Systems Manager
- Grafana Operator to add AWS data sources and create Grafana dashboards in Amazon Managed Grafana
- Flux will perform GitOps sync to the EKS cluster of a Git repository hosting configuration of Grafana dashboards and AWS data sources. Please check our GitOps with Amazon Managed Grafana module in One Observability Workshop to learn more on this topic.
In the following diagram, you can see the metrics from source to destination through the main components of the pattern:

Figure 1: Architecture Diagram for monitoring infrastructure and workloads on Amazon EKS on AWS Fargate with AWS-managed open-source services.
Prerequisites
You will need the following to complete the steps in this post:
- AWS CLI version 2
- AWS CDK version 2.86.0 or later
- Homebrew to install required packages for macOS or Linux
- An existing Amazon Managed Grafana Workspace
- Node version 20.0.0 or later
- NPM version 10.0.0 or later
- Kubectl
- Git
- Make
Environment setup
Let’s start by setting a few environment variables:
Next, let’s get the workspace id of any existing Amazon Managed Grafana workspace:
Take note of the workspace ID and endpoint of the Grafana workspace you wish to use and save them in bash variables as shown below.
export COA_AMG_WORKSPACE_ID="<<Your-WORKSPACE-ID>>"
export COA_AMG_ENDPOINT_URL="https://<<YOUR-WORKSPACE-URL>>"
Next, create a Grafana API key from Amazon Managed Grafana workspace and create a secret in AWS Systems Manager. The secret will be accessed by the External Secrets add-on and made available as a native Kubernetes secret in the Amazon EKS cluster:
Bootstrap CDK
The first step to any AWS Cloud Development Kit (AWS CDK) deployment is bootstrapping the environment. CDK bootstrap is a tool in the AWS CDK command-line interface (AWS CLI) responsible for preparing the environment (i.e., a combination of AWS account and AWS Region) with resources CDK requires to perform deployments into that environment. CDK bootstrapping is needed for each account/region combination, so you don’t need to repeat the process if you have already bootstrapped CDK in a region.
Execute the commands below to bootstrap the AWS environment in your region:
cdk bootstrap aws://$ACCOUNT_ID/$AWS_REGION
Deploying the Fargate open-source observability pattern
Clone the cdk-aws-observability-accelerator repository and install the dependency packages. This repository contains CDK v2 code written in TypeScript.
The settings for Grafana dashboard json
files are expected to be specified in the CDK context. Generally, such settings are defined inside the cdk.context.json
file of the current directory or in ~/cdk.json
in your home directory. You will need to update the context in cdk.json
file located in cdk-aws-observability-accelerator
directory.
Settings for this deployment are:
Now, run the below command to deploy the open-source observability pattern.
make pattern single-new-eks-fargate-opensource-observability deploy
Once the deployment completes, run the update-kubeconfig command.
Validation
To validate the status of the resources created by our deployment, we check the status of the Pods by running the below command –
kubectl get pods -A

Figure 2: Resources deployed by the Fargate Open Source CDK Observability Accelerator Pattern
We can confirm that these pods run on separate fargate instances by fetching the nodes. Fargate nodes have the prefix fargate-ip
.
kubectl get nodes
Next, confirm whether the Grafana dashboards are deployed as expected.
Note: If you do not see Grafana dashboards, please check whether AMG_API_KEY
and the Secrets Manager parameter have been created as described above.
kubectl get grafanadashboards -A

Figure 3: Grafana dashboards deployed via FluxCD
EKS Fargate Infrastructure Monitoring
Login in to your Amazon Managed Grafana workspace and navigate to the dashboards panel. You should see a list of dashboards under the Observability Accelerator Dashboards
. Grafana Operator and Flux always work together to synchronize your dashboards with Git. If you delete your Grafana dashboards by accident, they will be re-provisioned automatically.

Figure 4: CDK Observability Accelerator Grafana Dashboards
Feel free to explore the other dashboards developed as part of the AWS observability accelerator. Open the Kubelet
dashboard and you should be able to view its visualization as shown below:

Figure 5: Kubelet dashboard from CDK Observability Accelerator
A fully managed EKS Fargate cluster with AWS-managed open-source observability is now available for deploying our workloads. Next, let us deploy additional components that reflect real-world applications you will run in a production environment. We will deploy an NGINX ingress controller and then a Java Tomcat application to monitor using the Grafana dashboards.
NGINX Ingress monitoring
EKS clusters often run multiple services in the same cluster, and incoming requests must be routed to the appropriate service. Ingress-nginx is an ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer that lets us do just that. It is a production-grade ingress controller that can facilitate multi-tenancy and segregation of workload ingresses based on hostname (host-based routing) and/or URL Path (path-based routing). It lets you expose your applications to end-users while providing basic functionalities, such as URL redirection, routing, and load balancing.
To deploy the ingress controller, we will be using the NginxAddOn
, which supports Classic Load Balancer (CLB), Network Load Balancer (NLB), or Application Load Balancer (ALB). We must modify our CDK code for the EKS Fargate OSS pattern to include the ingress controller.
- Navigate to
lib/single-new-eks-fargate-opensource-observability-pattern/index.ts
in your cloned git repository. - Add the CDK blueprints Quickstart NGINX addon to the
addOns
variable.
- Add another Fargate profile with the appropriate namespaces in the same file so the pods can be scheduled.
- Finally, we must update OpenTelemetry Collector to scrape NGINX metrics and add a new Grafana dashboard to visualize the data. This can be done by updating the
cdk.json
. Please pay attention tonginx.pattern.enabled
below for scraping and alerting of NGINX metrics via OpenTelemetry Collector.