Amazon CloudWatch Insights for Amazon EKS on EC2 using AWS Distro for OpenTelemetry Helm charts

This blog provides a simplified three-step solution to collect metrics and logs from an Amazon Elastic Kubernetes Service (Amazon EKS) cluster on Amazon Elastic Compute Cloud (Amazon EC2) using the AWS Distro for OpenTelemetry (ADOT) Helm charts repository and send them to Amazon CloudWatch Logs and Amazon CloudWatch Container Insights. The ADOT Helm charts repository contains Helm charts to provide easy mechanisms to set up the ADOT Collector and other collection agents like fluentbit to collect telemetry data such as metrics, logs and traces to send to AWS monitoring services.

Amazon EKS is a managed Kubernetes service that makes it easy for organizations to run Kubernetes on AWS Cloud and on premises. Organizations use Amazon EKS to automatically manage the availability and scalability of the Kubernetes control plane nodes responsible for scheduling containers, managing application availability, storing cluster data, and performing other key tasks. ADOT is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Applications can set up ADOT Collector and other collector agents only once to send correlated metrics and traces to multiple AWS and Partner monitoring solutions. Fluent Bit is an open-source log processor and forwarder that you can use to collect data such as metrics and logs from different sources. Helm deploys packaged applications to Kubernetes and structures them into Helm charts.

Solution overview

A high-level architecture diagram depicted in Figure 1 shows a simple solution for collecting metrics and logs to send to Amazon CloudWatch Container Insights by installing an ADOT Helm chart on your existing or new Amazon EKS cluster.

Here are the steps to set up an ADOT and fluentbit collector:

1. Set up your environment and install the necessary tools to connect to an existing or newly created Amazon EKS cluster.
2. Configure the necessary roles for AWS Identity and Access Management (IAM) roles for service accounts and install Helm charts for ADOT, enabling fluentbit.
3. Monitor logs, metrics, and traces from Amazon CloudWatch Logs and Container Insights.

Figure 1. Architecture diagram for Helm chart installation of ADOT and fluentbit to an existing Amazon EKS cluster

Prerequisites

• Existing AWS account with access to AWS Management Console
• Intermediate-level knowledge and understanding of Amazon EKS
• An existing or new Amazon EKS cluster

Install the tools

In this blog, AWS Cloud9 is used as an environment to connect to the Amazon EKS cluster and install Helm charts. If you choose to use AWS Cloud9, follow the step-by-step instructions provided in Creating an EC2 Environment. Refer to Getting started with Amazon EKS for additional instructions to install eksctl, create EKS clusters, and set up required IAM permissions for connecting to an EKS cluster.

1. Log in to your Amazon EKS cluster and inspect the cluster. Select an EKS cluster in AWS Management Console. On the Resources tab, check the DaemonSets, as in Figure 2a.
   

   Figure 2a. EKS cluster DaemonSets

2. Open Amazon CloudWatch and inspect the Log groups and Amazon CloudWatch Container Insights. Note that the Log groups and Amazon CloudWatch Container Insights in Figure 2b do not show any EKS cluster-specific logs.
   

   Figure 2b. Container Insights before ADOT and fluentbit collector installation

Install Helm and configure IAM roles

1. Run the following command to install Helm, verify the version, and configure Bash completion for the Helm command:

   curl -ssl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
   helm version --short
   
   helm completion bash >> ~/.bash_completion
   . /etc/profile.d/bash_completion.sh
   . ~/.bash_completion
   source <(helm completion bash)

2. Set up IAM roles for service accounts.
   Replace XXX in the following commands with your EKS Cluster name.

   eksctl create iamserviceaccount \
   --name fluent-bit \
   --role-name EKS-ADOT-CWCI-Helm-Chart-Role-CW \
   --namespace amazon-cloudwatch \
   --cluster XXX \
   --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
   --role-only \
   --approve
   

   eksctl create iamserviceaccount \
   --name adot-collector-sa \
   --role-name EKS-ADOT-CWCI-Helm-Chart-Role-METRICS \
   --namespace amazon-metrics \
   --cluster XXX \
   --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
   --role-only \
   --approve
   

3. Deploy the ADOT Helm chart.
   Replace XXX in the following code with your EKS Cluster name.

   CWCI_ADOT_HELM_ROLE_ARN_CW=$(aws iam get-role --role-name EKS-ADOT-CWCI-Helm-Chart-Role-CW | jq .Role.Arn -r)
   CWCI_ADOT_HELM_ROLE_ARN_METRICS=$(aws iam get-role --role-name EKS-ADOT-CWCI-Helm-Chart-Role-METRICS | jq .Role.Arn -r)
   helm repo add adot-helm-repo https://aws-observability.github.io/aws-otel-helm-charts
   helm install adot-release adot-helm-repo/adot-exporter-for-eks-on-ec2  \
   --set clusterName=XXX --set awsRegion=us-east-1 --set fluentbit.enabled=true \
   --set adotCollector.daemonSet.service.metrics.receivers={awscontainerinsightreceiver} \
   --set adotCollector.daemonSet.service.metrics.exporters={awsemf} \
   --set adotCollector.daemonSet.cwexporters.logStreamName=EKSNode \
   

4. Run the following commands to validate the successful deployment.
   • Verify that two new namespaces have been created.
     kubectl get ns
     The result should be:

     $ kubectl get ns
     NAME                STATUS           AGE
     amazon-cloudwatch   Active           2d20h
     amazon-metrics      Active           2d20h

   • Verify that a fluentbit pod was enabled as part of the ADOT Helm Chart under the amazon-cloudwatch namespace.
     kubectl get all -n amazon-cloudwatch
     The result should be:

     kubectl get all -n amazon-cloudwatch
     NAME                   READY   STATUS    RESTARTS   AGE
     pod/fluent-bit-9lrnt   1/1     Running   0          2d20h
     pod/fluent-bit-h9lvt   1/1     Running   0          2d20h
     pod/fluent-bit-nbqjm   1/1     Running   0          2d20h
     
     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE

   • Verify the adot-collector-pod under the amazon-metrics namespace.
     kubectl get all -n amazon-metrics
     The result should be:

     $ kubectl get all -n amazon-metrics
     NAME                                 READY   STATUS    RESTARTS   AGE
     pod/adot-collector-daemonset-6qcsd   1/1     Running   0          2d20h
     pod/adot-collector-daemonset-f92fr   1/1     Running   0          2d20h
     pod/adot-collector-daemonset-gmhbx   1/1     Running   0          2d20h
     
     NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
     daemonset.apps/adot-collector-daemonset   3         3         3       3            3           <none>          2d20h

5. Validate the installation through the Amazon EKS cluster.
   Go to the Amazon EKS cluster and select the Resources tab. Under Workloads, select DaemonSets, and find the fluent-bit and adot-collector-daemonsets as demonstrated in Figure 3.
   

   Figure 3. DaemonSet under Amazon EKS cluster resources

Monitor logs, metrics, and traces

Monitor the CloudWatch Logs and CloudWatch Insights.

• In the Logs section, choose Log groups to view Amazon EKS cluster log groups with a prefix of /aws/containerinsights, as in Figure 4a.
  

  Figure 4a. EKS cluster log groups

• In the Insights section, choose Container Insights to view all the resources within your Amazon EKS cluster, as in Figure 4b.
  

  Figure 4b. EKS cluster’s Container Insights resources

• On the Container Insights page, select Container map from the dropdown to check the container map for Amazon EKS clusters, as demonstrated in Figure 4c.
  

  Figure 4c. EKS cluster’s Container Insights container map

• On the Container Insights page, select Performance monitoring from the dropdown to view various performance metrics for Amazon EKS cluster, as demonstrated in Figure 4d.
  

  Figure 4d. EKS cluster’s Container Insights performance monitoring

Cleanup

If you are no longer using the resources discussed in this blog, remove the excess AWS resources to avoid incurring charges. After you finish setting up ADOT and fluentbit collectors to send logs and metrics to Amazon CloudWatch Logs and Container Insights, clean up resources by uninstalling the ADOT Helm chart, deleting IAM Roles created for the services, deleting CloudWatch Logs, and deleting Container Insights.

Conclusion

In this blog we walked through a simple three-step solution to set up Amazon EKS cluster logs and Container Insights using Helm charts. The Helm chart installs ADOT and fluentbit as a DaemonSet in the existing EKS cluster to collect and port logs, metrics, and traces to Amazon CloudWatch Logs and Container Insights. The Amazon CloudWatch Container Insights provide insights into resources, monitor performance, and container map of all the resources within the Amazon EKS cluster.