How eCloudvalley Enables Monitoring and Observability in Amazon EKS with AWS Distro for OpenTelemetry

By Geoffrey Aaron, Sr. Solutions Architect, Regional – eCloudvalley
By Ken Lam, Solution Lead, Cloud Native Development – eCloudvalley
By Chris Zheng, Sr. Partner Solutions Architect, App Modernization – AWS

eCloudvalley

Monitoring containerized applications can be challenging due to the isolated environment that containers provide. This makes it difficult to monitor the performance and health of applications, especially when there’s a need to manage thousands or tens of thousands of containers in Amazon Elastic Kubernetes Service (Amazon EKS).

To overcome these challenges, a comprehensive monitoring strategy is essential, including how to choose tools to collect and analyze resource usage, network traffic, and application logs. With the right monitoring tools and strategies in place, companies can ensure the performance, availability, and scalability of their containerized applications in production environments.

eCloudvalley is an AWS Premier Tier Services Partner and Managed Service Provider (MSP) with the Amazon EKS service delivery specialization. eCloudvalley has delivered hundreds of projects for customers on Amazon Web Services (AWS), and with AWS Distro for OpenTelemetry it can help customers centralize their monitoring in Amazon CloudWatch for their EKS environment.

In this post, we will discuss eCloudvalley’s experience setting up AWS Distro for OpenTelemetry for a customer in the healthcare industry. Since this sector has been historically cautious when it comes to technology adoption, the system’s development cycle and orchestrator tools were the main factors needed to achieve the transformation the customer wanted.

Business Case

eCloudvalley’s customer believes in using technology and innovation to transform the way people access healthcare and are providing a seamless integration of the online and offline healthcare experience. With multi-region customers around ASEAN, they need to ensure the security, stability, agility, availability, and consistency of their system—from development to production.

The company’s vision requires them to overcome several pain points:

They are currently deploying their entire system manually, which consumes a considerable amount of time and effort. This process can cause interruptions to business outcomes, especially if they need to expand into new markets.
The organization struggles to meet sudden surges in user traffic and requests. Scaling their resources requires several employees to stand by, which is ineffective and impacts the system’s consistency.
Updating major and minor versions of the system involves significant manual work and can cause disruptions leading to poor user interaction.

Troubleshooting is challenging since the company lacks proper logging and monitoring tools. Additionally, their limited understanding of the development cycle makes it difficult to achieve their vision of a system with complex requirements.

Solution Overview

With eCloudvalley’s standard approach, infrastructure as code (IaC) with the AWS Cloud Development Kit (AWS CDK) helps reduce manual deployment and minimize human error when provisioning resources.

In addition, deploying workloads on Amazon EKS can handle complex systems with a lot of microservices, without worrying about managing the cluster itself. By utilizing EKS with Kubernetes autoscaler, workload capacity can be automated based on user demand.

For troubleshooting purposes, Amazon CloudWatch can be integrated with AWS Distro for OpenTelemetry and FluentBit. Setting up an AWS CodePipeline ensures an agile and smooth deployment cycle and is important to advise scanning applications inside the image with ECR scanning, Trend Micro, or similar tools for security purposes.

Figure 1 – High-level architecture.

How AWS Distro for OpenTelemetry Can Help

Normally, when users want to collect data they need to decide which endpoint to use to collect specific data. AWS Distro for OpenTelemetry plays a crucial role in this regard, reducing the complexity and correlate those data.

OpenTelemetry provides open-source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. With AWS Distro for OpenTelemetry, you can collect metadata from your AWS resources and managed services to correlate application performance data with underlying infrastructure data, reducing the mean time to problem resolution.

By implementing AWS Distro for OpenTelemetry, logs, metrics, and traces are collected through single collector binary and OpenTelemetry protocol (OTLP) in the AWS Distro for OpenTelemetry agent. You can stream those multiple sources to a single or multiple designated destinations instead of trying to collect them from multiple agents within the application.

Figure 2 – How AWS Distro for OpenTelemetry can help.

AWS Distro for OpenTelemetry is one of eCloudvalley’s standard implementations of collector agents to aggregate logs and metrics from multiple containers and pods. The diagram below illustrates what it looks like in an EKS environment.

Figure 3 – AWS Distro for OpenTelemetry installation and data collection flow.

Deploying a DaemonSet with a Helm chart allows for the deployment of an agent on each node, ensuring that all nodes have the required agent.

Install Helm Chart for AWS Distro for OpenTelemetry

In this walkthrough, we adopt AWS CDK 2.0 to implement IaC. The code snippet below installs the Helm Chart in AWS CDK 2.0.

class ContainerInsight extends cdk.NestedStack {
  constructor(scope: Construct, id: string, props: ContainerInsightProps) {
    super(scope, id, props);

    const { cluster } = props;

    new eks.HelmChart(this, "AdotExporter", {
      cluster: cluster,
      chart: "adot-exporter-for-eks-on-ec2",
      version: "0.1.0",
      release: "aws-distro-open-telemetry",
      repository: "https://aws-observability.github.io/aws-otel-helm-charts",
      values: {
        clusterName: cluster.clusterName,
        fluentbit: {
          enabled: false,
        },
      },
    });
    new eks.HelmChart(this, "FluentBit", {
      cluster: cluster,
      chart: "aws-for-fluent-bit",
      version: "0.1.14",
      release: "fluent-bit",
      namespace: "amazon-cloudwatch",
      repository: "https://aws.github.io/eks-charts",
      values: {
        cloudWatch: {
          region: "ap-southeast-1",
        },
        firehose: {
          enabled: false,
        },
        kinesis: {
          enabled: false,
        },
        elasticsearch: {
          enabled: false,
        },
      },
    });
  }
}

By running code above, we establish the agent of AWS Distro for OpenTelemetry as the collector of metrics. Since the feature of AWS Distro for OpenTelemetry to collect logs and send to CloudWatch are still in development, eCloudvalley chose to leverage FluentBit as collector for logs.

AWS Distro for OpenTelemetry SDK can be used to send data to the collector from the application. This helps to set up the connection between the application and agent collector with OTLP protocol by gRPC or HTTP.

There are three application initializations steps that have been done to establish the connections:

OpenTelemetry receiver sends data to ADOT collector.
OpenTelemetry processor will process the data.
OpenTelemetry exporter decides the type of export data based on destination.

eCloudvalley initialized the OpenTelemetry receiver with “otlpreceiver” as general receiver, followed by initializing general OpenTelemetry processor based on customer needs (for example, “attributesprocessor”). Lastly, the team initialized the OpenTelemetry exporter with “otlpexporter” as general exporter.

Summary

Comprehensive monitoring and observability are important factors in maintaining the stability and availability of cloud-based applications. eCloudvalley can help instrument your application with AWS Distro for OpenTelemetry, aligning your systems to your specific needs and vision.

For more information about setting up AWS Distro for OpenTelemetry in advance, contact eCloudvalley. You can also learn more about eCloudvalley in AWS Marketplace.

.

.

eCloudvalley – AWS Partner Spotlight

eCloudvalley is an AWS Premier Tier Services Partner and MSP with a mission to evolve customers’ businesses in the cloud. eCloudvalley has been the trusted cloud advisor of 1,000+ customers by a team of cloud experts with 500+ AWS Certificates.

Contact eCloudvalley | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog