.NET Observability with OpenTelemetry – Part 1: Metrics using Amazon Managed Prometheus and Grafana

Microservice architectures, while modular and scalable, introduce complexity in observability due to the distributed nature of services. This often leads to a proliferation of inter-service communication paths and dependencies.

It is not uncommon for solutions to have microservices interacting with each other, native cloud services, and partner solutions. In distributed microservice architectures, developers face the challenge of identifying where exceptions occur along the request path and collecting relevant context. A popular approach to ease this challenge is through distributed request tracing.

In this first post of the series, you are going to learn about the implementation of OpenTelemetry in an ASP.NET website to capture application metrics. The subsequent blog posts will cover the implementation of logs using FluentBit and Amazon OpenSearch Service, and distributed tracing in .NET applications using OpenTelemetry and AWS X-Ray (X-Ray). To get the most out of this series, you should have a grasp of OpenTelemetry principles as we will be discussing how to integrate and leverage its telemetry capabilities within a .NET environment.

Note that although this post uses an Amazon Elastic Container Service (Amazon ECS) environment, the concept also applies to Amazon Elastic Kubernetes Service (Amazon EKS).

Solution Architecture Overview

This blog series uses a sample C# .NET application hosted in the aws-samples GitHub repository. The application will run in an Amazon ECS cluster using Amazon Elastic Compute Cloud (Amazon EC2) launch type.

The AWS Distro for OpenTelemetry (ADOT) collector is deployed as an isolated task on Amazon ECS Fargate. The ADOT collector has metrics and traces pipelines, as shown in figure 1, to push data to Amazon Managed Service for Prometheus and X-Ray. For monitoring and analysis, metrics are visualized using Amazon Managed Grafana, traces in X-Ray, and logs in Amazon OpenSearch. This first post will focus on monitoring and analysis of metrics.

Figure 1: An architecture diagram of the solution. A user interacts with the application and telemetry artifacts are sent to their respective destinations

Prerequisites

The following prerequisites are required on your system to test and deploy this solution:

.NET 6.0 Software Development Kit (SDK)
The latest AWS Command Line Interface (AWS CLI)
AWS CDK v2
Docker
Visual Studio Code (or your preferred IDE )
AWS credentials for your AWS account configured locally
Git
AWS Organizations configured through AWS IAM Identity Center (Successor to AWS Single Sign-On).

Walkthrough

With the prerequisites satisfied, the next steps will guide you in deploying the application and ADOT services and visualize metrics.

Download the aws-dotnet-ecs-contains-observability sample. It contains an instrumented sample application that makes API calls to Amazon.com and a CloudFormation Template to deploy the infrastructure.

Prepare your Environment

We will use the CloudFormation Template found in the project files to deploy the following resources in your account:

Amazon ECS cluster to run container images.
Application Load Balancer and target group to distribute traffic to the application.
Amazon Managed Service for Prometheus workspace to store metrics from the ADOT collector.
Amazon OpenSearch Service to store logs from the AWS for Fluent Bit sidecar.
Amazon Managed Grafana workspace to aggregate and visualize metrics, traces, and logs.
AWS Identity and Access Management (IAM) roles to provide Amazon ECS tasks with write permissions to AWS X-Ray, Amazon Managed Service for Prometheus, and Amazon OpenSearch Service.

Browse to the location where you downloaded the sample code. Navigate to BlogSample-ASPDotNetApp/BlogResources folder and run the following command to deploy the CloudFormation template. We recommend deploying this in a test environment as not to interfere with production workloads.

aws cloudformation deploy --template ./blog-cf-template.yml --stack-name observability-solution-stack --capabilities CAPABILITY_NAMED_IAM

Navigate to CloudFormation in the AWS management console and browse the new stack to see the outputs needed in the upcoming steps. The CloudFormation stack will take up to 15 minutes to provision the resources.

Figure 2: Outputs generated by the CloudFormation stack

Inspecting our .NET Application

We will use OpenTelemetry (OTEL) to collect metrics from your application. In this section, inspect OpenTelemetry integration in the sample .NET application.

The application is available as an image on the Amazon Elastic Container Registry (Amazon ECR) public gallery for the deployment.

Inspect the NuGet Packages

AWS Distro for OpenTelemetry is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. AWS updates and manages ADOT but relies on OpenTelemetry to support and maintain the libraries used in the source code. The ADOT collector allows you to use a standardized set of open-source APIs, SDKs, and agents to instrument applications once, and then send correlated metrics and traces to multiple AWS and partner monitoring solutions. The ADOT collector also collects metadata from your AWS resources and managed services, letting you measure application performance against infrastructure data and speeding up the time needed to resolve problems.

Browse to the location where you downloaded the sample application. Open the BlogSample-ASPDotNet.sln solution file using Visual Studio or your preferred integrated development environment (IDE). You can find the solution file in the main folder of the sample application. Inside the solution, there is a Startup.cs file that includes essential packages and code for sending data to the ADOT collector. If you’re interested in creating your own solution, examine the packages specified in the solution file for reference.

Figure 3: Adding the required OpenTelemetry NuGet packages

Inspect the Code used to Instrument the Application with OpenTelemetry

Simply adding the packages does not instrument your application. OpenTelemetry services are required in your project on startup. In the sample application, this is done in Startup.cs, however, implement this code where you configure IServiceCollection.

This code helps instrument the application for monitoring and performance analysis by allowing the application to generate metrics and traces. Metrics provide insights into various performance indicators such as server response times, memory usage, and error rates. Traces consist of interconnected spans, each representing a specific task or action in a service or module. When a request starts in a service, its trace information is sent with it by including trace headers in the request. This allows other services to participate in the same trace.

...
 
#region OpenTelemetry

        var serviceName = "sample-app";
        var serviceVersion = "1.0";
    
        var appResourceBuilder = ResourceBuilder.CreateDefault()
                .AddService(serviceName: serviceName, serviceVersion: serviceVersion);
    
        //Configure important OpenTelemetry settings, the console exporter, and instrumentation library
    
        var meter = new Meter(serviceName);
        services.AddSingleton<Meter>(meter);
        services.AddOpenTelemetry().WithMetrics(metricProviderBuilder =>
        {
           metricProviderBuilder
               .AddConsoleExporter()
               .AddOtlpExporter(options =>
               {
                    options.Protocol = OtlpExportProtocol.Grpc;
                    options.Endpoint = new Uri(Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT"));
                })
                 .AddMeter(meter.Name)
                 .SetResourceBuilder(appResourceBuilder)
                 .AddAspNetCoreInstrumentation()
                 .AddHttpClientInstrumentation();

        });
    
        services.AddOpenTelemetry().WithTracing(tracerProviderBuilder =>
        {
            tracerProviderBuilder
                .AddConsoleExporter()
                .AddOtlpExporter(options =>
                {
                    options.Protocol = OtlpExportProtocol.Grpc;
                    options.Endpoint = new Uri(Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT"));
    
                })
                .AddSource(serviceName)
                .SetResourceBuilder(appResourceBuilder.AddTelemetrySdk())
                .AddXRayTraceId() // Creates an Xray Compatible Trace Id.
                .AddAWSInstrumentation()
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation();
    
        });`
    
        Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());
    
        
    
#endregion
...

Deploy the ADOT Collector to Amazon ECS Fargate

This section describes how to configure the ADOT collector to receive metrics from your .NET app, and export the metrics to Amazon Managed Service for Prometheus. It is deployed as an isolated task on ECS Fargate.

Inspect the ADOT Collector Configuration

To start, the ADOT collector requires a configuration ingress and egress telemetry signals.

Below is the receiver configuration for ADOT found in Parameter Store, a capability of AWS Systems Manager. Parameter Store provides secure, hierarchical storage for configuration data management and secrets management.

receivers:
   otlp:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317


 processors:
   batch:


 exporters:
   prometheusremotewrite:
     endpoint: <YOUR-APS-WORKSPACE-ENDPOINT>/api/v1/remote_write
     auth:
       authenticator: sigv4auth
   awsxray:
    region: <YOUR-REGION>
   logging:
     loglevel: debug


 extensions:
   sigv4auth:
     region: <YOUR-REGION>
     
 service:
   extensions: [sigv4auth]
   pipelines:
     traces:
       receivers:  [otlp]
       processors: [batch]
       exporters:  [awsxray]
     metrics:
       receivers:  [otlp]
       processors: [batch]
       exporters:  [prometheusremotewrite]

The ADOT Collector will pull configuration directly from SSM instead of the local filesystem, removing the need to make a custom image. This approach is documented in Use custom OpenTelemetry configuration file from SSM Parameter. Refer to the ADOT Collector Configuration documentation for configuration syntax and options.

Launch ADOT Fargate Service

Find the service definition of adot-service under the service-definitions folder. Update the subnets key with the IDs of your private subnets and the securityGroups key with the ADOTSecurityGroupId. Retrieve these IDs from the output tab of our stack in the CloudFormation console.

{
     "cluster": "BlogCluster",
     "deploymentConfiguration": {
         "maximumPercent": 200,
         "minimumHealthyPercent": 0
     },
     "deploymentController": {
         "type": "ECS"
     },
     "desiredCount": 1,
     "enableECSManagedTags": true,
     "enableExecuteCommand": true,
     "launchType": "FARGATE",
     "networkConfiguration": {
         "awsvpcConfiguration": {
             "assignPublicIp": "ENABLED",
             "securityGroups": [
                "<ADOTSecurityGroupId>"
             ],
             "subnets": [
               "<BlogPrivateSubnetAz1Id>",
               "<BlogPrivateSubnetAz2Id>"
             ]
            }
     },
    ...

Launch the ADOT service through the console or using the following command:

aws ecs create-service --cluster BlogCluster --cli-input-json file://adot-service.json

After the completion of this command, you will see your ADOT Service running on your cluster.

Figure 4: The ADOT service running on a ECS cluster

Deploy your Application Service

The next step is to deploy the Application service on ECS. Open the sample-app-service.json and update the “TargetGroupArn” to the target group found in the CloudFormation stack outputs.

...
         "loadBalancers": [
         {

            "targetGroupArn": "<TargetGroupArn>",
             "loadBalancerName": "",
             "containerName": "sample-app",
             "containerPort": 80
         }
     ],
 ...

Launch application service either through the console or by navigating into the service-definitions folder and run the following command:

aws ecs create-service --cluster BlogCluster --cli-input-json file://sample-app-service.json

For inter-service communication, we use AWS CloudMap. Under the hood, AWS Cloud Map uses Route 53 and creates DNS records in a private hosted zone for each registered task.

If everything is configured properly then both services will be running. Next, we will visit the load balancer endpoint for accessing our application.

Figure 5: Both Services Running on the cluster

Access and Navigate the Application

This section explains how to access and navigate to the application to generate signals to send to the endpoints.

Open the cluster in the Amazon ECS console and select app-service. Select the networking tab and then copy the DNS name of the load balancer to access your application. Paste the URL in your browser to access the application and begin generating metrics.

Figure 6: Access your load balancer’s endpoint

Figure 7: The application home page

Create an IAM Identity Center User for Grafana Access

To visualize our metrics in Amazon Managed Service for Prometheus, we will use Amazon Managed Grafana which were provisioned earlier from the CloudFormation stack. But first we need to create a user to access it.

Users are authenticated to use the Grafana console in an Amazon Managed Grafana workspace by single sign-on using your organization’s identity provider. For this post, you will use AWS IAM Identity Center to authenticate a user. For more information, see Managing user and group access to Amazon Managed Grafana.

Navigate to Users in Amazon IAM Identity Center and create a new user. This user will be associated with our Amazon Grafana workspace.

Figure 8: : Creating a new user for our Amazon Managed Grafana workspace

Access the Grafana Workspace

After a user is created, open your workspace in the Amazon Managed Grafana console. Under authentication, assign a new user or group to the workspace and edit the user type from viewer to Admin. This will give you permissions to connect your data sources to Grafana.

Figure 9: Assign a user to the workspace

Figure 10: Adding a user to the Amazon Managed Grafana workspace as an admin

Add your Amazon Prometheus Data Source

Navigate to the Amazon Managed Grafana workspace URL defined in the CloudFormation stack output and login using the username and password you entered for your user.

Expand the collapsed menu bar and navigate to Data Sources under Apps. Select Amazon Managed Service for Prometheus and select your region to find your workspace to add.

Your metrics now appear in the explore section.

Figure 11: Application Metrics are visible in Amazon Managed Grafana

Clean Up

Clean up the resources created in this tutorial. It is a good practice to delete resources that you are no longer using. By deleting a resource you’ll also stop incurring charges for that resource.

Open the Amazon ECS console stop the App and adot services on your Cluster.
Open CloudFormation in the AWS console and delete the template deployed earlier. This will delete the resources provisioned which can take 10-15 minutes. If any resources fail to delete, they can be manually deleted.

Conclusion

In this post we showed how to instrument a .NET application to generate metrics. We then demonstrated how to use the ADOT collector to send metrics to Amazon Managed Service for Prometheus and use Amazon Managed Grafana for analyzing and monitoring.

Observability doesn’t stop at instrumenting your .NET application with metrics, logs, and traces. Use the capabilities of Amazon Managed Grafana to create dashboards for monitoring, detecting anomalies, and generating alerts. Visit the Amazon Managed Grafana features and Amazon Managed Service for Prometheus features pages to learn more.

To learn about centralized logging or distributed tracing implementations, read the upcoming Part 2 and Part 3 of this series.

.NET on AWS Blog