Microsoft Workloads on AWS

.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 3 – Distributed Trace

Building a well-architected .NET application goes beyond just coding and deploying. You must monitor performance, trace transactions, collect logs, gather metrics, and trigger alarms when metrics breach thresholds. To achieve this, you can design and implement telemetry to enable observability capabilities.

In the first post of the series, I covered the implementation of metrics, and in the second post, I covered the implementation of logs. This post, the third and final in the series, explores the implementation of distributed tracing in .NET applications using Amazon CloudWatch and AWS X-Ray.

Distributed Trace is a key observability capability. It enables collecting and recording traces as requests propagate through the system. You can then use CloudWatch and AWS X-Ray to use these traces to generate maps that show how transactions flow across all services in the workload, helping you gain insight into the relationships between upstream and downstream components and identify or analyze issues in near real-time.

Solution overview

A system or platform in an actual production scenario could have dozens, hundreds, or even thousands of microservices, but for demonstration purposes, I’m using a smaller example application. It is a containerized .NET workload running on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate (Figure 1). It comprises three .NET microservices: an ASP.NET Core web API, two Worker Services, plus the cloud resources they use. Because of the distributed nature of this type of distributed system, in a failure scenario it might be challenging to figure out where to focus your investigation, or determine the relationship and dependency of the failing requests.

To gain deep insight into a containerized .NET workload like this one or more complex ones, you can instrument each application to collect and record distributed traces. For example, the first application that receives a customer request should generate the AWS X-Ray trace ID. In this sample, the web API generates the trace ID, adds metadata, and then propagates that same trace ID through the two other microservices via SNS Topics. Each of those will also add their own metadata and pass the same trace ID to the cloud components they interact with. Since they’re using the AWS SDK for .NET to communicate with these cloud components, the AWS SDK automatically adds the same trace ID to metadata and logs. As a result, CloudWatch and X-Ray can use these traces to aggregate all the events related to the trace ID and generate enriched views, maps, and timelines of the transaction to help accelerate the troubleshooting and visualization of the relationships, dependencies, and application’s health.

Architecture Diagram

Figure 1: .NET Microservice solution

Prerequisites

For this example, you can use the GitHub repository “microservices-dotnet-aws-cdk”, which contains the sample code for all three microservices. It uses the AWS Cloud Development Kit (CDK) to define and provision the Infrastructure as code using the C# programming language.

The following prerequisites are required on your system to test and deploy this solution:

You can clone the repository from the command line with the following command:

git clone https://github.com/aws-samples/microservices-dotnet-aws-cdk.git

Open the solution in an IDE, such as Visual Studio Code, to explore the implementation.

Implement tracing with AWS X-Ray in a .NET Application

I have added the NuGet package AWSXRayRecorder to the .NET projects in order to implement distributed tracing with AWS X-Ray. In the Program.cs file for each project, I initialized and registered AWS X-Ray. The initialization process is slightly different depending on the type of project. These code samples illustrate how to initialize AWS X-Ray for the web API and Worker Service projects.

This code sample demonstrates web application registration and initialization:

...
const string MY_SERVICE_NAME = "demo-web-api";
...
//Register X-Ray
AWSSDKHandler.RegisterXRayForAllServices();
...
var app = builder.Build();
//Initialize X-Ray
app.UseXRay(MY_SERVICE_NAME);
..
app.Run();

This sample demonstrates registration for the Worker Services:

…
IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices((hostContext, services) =>
    {
        …
  //Initialize & Register X-Ray
        AWSXRayRecorder.InitializeInstance(hostContext.Configuration);
        AWSSDKHandler.RegisterXRayForAllServices();
	  …
    })
    .Build();

await host.RunAsync();

Provision the AWS X-Ray daemon

To transmit all the traces to the AWS X-Ray API, you need to deploy the AWS X-Ray daemon, a software application that listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray service. The daemon must be running continuously so that data sent by the application components can reach the X-Ray service. The installation may vary depending on the AWS services you’re using for your .NET Application.

In the following code sample, I’m provisioning the X-Ray daemon sidecar using C# with AWS CDK. AWS CDK lets you build applications in the cloud as code, with the benefit of the expressive power of programming languages, such as C#, which I am using here. The daemon runs alongside the application containers on Amazon ECS with AWS Fargate.

public static TaskDefinition AddXRayDaemon(this TaskDefinition taskDefinition, XRayDaemonProps xRayDaemonProps)
{
    taskDefinition.AddContainer("x-ray-daemon", new ContainerDefinitionOptions
    {
        ...
        PortMappings = new PortMapping[]{
            new PortMapping{
                ContainerPort = 2000,
                Protocol = Protocol.UDP
            }},
        Image = ContainerImage.FromRegistry("public.ecr.aws/xray/aws-xray-daemon:latest"),
        ...
    });

     //Grant permission to write X-Ray segments
    taskDefinition.TaskRole
        .AddManagedPolicy(ManagedPolicy.FromAwsManagedPolicyName("AWSXRayDaemonWriteAccess"));

    return taskDefinition;
}

Deploy the example solution

The microservices-dotnet-aws-cdk GitHub repository contains the full implementation of this solution, allowing you to make an HTTP request to the sample Web API to test it. To deploy, run one of the following deployment scripts in your environment using either bash (Linux or Mac) or PowerShell to deploy the solution.

Using bash:

./deploy.sh

Using PowerShell:

.\deploy.ps1

After deploying, copy the URL printed by the deployment script. It has the following format: http://WebAp-demos-xxxxxxxxxx-99999999.us-west-2.elb.amazonaws.com/api/Books. The Xs and 9s will be alphanumeric characters representing your deployment’s unique ID. Then, using a REST API client (such as Thunder Client for VS Code), test the solution by submitting an HTTP POST request to the URL with the following JSON payload. When you submit the HTTP POST, you should receive a response of status 200, and the TraceId result. Copy the TraceId value for later use.

{
    "Year" : 2022,
    "Title": "Demo book payload",
    "ISBN": 12345612,
    "Authors": ["Author1", "Author2"],
    "CoverPage": "picture1.jpg"
}

Figure 2 illustrates the example API call.

Example API call

Figure 2: Example API call

Visualize the results

After making a test request to the API, you can navigate to Amazon CloudWatch to review the generated Map of the System with all the microservices and cloud components.

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose X-Ray Trace, Traces.
  3. Leave the Filter by X-Ray Group blank, enter the trace ID you’ve copied on the following field, and choose Run query.

The console will present a page similar to Figure 3 and Figure 4. CloudWatch will aggregate all the data for the given trace ID and display a distributed trace map showing:

  • All three microservices
  • The cloud components each one interacted with during the request
  • Metrics for segments
  • The timeline of the request, including the status, response code, and duration for each communication with downstream components

Distributed Trace Map

Figure 3: Distributed Trace Map

Trace Logs

Figure 4: Trace logs

Cleanup

You should clean up the resources created by running this demo to avoid unexpected charges. To do so, run the script from the root folder where you’ve cloned the GitHub repository.

Using bash:

./clean.sh

Using PowerShell:

.\clean.sh

Conclusion

In this final post in the blog post series on implementing observability for your .NET applications on AWS, I’ve demonstrated how to implement distributed tracing using AWS X-Ray on a .NET workload to collect, record, and transmit trace events to AWS X-Ray. I also demonstrated how Amazon CloudWatch uses these traces generated from your application to show the map, relationship, and dependencies of all the system’s microservices.

To learn about metrics or logging implementations, check out the other two parts of this series: “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 1 – Metrics” and “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 2 – Logging”.

Observability goes beyond instrumenting your .NET application to generate traces, logs, and metrics. You can use Amazon CloudWatch capabilities for monitoring, alarms, detecting anomalies, and more. See the Amazon CloudWatch Features page to learn more, and for hands-on experience, check out the One Observability Workshop.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your migration journey today.

TAGS: , ,
Ulili Nhaga

Ulili Nhaga

Ulili Nhaga is a Cloud Application Architect at Amazon Web Services in San Diego, California. He helps customers migrate, modernize, architect, and build highly scalable cloud-native applications on AWS. Outside of work, Ulili loves playing soccer, running, cycling, Brazilian BBQ, and enjoying time on the beach.