Enhancing Observability in the AWS SDK for .NET with OpenTelemetry

Starting with version 3.7.400, the AWS SDK for .NET added significant observability enhancements. It introduced powerful tracing and metrics capabilities with OpenTelemetry support, an industry-standard for observability. With these enhancements, developers can now gain deeper insights into their applications’ behavior, from tracking API call durations to monitoring system metrics.

In this blog post, we’ll guide you through the new observability features in the AWS SDK for .NET, starting with how to easily integrate OpenTelemetry for comprehensive tracing and metrics. We’ll then explore how you can leverage the TelemetryProvider to register your own custom implementation, providing you with full control over your application’s telemetry data.

Simplifying Observability with OpenTelemetry

As applications become increasingly complex, achieving comprehensive observability is essential for maintaining reliability, understanding performance, and diagnosing issues in production. This is particularly crucial in distributed environments, where pinpointing the root cause of problems can be challenging.

For most developers, the quickest path to enhanced observability is through OpenTelemetry, a widely adopted standard that simplifies the collection and processing of telemetry data. That’s why we’ve made it easier to integrate with OpenTelemetry via the OpenTelemetry.Instrumentation.AWS NuGet package.

The Difference Between OpenTelemetry Packages

When integrating OpenTelemetry, you might notice two different packages related to AWS: OpenTelemetry.Instrumentation.AWS and OpenTelemetry.Contrib.Instrumentation.AWS. It’s important to know the difference:

OpenTelemetry.Instrumentation.AWS is the latest package designed to provide comprehensive tracing and metrics support for the AWS SDK for .NET. This package is currently in beta, which means it might not appear in NuGet searches unless you have enabled the option to include preview versions.
OpenTelemetry.Contrib.Instrumentation.AWS is an older package that provided basic instrumentation that captures only fundamental trace information and does not provide any metrics instrumentation. No new development will be done on this package as development has moved to the OpenTelemetry.Instrumentation.AWS NuGet package.

Do not use OpenTelemetry.Contrib.Instrumentation.AWS for new or ongoing development because it will be soon deprecated. Instead, use OpenTelemetry.Instrumentation.AWS to leverage its comprehensive tracing and metrics capabilities.

Integrate OpenTelemetry

Let’s walk through creating a simple .NET console application that interacts with Amazon S3. This example will demonstrate how to observe and monitor your application using OpenTelemetry, including handling exceptions and tracking metrics.

First, create a new console application and add the S3 package

dotnet new console --framework net8.0 --name DemoOtel
cd .\DemoOtel\
dotnet add package AWSSDK.S3

Next, replace the code in Program.cs with the following:

using Amazon.S3;

var s3Client = new AmazonS3Client();

try
{
    var listBucketsResponse = await s3Client.ListBucketsAsync();
    var deleteBucketResponse = await s3Client.DeleteBucketAsync("amzn-s3-demo-bucket");
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

Console.Read();

Execute the application using the dotnet run command, and you should see the following output:

The specified bucket does not exist

This exception occurs because we attempted to delete a bucket that doesn’t exist. In a production environment, such exception could easily be missed if not logged properly. This is where OpenTelemetry comes in handy, allowing you to trace and monitor your application effectively.

Adding OpenTelemetry Tracing

Let’s enhance our application with OpenTelemetry to observe traces, including the exception that occurred.
First, add the necessary OpenTelemetry packages:

dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Instrumentation.AWS --prerelease

To configure OpenTelemetry tracing, you will add the following code snippet to your application. (To see where, look at the full code later in this section.)

Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddService("DemoOtel"))
    .AddAWSInstrumentation()
    .AddConsoleExporter()
    .Build();

The AddAWSInstrumentation method shown in the preceding code snippet automatically registers the OpenTelemetry-based tracing providers. This method is designed to be simple and straightforward, allowing you to quickly add OpenTelemetry to your application without worrying about the underlying implementation details.

With the AddConsoleExporter method, we’ll use the Console exporter for simplicity, but you can explore more advanced exporters that suit your needs at OpenTelemetry Exporters.

Following is the full version of Program.cs with the OpenTelemetry setup that includes the AWS instrumentation and console exporter:

using Amazon.S3;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddService("DemoOtel"))
    .AddAWSInstrumentation()
    .AddConsoleExporter()
    .Build();

var s3Client = new AmazonS3Client();

try
{
    var listBucketsResponse = await s3Client.ListBucketsAsync();
    var deleteBucketResponse = await s3Client.DeleteBucketAsync("amzn-s3-demo-bucket");
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

Console.Read();

Run the application again using dotnet run. This time, you’ll see trace information in the console. The traces will include details about the S3.ListBuckets and S3.DeleteBucket operations, such as their duration and any associated exceptions.

Following is a screenshot of the terminal output, highlighting the exception:

Additionally, you’ll see traces for HttpRequest and CredentialsRetrieval, with the S3.ListBuckets or S3.DeleteBucket traces as their parents. While the console output is useful, visualizing these traces using a more advanced exporter like the ones listed in OpenTelemetry Exporters can make it easier to understand the relationships between them.

Following is an example of how the traces look after visualizing them in Amazon Managed Grafana:

Adding OpenTelemetry Metrics

Beyond just tracing, keeping an eye on metrics is important for monitoring how your app is performing and staying healthy. Metrics give you quantitative data like the number of requests, errors, or how long operations take. This helps you spot trends and potential issues before they start affecting your users.

To begin, you need to add metrics instrumentation to your OpenTelemetry setup, which is again accomplished by using the AddAWSInstrumentation method. We’ll continue using the Console exporter for simplicity, but the approach can be adapted to use other exporters depending on your environment and monitoring tools.

Following is how to update your Program.cs to include both tracing and metrics:

using Amazon.S3;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(e => e.AddService("DemoOtel"))
    .AddAWSInstrumentation()
    .AddConsoleExporter()
    .Build();

Sdk.CreateMeterProviderBuilder()
    .ConfigureResource(e => e.AddService("DemoOtel"))
    .AddAWSInstrumentation()
    .AddConsoleExporter()
    .Build();

var s3Client = new AmazonS3Client();

try
{
    var listBucketsResponse = await s3Client.ListBucketsAsync();
    var deleteBucketResponse = await s3Client.DeleteBucketAsync("amzn-s3-demo-bucket");
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

Console.Read();

After updating the code, run the application again using dotnet run. This time, in addition to the traces we observed earlier, you’ll also see various metrics output to the console. Following is an example of the errors metric:

Metric Name: client.call.errors, Unit: {error}, Meter: AWSSDK.S3
...
Value: 1

This output tells you that one error occurred during the execution of an AWS SDK operation. Metrics like this are essential for monitoring the health of your application, providing insights into how often errors occur, how long operations take, and more.

Supported Metrics in the AWS SDK for .NET

To give you a better idea of what you can monitor, following is a list of the metrics supported by the AWS SDK for .NET:

client.call.duration: How long specific AWS SDK operations take, which can help in identifying performance bottlenecks.
client.uptime: The duration for which a client instance has been running, useful for detecting inefficient client management or reuse patterns.
client.call.attempts: The number of attempts made for each operation, providing insights into retry behavior and service reliability.
client.call.errors: The count of errors encountered during operations, helping to identify frequently failing operations.
client.call.attempt_duration: The time taken for each individual attempt of an operation, which helps in analyzing performance for each request.
client.call.resolve_endpoint_duration: The time taken to resolve the service endpoint, which is essential for diagnosing performance and identifying operational issues.
client.call.serialization_duration: The time required to serialize request data before sending it to AWS, useful for optimizing data processing and transmission.
client.call.deserialization_duration: The time taken to deserialize response data received from AWS, helping to improve the speed of processing incoming data.
client.call.auth.signing_duration: The time required to sign requests for authentication, which is important for understanding the impact of signing processes on request latency.
client.call.auth.resolve_identity_duration: The duration taken to acquire authentication credentials, useful for ensuring that identity resolution does not delay operations.
client.http.bytes_sent: The total amount of data sent by the client in HTTP requests, which helps monitor and optimize outbound data flow.
client.http.bytes_received: The total amount of data received by the client in HTTP responses, useful for tracking and managing inbound data volume.

These metrics are essential for understanding the performance and reliability of your AWS SDK operations, helping you identify trends, performance bottlenecks, and areas where errors are occurring.

Following is an example of how these metrics look when visualizing them in Amazon Managed Grafana after running the application for dozens of iterations:

Creating Your Own Telemetry Providers

While OpenTelemetry provides a powerful and standardized approach to observability, we also offer flexibility for those who need more control. If you have specific telemetry requirements, the AWS SDK for .NET now includes the TelemetryProvider, which is a central point for registering custom implementations of tracing and metrics providers. This means that you can use your own custom telemetry solutions.

If you have a specific telemetry solution in mind, or if you need fine-grained control over how telemetry data is captured and processed, you can register your own implementation with the TelemetryProvider.
Following is a simple example of how you can register your own TracerProvider and MeterProvider:

using Amazon;
using Amazon.Runtime.Telemetry;
using Amazon.Runtime.Telemetry.Metrics;
using Amazon.Runtime.Telemetry.Tracing;

public class CustomTracerProvider : TracerProvider
{
    // Implement custom tracing logic here
}
public class CustomMeterProvider : MeterProvider
{
    // Implement custom metrics logic here
}

// Register custom implementations
AWSConfigs.TelemetryProvider.RegisterTracerProvider(new CustomTracerProvider());
AWSConfigs.TelemetryProvider.RegisterMeterProvider(new CustomMeterProvider());

By registering your own TracerProvider and MeterProvider implementations, you can have complete control over how telemetry data is captured, processed, and exported from your application.

Conclusion

In this post, we’ve explored how you can enhance observability in your .NET applications by integrating OpenTelemetry with the AWS SDK for .NET. We walked through the steps to add OpenTelemetry instrumentation to your application, highlighted the new metrics support, and discussed how you can create custom telemetry providers, if needed.

By adding OpenTelemetry instrumentation to your .NET application, you can start capturing detailed traces and metrics out of the box without having to worry about the underlying implementation. This provides you with valuable insights into your application’s behavior, from tracking API call durations to monitoring system metrics.

We encourage you to download the latest version of the AWS SDK for .NET and OpenTelemetry.Instrumentation.AWS, try out the new observability features, and explore the rich ecosystem of OpenTelemetry exporters to find the one that best fits your monitoring and observability needs. If you have any feedback or issues, you can reach out to us on the AWS SDK for .NET repository on GitHub.

About the author:

AWS Developer Tools Blog