Analyze and debug applications using AWS X-Ray trace data with Grafana

Today, AWS and Grafana Labs are making available a free and open-source AWS X-Ray data source plugin. You can use the latest release of Grafana (version 7.2.0 or later) to visualize AWS X-Ray traces directly in your Grafana dashboards in order to triage performance issues in applications instrumented with X-Ray. This enables you to build a single dashboard in Grafana to visualize and analyze your X-Ray traces alongside your Amazon CloudWatch metrics and logs, for a unified view of your application metrics, logs, and traces.

AWS X-Ray helps developers and DevOps engineers analyze and debug distributed applications, such as those built using a microservices architecture. Grafana is an open source data visualization and analytics software that enables you to query, visualize, alert on, and explore your monitoring data no matter where it is stored.

With this launch, you can use Grafana to connect to your AWS X-Ray account, and use the analytics and advanced visualizations provided in Grafana through charts, graphs, and alerts to monitor your applications that are instrumented using X-Ray. You can analyze X-Ray traces using Grafana Explore view to triage performance issues (such as increases in error and fault rates), determine the root cause of the issue, and identify the end users who are affected. With this launch, you can also create Grafana dashboards and use Grafana’s alerting and notifications feature. You can view the trace timeline to determine the time spent in each service, and view the impact of an error in upstream and downstream services.

“Customers have told us that they would like to combine the rich interactive data visualizations in Grafana with the powerful insights into tracing and the behavior of distributed application that AWS X-Ray empowers,” said Mark Carter, General Manager at AWS. “The new Grafana AWS X-Ray plug-in brings the power of X-Ray into a correlated experience with logs and metrics from dozens of popular data sources. This Grafana X-Ray integration underscores AWS’ continued commitment to open source and our ongoing partnership with Grafana labs.”

You can use the X-Ray plugin in Grafana to monitor your applications using trace data from applications instrumented with X-Ray, alongside other observability data like logs and metrics, to provide a single end-to-end triaging experience. With the AWS X-Ray plugin in Grafana you can:

Use cases

Let’s walk through some scenarios where the X-Ray plugin in Grafana can help you better monitor your applications. We will take an example of a serverless application built using a microservices architecture with a mix of services such as Amazon API Gateway, AWS Lambda, Amazon Simple Notification Service (SNS), and Amazon DynamoDB.

Unified dashboard across all observability data

To monitor this serverless application, you aim to bring all observability data into one unified triaging experience. You can now view X-Ray traces and Amazon CloudWatch logs and metrics in a single debugging workflow with shareable Grafana dashboards. This helps you understand the performance of your application by analyzing all your monitoring data in one place, so you can you detect and resolve issues more quickly.

Determine latency degradation or increase in error or fault rates

For this application, I’d also like to see a historical view of app performance to better understand trends in latency, error, and fault rates so that I can better tune my application. These trends also help determine thresholds or acceptable values so that you can create alerts based on the specific application use case. With Trace Statistics, you can view time series data for your traces, which can then be used in graphs and alerts. You can filter the trace data for a specific service, error code, or HTTP status (among others) to narrow down the query, as well as filter by X-Ray groups. You can also choose Resolution to customize how data will be aggregated and which columns should be returned.

Grafana dashboard panel showing X-Ray data source

Determine the root cause of the issue and the end users impacted

Now that I have detected the issue in my serverless application, I need to understand the cause and determine the impact across all users of my application. Trace Analytics provided by the AWS X-Ray plugin in Grafana lets you narrow down the probable cause, users affected, and other metrics associated with the requests such as HTTP URL, and HTTP status code. You can further determine whether the root cause is due to errors, faults, or increased latency in your application, and see the end-to-end request path.

Grafana Explore screen with X-Ray datasource

Create actionable insights based on anomaly detection

In addition to the existing alerting mechanisms provided by Grafana, you can use the X-Ray Insights feature via Grafana. X-Ray Insights uses anomaly detection to determine abnormal behavior, monitoring the fault rate for all services in the application, and creates an Insight or Incident if the current fault rate breaches the acceptable band. X-Ray Insights also tracks the progression of the incident from the point it started to the point when the incident was closed. Each Insight in Grafana also links to the AWS X-Ray console for further analysis.

Grafana Explore screen with X-Ray datasource and an Insight identified

Getting started

The X-Ray data source is compatible with Grafana version 7.2+ in open source or Enterprise. You can visit the Grafana plugins page to download the X-Ray data source plugin or install it on your Grafana Cloud instance.

The AWS X-Ray data source can be used for any AWS Region in which X-Ray is available. For information on X-Ray’s availability please refer to the AWS Region Table. To learn more, see What’s New in Grafana v7.2 or visit the Grafana documentation.

Let us know what you think

We’re excited to have you try out the AWS X-Ray plugin for Grafana along with the previously launched plugins for Amazon Timestream and Amazon CloudWatch. The plugin is free to use. You only pay for accessing X-Ray trace data. Refer to AWS X-Ray pricing for details. Using this plugin, you can now visualize, analyze, and alert on your AWS X-Ray trace data in the same dashboard as your Amazon CloudWatch metrics, logs, and other observability data, simplifying monitoring and incident triaging in your Grafana console. Please let us know what you think by reaching out to us at xray-grafana-feedback@amazon.com.

Also read: Introducing the AWS X-Ray integration with Grafana

About the Author

Nizar Tyrewalla is a Senior Product Manager in AWS focused on monitoring distributed applications built using microservices architecture. Currently he is leading the distributed tracing service with AWS X-Ray and ingestion of observability data using open source tools and frameworks like OpenTelemetry.