Send real-time alerts about application anomalies using AWS X-Ray insights

Today AWS X-Ray launches support for notifications to its insights. This means that on an X-Ray group where insights are enabled, you can now configure notifications to be sent to Amazon EventBridge.

Through the use of anomaly detection, AWS X-Ray helps you analyze and debug distributed applications. AWS X-Ray Insights uses anomaly detection to create actionable insights on anomalies identified through traces from your applications.

With this launch, you can enable real time alerts through notifications on any anomalies in your application.

Here are some potential use cases for this feature:

• Alert when fault rate for a specific service is over X%.
• Alert when anomalies are detected for a specified group.
• Alert when the root cause of the issue is a specified service.
• Alert when an insight state is active or closed.

How it works

After you can enable this feature on an X-Ray group that also has insights enabled, X-Ray will send notifications to Amazon EventBridge. You can set up a conditional rule that can eventually take actions such Amazon SNS notification, AWS Lambda invocation, posting a message to Amazon SQS queue or any of the variety of targets that EventBridge supports.

Figure 1: Flow of notification sent to EventBridge

Getting started

1. Open the AWS X-Ray console, choose Groups

2. Under Edit group, select the Enable Notifications check box as shown in Figure 2.

Figure 2: Edit group section of the AWS X-Ray console with Enable Notifications selected

3. Open the Amazon EventBridge console, choose Events, and then choose Rules.

4. On the Create rule page, under Event matching pattern, choose Custom pattern.

5. Add an event pattern as shown below to match all incoming events from insights and configure target(s) for the rule.

Figure 3: Create rule page in the Amazon EventBridge console

You’ll see the source field contains aws.xray and the detail-type field contains AWS X-Ray Insight Update. You can use content based filter matching with event patterns to write complex rules that are triggered only under conditions you specify and also transform input while passing the event to configured targets.

User scenarios

After you have enabled notifications for an X-Ray group that has insights enabled, here are some of the anomaly detection scenarios that might be of interest:

1. Send an email when an anomaly is detected for the X-Ray group “SampleGroup”.

a. Create an EventBridge rule with following event pattern, specifying GroupName field:

{
"source": [ "aws.xray" ],
"detail-type": [ "AWS X-Ray Insight Update" ],
"detail": {
        "GroupName": [ " SampleGroup " ]
  }
}

b. Assuming the email lists are subscribed to an SNS topic, add the topic as a target for the rule. You can either pass the matched event as it is or customize the input using an input transformer before passing it to a target as shown in Figure 4:

Figure 4: Select targets page

c. Based on the target configuration, an email will be received when an anomaly is detected for the group.

2. Invoke a Lambda function when an anomaly is detected for a root cause service, “name.service.com”, which belongs to any insights-enabled group.

a. Create an EventBridge rule with following event pattern, specifying RootCauseServiceId and Name field:

{
"source": [ "aws.xray" ],
"detail-type": [ "AWS X-Ray Insight Update" ],
"detail": {
        "RootCauseServiceId": {
             “Name”: [“name.service.com”] 
         }
   }
}

b. Add the Lambda function as a target for the rule. The function can have custom business logic to process incoming events and invoke various workflows.

Conclusion

Notifications functionality is available globally in all AWS Regions. For more information, see Using insights in the AWS X-Ray Developer Guide.

If you have any feedback, please send it to xray-insights-feedback@amazon.com.

Authors

Yogiraj Awati is a Software Development Engineer in AWS focused on monitoring distributed applications built using micro-services architecture. He is passionate about solving hard problems and currently is a member of the distributed tracing service with AWS X-Ray.

Imaya is a Senior Solution Architect focused on Amazon CloudWatch and AWS X-Ray. He is passionate about Monitoring and Observability and has a strong application development and architecture background. He likes working on distributed systems and is excited to talk about micro-service architecture design. He loves programming on C#, working with Containers and Serverless technologies.