Instrumenting Web Apps Using AWS X-Ray

This post was written by James Bowman, Software Development Engineer, AWS X-Ray

AWS X-Ray helps developers analyze and debug distributed applications and underlying services in production. You can identify and analyze root-causes of performance issues and errors, understand customer impact, and extract statistical aggregations (such as histograms) for optimization.

In this blog post, I will provide a step-by-step walkthrough for enabling X-Ray tracing in the Go programming language. You can use these steps to add X-Ray tracing to any distributed application.

Revel: A web framework for the Go language

This section will assist you with designing a guestbook application. Skip to “Instrumenting with AWS X-Ray” section below if you already have a Go language application.

Revel is a web framework for the Go language. It facilitates the rapid development of web applications by providing a predefined framework for controllers, views, routes, filters, and more.

To get started with Revel, run revel new github.com/jamesdbowman/guestbook. A project base is then copied to $GOPATH/src/github.com/jamesdbowman/guestbook.

$ tree -L 2 . ├── README.md ├── app │ ├── controllers │ ├── init.go │ ├── routes │ ├── tmp │ └── views ├── conf │ ├── app.conf │ └── routes ├── messages │ └── sample.en ├── public │ ├── css │ ├── fonts │ ├── img │ └── js └── tests └── apptest.go

Writing a guestbook application

A basic guestbook application can consist of just two routes: one to sign the guestbook and another to list all entries.
Let’s set up these routes by adding a Book controller, which can be routed to by modifying ./conf/routes.

./app/controllers/book.go:
    package controllers
     
    import (
    	"math/rand"
    	"time"
     
    	"github.com/aws/aws-sdk-go/aws"
    	"github.com/aws/aws-sdk-go/aws/endpoints"
    	"github.com/aws/aws-sdk-go/aws/session"
    	"github.com/aws/aws-sdk-go/service/dynamodb"
    	"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
    	"github.com/bowmessage/test/xray"
    	"github.com/revel/revel"
    )
     
    const tableName = "guestbook"
    const success = "Success.\n"
     
    var letters = []rune("ABCDEFGHIJKLMNOPQRSTUVWXYZ")
     
    func init() {
    	rand.Seed(time.Now().UnixNano())
    }
     
    // randString returns a random string of len n, used for DynamoDB Hash key.
    func randString(n int) string {
    	b := make([]rune, n)
    	for i := range b {
    		b[i] = letters[rand.Intn(len(letters))]
    	}
    	return string(b)
    }
     
    // Book controls interactions with the guestbook.
    type Book struct {
    	*revel.Controller
    	ddbClient *dynamodb.DynamoDB
    }
     
    // Signature represents a user's signature.
    type Signature struct {
    	Message string
    	Epoch   int64
    	ID      string
    }
     
    // ddb returns the controller's DynamoDB client, instatiating a new client if necessary.
    func (c Book) ddb() *dynamodb.DynamoDB {
    	if c.ddbClient == nil {
    		sess := session.Must(session.NewSession(&aws.Config{
    			Region:     aws.String(endpoints.UsWest2RegionID),
    			MaxRetries: aws.Int(3),
    		}))
    		c.ddbClient = dynamodb.New(sess)
    		xray.AWS(c.ddbClient.Client) // add subsegment-generating X-Ray handlers to this client
    	}
    	return c.ddbClient
    }
     
    // Sign allows users to sign the book.
    // The message is to be passed as application/json typed content, listed under the "message" top level key.
    func (c Book) Sign() revel.Result {
    	var s Signature
     
    	err := c.Params.BindJSON(&s)
    	if err != nil {
    		return c.RenderError(err)
    	}
    	now := time.Now()
    	s.Epoch = now.Unix()
    	s.ID = randString(20)
     
    	item, err := dynamodbattribute.MarshalMap(s)
    	if err != nil {
    		return c.RenderError(err)
    	}
     
    	putItemInput := &dynamodb.PutItemInput{
    		TableName: aws.String(tableName),
    		Item:      item,
    	}
     
    	goRequest := c.Request.In.(*revel.GoRequest)
    	_, err = c.ddb().PutItemWithContext(goRequest.Original.Context(), putItemInput)
    	if err != nil {
    		return c.RenderError(err)
    	}
     
    	return c.RenderText(success)
    }
     
    // List allows users to list all signatures in the book.
    func (c Book) List() revel.Result {
    	scanInput := &dynamodb.ScanInput{
    		TableName: aws.String(tableName),
    		Limit:     aws.Int64(100),
    	}
     
    	goRequest := c.Request.In.(*revel.GoRequest)
    	res, err := c.ddb().ScanWithContext(goRequest.Original.Context(), scanInput)
    	if err != nil {
    		return c.RenderError(err)
    	}
     
    	messages := make([]string, 0)
    	for _, v := range res.Items {
    		messages = append(messages, *(v["Message"].S))
    	}
    	return c.RenderJSON(messages)
    }

./conf/routes: POST /sign Book.Sign GET /list Book.List

Creating the resources and testing

For the purposes of this blog post, the application will be run and tested locally. We will store and retrieve messages from an Amazon DynamoDB table. Use the following AWS CLI command to create the guestbook table:

aws dynamodb create-table --region us-west-2 --table-name "guestbook" --attribute-definitions AttributeName=ID,AttributeType=S AttributeName=Epoch,AttributeType=N --key-schema AttributeName=ID,KeyType=HASH AttributeName=Epoch,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Now, let’s test our sign and list routes. If everything is working correctly, the following result appears:

$ curl -d '{"message":"Hello from cURL!"}' -H "Content-Type: application/json" http://localhost:9000/book/sign
Success.
$ curl http://localhost:9000/book/list
[
  "Hello from cURL!"
]%

Integrating with AWS X-Ray

Download and run the AWS X-Ray daemon

The AWS SDKs emit trace segments over UDP on port 2000. (This port can be configured.) In order for the trace segments to make it to the X-Ray service, the daemon must listen on this port and batch the segments in calls to the PutTraceSegments API.
For information about downloading and running the X-Ray daemon, see the AWS X-Ray Developer Guide.

Installing the AWS X-Ray SDK for Go

To download the SDK from GitHub, run go get -u github.com/aws/aws-xray-sdk-go/... The SDK will appear in the $GOPATH.

Enabling the incoming request filter

The first step to instrumenting an application with AWS X-Ray is to enable the generation of trace segments on incoming requests. The SDK conveniently provides an implementation of http.Handler which does exactly that. To ensure incoming web requests travel through this handler, we can modify app/init.go, adding a custom function to be run on application start.

import (
        "github.com/aws/aws-xray-sdk-go/xray"
        "github.com/revel/revel"
    )
     
    ...
     
    func init() {
      ...
        revel.OnAppStart(installXRayHandler)
    }
     
    func installXRayHandler() {
    	server := revel.CurrentEngine.Engine().(*http.Server)
    	server.Handler = xray.Handler(xray.NewFixedSegmentNamer("GuestbookApp"), server.Handler)
}

The application will now emit a segment for each incoming web request. The service graph appears:

You can customize the name of the segment to make it more descriptive by providing an alternate implementation of SegmentNamer to xray.Handler. For example, you can use xray.NewDynamicSegmentNamer(fallback, pattern) in place of the fixed namer. This namer will use the host name from the incoming web request (if it matches pattern) as the segment name. This is often useful when you are trying to separate different instances of the same application.

In addition, HTTP-centric information such as method and URL is collected in the segment’s http subsection:

"http": {
    "request": {
        "url": "/book/list",
        "method": "GET",
        "user_agent": "curl/7.54.0",
        "client_ip": "::1"
    },
    "response": {
        "status": 200
    }
},

Instrumenting outbound calls

To provide detailed performance metrics for distributed applications, the AWS X-Ray SDK needs to measure the time it takes to make outbound requests. Trace context is passed to downstream services using the X-Amzn-Trace-Id header. To draw a detailed and accurate representation of a distributed application, outbound call instrumentation is required.

AWS SDK calls

The AWS X-Ray SDK for Go provides a one-line AWS client wrapper that enables the collection of detailed per-call metrics for any AWS client. We can modify the DynamoDB client instantiation to include this line:

// ddb returns the controller's DynamoDB client, instatiating a new client if necessary.
func (c Book) ddb() *dynamodb.DynamoDB {
    if c.ddbClient == nil {
        sess := session.Must(session.NewSession(&aws.Config{
            Region: aws.String(endpoints.UsWest2RegionID),
        }))
        c.ddbClient = dynamodb.New(sess)
        xray.AWS(c.ddbClient.Client) // add subsegment-generating X-Ray handlers to this client
    }
    return c.ddbClient
}

We also need to ensure that the segment generated by our xray.Handler is passed to these AWS calls so that the X-Ray SDK knows to which segment these generated subsegments belong. In Go, the context.Context object is passed throughout the call path to achieve this goal. (In most other languages, some variant of ThreadLocal is used.) AWS clients provide a *WithContext method variant for each AWS operation, which we need to switch to:

_, err = c.ddb().PutItemWithContext(c.Request.Context(), putItemInput)
    res, err := c.ddb().ScanWithContext(c.Request.Context(), scanInput)

We now see much more detail in the Timeline view of the trace for the sign and list operations:

We can use this detail to help diagnose throttling on our DynamoDB table. In the following screenshot, the purple in the DynamoDB service graph node indicates that our table is underprovisioned. The red in the GuestbookApp node indicates that the application is throwing faults due to this throttling.

HTTP calls

Although the guestbook application does not make any non-AWS outbound HTTP calls in its current state, there is a similar one-liner to wrap HTTP clients that make outbound requests. xray.Client(c *http.Client) wraps an existing http.Client (or nil if you want to use a default HTTP client). For example:

resp, err := ctxhttp.Get(ctx, xray.Client(nil), "https://aws.amazon.com/")

Instrumenting local operations

X-Ray can also assist in measuring the performance of local compute operations. To see this in action, let’s create a custom subsegment inside the randString method:


// randString returns a random string of len n, used for DynamoDB Hash key.
func randString(ctx context.Context, n int) string {
    xray.Capture(ctx, "randString", func(innerCtx context.Context) {
        b := make([]rune, n)
        for i := range b {
            b[i] = letters[rand.Intn(len(letters))]
        }
        s := string(b)
    })
    return s
}

// we'll also need to change the callsite

s.ID = randString(c.Request.Context(), 20)

Summary

By now, you are an expert on how to instrument X-Ray for your Go applications. Instrumenting X-Ray with your applications is an easy way to analyze and debug performance issues and understand customer impact. Please feel free to give any feedback or comments below.

For more information about advanced configuration of the AWS X-Ray SDK for Go, see the AWS X-Ray SDK for Go in the AWS X-Ray Developer Guide and the aws/aws-xray-sdk-go GitHub repository.

For more information about some of the advanced X-Ray features such as histograms, annotations, and filter expressions, see the Analyzing Performance for Amazon Rekognition Apps Written on AWS Lambda Using AWS X-Ray blog post.

AWS DevOps & Developer Productivity Blog