How Wealthfront utilizes AWS X-Ray to analyze and debug distributed applications

This blog post was written by Harichandan Pulagam, a Data Engineer at Wealthfront

In this blog post, we describe how Wealthfront used AWS X-Ray to streamline the development and operations of a distributed application.

About Wealthfront

Wealthfront’s mission is to build a financial system that favors people, not institutions. They strive to provide better experiences and financial outcomes for clients by understanding their financial situations and aspirations. As clients link their Wealthront account to financial accounts at other institutions, Wealthfront helps them gain a holistic view of their finances.

Challenge

The backend workflow that enables customers to link outside accounts involves multiple microservices deployed to the AWS cloud as well as a few services deployed to Wealthfront datacenters. Observability of this distributed application, which we refer to as the “linking flow”, was fragmented. Investigating performance issues required developers to sift through logs in Elasticsearch and metrics in Amazon CloudWatch.

Throughout the remainder of this post, we’ll look at how Wealthfront improved their ability to debug issues, identify performance bottlenecks, and increase developer productivity by adding distributed tracing to the linking flow.

Services used in the solution

Amazon API Gateway
Amazon CloudWatch
Amazon SQS
AWS Lambda
AWS Fargate
AWS X-Ray

Data Flow Diagram

Data Flow Diagram to show the services in a Hybrid environment.

Data Flow Diagram of the linking flow

The data flow starts with the API Gateway receiving webhook requests from a third-party vendor. The webhook requests do not have any logical grouping and contain the link data locators for multiple accounts. These requests are flattened in a Lambda function and each individual request is pushed into an SQS queue. Another Lambda function reads the messages from SQS and makes API calls to the third-party vendor to get the transactional data for each account and add this data to a Kafka topic. The data in this topic is consumed by a service in our datacenter that converts this into a canonical format and writes it to another Kafka topic. The data in this topic is read by the transaction classifier on AWS Fargate. The classified data is output to a Kafka topic, which is read by a service in the datacenter and finally output to a database.

AWS X-Ray for distributed tracing

After a thorough analysis of various tracing solutions, Wealthfront picked AWS X-Ray based on the features it exposes through X-Ray Analytics and X-Ray Insights as well it’s tight integration with Amazon API Gateway and AWS Lambda.

Wealthfront decided to that they wanted the linking flow to emit two distinct traces. The first trace begins at API Gateway and ends at SQS. The second trace begins at the second Lambda and ends at the application in their datacenter that writes to the database.

AWS X-Ray ServiceMap

Adding tracing to Amazon API Gateway stages

Wealthfront enabled X-Ray tracing from Amazon API Gateway within various API stages.

Logs and tracing setting on API Gateway.

Adding Tracing to Lambda functions

Wealthfront bundled the X-Ray SDK into each of the two Lambda functions in the linking flow. The first lambda parses the API Gateway request and inserts messages into SQS. To instrument this function, they used the MessageSystemAttribute. AWSTraceHeader is the only supported message system attribute, and its value must be a correctly formatted AWS X-Ray trace header string.

Subsegment traceHeaderSS = xrayRecorder.beginSubsegment("sqs trace header");
String traceHeaderStr = new TraceHeader(traceHeaderSS.getTraceId()).toString();

SendMessageRequest request = new SendMessageRequest()
   .addMessageSystemAttributesEntry("AWSTraceHeader",
   new MessageSystemAttributeValue().withStringValue(traceHeaderStr))
   .withMessageBody(“test”);

traceHeaderSS.close();
xrayRecorder.sendSubsegment(traceHeaderSS);

The second lambda processes each message from SQS and adds the processed request to Kafka. Wealthfront used Kafka’s ProducerInterceptor interface to intercept the messages before they are published to the Kafka cluster and add tracing information to them. They also set the trace id in RecordHeader of the record that is published to Kafka. These techniques enabled Wealthfront to leave the original content of the messages untouched.

public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
  Subsegment producerInterceptorSS = recorder.beginSubsegment(“kafka_ss”);
  Segment currentSegment = recorder.getCurrentSegment();
  String parentId = currentSegment.getId();
  TraceID traceId = currentSegment.getTraceId();
  String traceInformation = traceId.toString() + "::" + parentId;
  RecordHeader recordHeader = new RecordHeader(“trace_id”,      traceInformation.getBytes());
  ProducerRecord<String, String> recordWithTrace = new ProducerRecord<>(
        record.topic(),
        record.partition(),
        record.key(),
        record.value(),
        ImmutableList.of(recordHeader));
  producerInterceptorSS.close();
  recorder.sendSubsegment(producerInterceptorSS);

  return recordWithTrace;
}

Adding Tracing to services running in the Datacenter

To send traces from the datacenter, Wealthfront used the standalone X-Ray agent. For added resilience, they wrapped it in a systemctl service.

Diagram showing X-Ray agent in a datacenter on-prem

Adding tracing to AWS Fargate container

The transaction classifier runs within an AWS Fargate container. To get traces from the classifier, Wealthfront set up a sidecar container to listen on UDP port 2000 and instrumented the application using the AWS X-Ray SDK to send traces to this sidecar.

Diagram showing the ECS task definition with an application container and the x-ray side-car container

Conclusion

Wealthfront can now visualize requests passing through the linking flow described above. Based on the success of their journey so far, they have decided to add AWS X-Ray tracing to other components with their broader data analytics platform as well as several other distributed applications within their organization.

Disclosure

This communication has been prepared solely for informational purposes only. Nothing in this communication should be construed as an offer, recommendation, or solicitation to buy or sell any security or a financial product. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront or its affiliates endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.

Wealthfront offers a free software-based financial advice engine that delivers automated financial planning tools to help users achieve better outcomes. Investment management and advisory services are provided by Wealthfront Advisers LLC, an SEC registered investment adviser, and brokerage related products are provided by Wealthfront Brokerage LLC, a member of FINRA/SIPC.

Wealthfront, Wealthfront Advisers and Wealthfront Brokerage are wholly owned subsidiaries of Wealthfront Corporation.

AWS Cloud Operations & Migrations Blog