Analyzing Performance for Amazon Rekognition Apps Written on AWS Lambda Using AWS X-Ray

AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. With X-Ray, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. It allows you to view, filter, and gain insights to identify issues and opportunities for optimization.

X-Ray helps you to both understand the performance of your app and to annotate important parameters. In this post, I look at a sample Amazon Rekognition app that stores images in Amazon S3. The app calls the RecognizeCelebrities action from an AWS Lambda function. The function uses X-Ray features to analyze the app’s performance in recognizing any celebrity in a single image or in multiple images. These features include:

• Annotations and traces
• Histogram
• Metadata
• Exceptions

Sample app overview

In the sample app, you can upload images and store them in the imagestoragexray bucket. Then you can choose Recognize faces for an individual image. This calls a Lambda function, which then calls RecognizeCelebrities to recognize the celebrities in the image. The Lambda function returns to the client a comma-separated list of celebrities that it found in the image.

Overview of the sample app’s architecture

X-Ray service map from the sample app

To search for a celebrity in the collection of images, you can type the celebrity’s name in the text box and choose Search celeb. This calls another Lambda function that runs RecognizeCelebrities on all the images in the imagestoragexray bucket. The Lambda function then returns the number of times that the celebrity was found in all the images.

Example of a sample app webpage with stored images

Annotations and traces

Annotations are indexed for grouping traces in the X-Ray console, based on parameters stored in the annotations. Annotations help you determine the performance of particular images.

Using annotations to record the celebrity image file name, you can filter to see traces for performance of your app on specific images. For example, all the celebrities recognized in the analyzed images are annotated with the average confidence with which Amazon Rekognition matched the celebrity. The number of faces found in an image is also annotated.

Resourceful annotations such as these provide you with a comprehensive outlook about your app’s performance.

X-Ray annotation in the sample app

Understanding Lambda function cold starts

When a Lambda function is invoked for the first time, or after it’s updated, Lambda launches a container based on the configuration settings that you provided. It takes time to set up a container and do the necessary bootstrapping, also known as a cold start.

The latency of a Lambda function is the amount of time between when a request starts and when it completes. Due to the cold start behavior, Lambda function executions may take longer on the first invocation or after a function has been updated. For subsequent invocations, Lambda tries to reuse the container to reduce the latency.

X-Ray helps you trace the cold start time as well as the overall latency for a Lambda function. In the following screenshot showing the 3.5-second duration of the Lambda function, you see that it took 473 milliseconds to initialize the function. For more information about execution time, see AWS Lambda: How It Works.

X-Ray trace showing initialization time for the Lambda function

X-Ray trace on subsequent invocations of the Lambda function without the cold start

Determining the time for image analysis on celebrities

You can determine that the time Amazon Rekognition takes differs with the number of celebrity images in the photo. For example, a photo without a celebrity takes a shorter time to process, compared to a photo with a celebrity that Amazon Rekognition was able to recognize. In the following screenshots, you can see that the Lambda function that calls Amazon Rekognition to find images with Steve Jobs took 1.7 seconds to complete. This is compared to 858 milliseconds, using the same Lambda function for an image with a random landscape

X-Ray trace showing the time that a Lambda function takes to call Amazon Rekognition and return

By choosing the particular traces that show up in the trace list for your annotation filter, you can investigate further. Specifically, you can look at the performance of your Amazon Rekognition call.

Amazon Rekognition took 1.3 seconds to analyze and provide results for the image with Steve Jobs. However, it took only 829 milliseconds to analyze and provide results for the random landscape image.

X-Ray trace showing the time that Amazon Rekognition takes to analyze the Steve Jobs image

X-Ray trace showing the time that Amazon Rekognition takes to analyze the random landscape image

Determining the time for image analysis on multiple faces

X-Ray can also help you to look at the performance of your app for images with multiple faces. Using the annotations feature, you can record the number of faces that Amazon Rekognition recognized in the image. Through this annotation, you can filter traces for images with a specific number of face counts that were recognized or not recognized in Amazon Rekognition, as shown in the following screenshot.

X-Ray annotations for face counts in images recognized by Amazon Rekognition

In the images below, notice that Amazon Rekognition takes more time to analyze images that contain more faces. For example, Amazon Rekognition took 8.9 seconds to analyze an image with 15 faces compared to only 6.0 seconds for an image with 5 faces. You can use the filter expression Annotation.Facecount > “5” to view requests for which Amazon Rekognition recognized more than 5 faces.

X-Ray traces for an image with 15 face counts

X-Ray traces for an image with 5 face counts

Histograms

When you select a node or edge on an X-Ray service map, the console shows a latency distribution histogram. It shows duration on the x-axis, and the percentage of requests that match each duration on the y-axis. Using this histogram, you can look at calls that have a high latency and try to improve their performance.

Using the latency histogram

In your service map, you can choose individual nodes to look at the response latency for calls. When you choose the Amazon Rekognition node AWS::rekognition in the service map, you can see the corresponding response latency from Amazon Rekognition, as shown in the following screenshot.

X-Ray response distribution histogram for response latency

Notice that the response latency increases with the number of face counts in the image. As you noted previously, image analysis time also increases with the face count increase.

For example, the random landscape image’s response latency is less than 1 second, whereas calls to Amazon Rekognition for images with one face count have a response latency between 1 second and 1.5 seconds. Images with multiple face counts have a much higher response latency, greater than 1.5 seconds.

X-Ray traces for images with a response latency between 1 second and 1.5 seconds

X-Ray response distribution histogram for a higher response latency

X-Ray traces for images with multiple face counts that have a higher response latency

Metadata

The metadata feature in X-Ray enables you to store app information for later reference. However, you cannot filter out traces based on the metadata. The sample app stores the response from Amazon Rekognition in metadata for your reference.

X-Ray metadata in the sample app

Exceptions

X-Ray also provides information on exceptions from other AWS services in the trace. For the sample app, Amazon Rekognition responded back with an exception when the RecognizeCelebrities call was made on a non-image file. You can also filter requests that had an exception in Amazon Rekognition, using the following expression:
service("rekognition") { error = true }

For more information, see Searching for Traces in the AWS X-Ray Console with Filter Expressions.

X-Ray trace showing an error from Amazon Rekognition

X-Ray trace showing the Amazon Rekognition exception in more detail

Summary

X-Ray helps you analyze your app and its performance. Through the annotations feature, you can index and filter specific traces. Annotating specific parameters in your app helps you get a comprehensive overview of your app’s performance specific to these parameters. The histogram helps you analyze calls with high response latency and gives you visibility into the time that individual services take. The metadata feature helps you store any relevant information that might be useful to improve your app’s performance. X-Ray also provides information on exceptions that happened on a particular trace to an AWS service.

Overall, by using X-Ray you can both improve performance of your app and understand the underlying services that your app uses. Use this information to troubleshoot and improve specific portions of your app, and save both time and money.

If you have questions or suggestions, please comment below.