API observability, which consists of logging, monitoring, and tracing, is a form of direct observation of a system and requires an agent or SDK to passively log API traffic. This data collection can be done within the application or at different points, such as with an API gateway. API observability can provide your business and engineering teams with deep insights into how your APIs are used. API observability can help:

• Product teams to understand API usage and business value
• Engineering teams to monitor and troubleshoot API issues
• Security teams to detect and protect from API threats

Moesif API Analytics is an API observability solution that you can use to better understand API usage. There is native integration with Amazon API Gateway, which makes deployment a matter of a few steps and does not require any code change or restarts. As the gateway to the rest of your infrastructure, API gateways are also the natural place to provide API observability to your various business and engineering teams.

In this post, I show you how to gain observability into your APIs by sending API access logs from your Amazon API Gateway to Moesif for analysis using an Amazon Kinesis Data Firehose as a buffer. I will also show how to use API observability to troubleshoot performance issues by creating a report on latency and how to better understand API usage by creating a funnel report.

Solution overview

The following solution adds API analytics to your APIs hosted behind Amazon API Gateway. It works by forwarding structured API access logs from your Amazon API Gateway instance to Moesif via an Amazon Kinesis Data Firehose. Deployment of the solution can be done in a few steps using the included AWS CloudFormation template and doesn’t require any downtime.

The following solution architecture diagram shows the customer’s infrastructure where the Amazon API Gateway, Amazon Kinesis Data Firehouse, and other compute resources reside. The API Gateway receives web traffic from the client—such as a computing or mobile device. The diagram also shows how the Moesif infrastructure processes.

Solution walkthrough

This demo works by sending API access logs from your Amazon API Gateway to Moesif for analysis using an Amazon Kinesis Data Firehose as a buffer. There are two types of logs from API Gateway logs: API Access Logs and CloudWatch Execution Logs. While execution logs are typical lines of free form text designed to be human readable, API access logs have a strict JSON structure and schema. This makes them more machine-parsable and suited for analytics. In addition, the logs contain the user identity, which makes them perfect for user behavior analytics tools like Moesif.

Prerequisites

• Set your AWS API Gateway protocol to REST. Other protocols are not supported by this solution.
• Have at least one API in the API Gateway instance to generate traffic. If you want to use an existing API, use the existing API example to quickly build a REST API.
• You must have permissions to deploy IAM resources.

A.  Obtain a Moesif Application Id

Sign up for a no-cost Moesif account in AWS Marketplace. During the onboarding steps, select AWS from the list of Plugins. You will receive your Moesif Application Id, which you need for the next step.

B. Launch CloudFormation stack

Use the CloudFormation template from Moesif to automatically create a Kinesis Data Firehose and configure it to send API Gateway access logs to Moesif. To get started, from within the Moesif onboarding dashboard, on the Quick Install page, choose the following launch stack button.

This action opens the Quick create stack page within the AWS Management Console. You’ll know you’re in the right place if the breadcrumbs on the top left show up as CloudFormation > Stacks > Create Stack. Enter your Moesif Application Id from step A. This field parameter may be auto-populated with the Application Id from your Moesif account.

Ensure you check the box next to the item I acknowledge that AWS CloudFormation might create IAM resources. Then choose Create stack.

C.  Enable API Gateway access logging

Enable API Access Logs in Amazon API Gateway and send it to the Amazon Kinesis Data Firehose you created automatically in step B by completing the following:

1. In the AWS Management Console, navigate to your AWS API Gateway instance.
2. From the left menu, select Stages and then select the Logs/Tracing tab.
3. Toggle on Enable Access Logging.
4. Add your Data Firehose ARN for Access Log Destination ARN created by the CloudFormation stack from step B.

• • To find this ARN, go to Amazon Kinesis within the AWS Management Console and select the Firehose instance.
  • By default, it is called amazon-apigateway-api-logs-to-moesif. Copy the data in the field under Delivery stream ARN.

5. Go back to the AWS API Gateway console Stages page and Logs/Tracing tab. Enter the ARN.

D.  Add the JSON log format

Still within your AWS API Gateway instance in the AWS Management Console, add the following JSON log format so the access log output is compatible with Moesif. Moesif will safely ignore any extra keys. To do this, copy the following JSON log format and paste it in the Log Format section of Custom Access Logging. Once you paste in this JSON format, choose Save Changes.

{

"apiId": "$context.apiId",

"requestId": "$context.requestId",

"requestTime": "$context.requestTime",

"protocol": "$context.protocol",

"httpMethod": "$context.httpMethod",

"resourcePath": "$context.resourcePath",

"requestHostHeader": "$context.domainName",

"requestUserAgentHeader": "$context.identity.userAgent",

"ip": "$context.identity.sourceIp",

"status": "$context.status",

"responseLength":"$context.responseLength",

"durationMs": "$context.responseLatency",

"caller": "$context.identity.caller",

"user": "$context.identity.user",

"principalId": "$context.authorizer.principalId",

"cognitoIdentityId": "$context.identity.cognitoIdentityId",

"userArn": "$context.identity.userArn",

"apiKey": "$context.identity.apiKey"

}

E.   Confirm API Logging

Your API Gateway integration is now complete. Complete the fourth and last step of onboarding within Moesif in the onboarding wizard.

To verify that API logs show up in Moesif, do the following:

1. Go to a REST client such as Postman or make an API call from within your Amazon API Gateway instance.
2. To see that you’ve been successful with the API calls, go to the Moesif portal top navigation and choose Events.
3. On the Events page, you should see your status code, URL, and other HTTP parameters captured.

How to troubleshoot performance issues by creating a report on latency

With high-cardinality, high-dimension API observability, you can slice and dice your API logs by any number of fields, including HTTP headers or response time. This approach makes it easy to troubleshoot issues without manually searching through logs. A core engineering metric for APIs is latency percentiles, such as the 90th percentile. A best practice is to look at 90th percentile latency over the average. This practice helps uncover large variations in your latency that can be masked by low averages. Your API users are looking for consistently low latency, not the lowest average, as spikes can wreak havoc in their own services.

Now that you have some data in Moesif, you can create reports such as latency by customer or usage by API. I want to plot the 90th percentile latency broken down by API.

To create a 90^th percentile latency report, do the following:

1. In the Moesif portal, under Events, choose the submenu item Time Series.
2. Within the chart area, under the label Group By, add Request >> URI Route.
3. For Metrics, select P90 Latency.

You should see a report showing 90^th percentile latency broken down by endpoint.

Moesif automatically tracks all the API routes and verbs within your Amazon API Gateway instance. Endpoints like /items/1 and /items/2 are automatically consolidated to a single route /items/:id.

How to understand API usage by creating a funnel report

A key benefit of API analytics is understanding how customers adopt and use your APIs, such as which customers are using your APIs the most. You can use Moesif API analytics to better understand API usage across multiple web and API products. This approach makes it easier to understand your customer journey.

Funnels are a type of report that show the percentage of your users who get to the next step. A funnel report also provides metrics like Time to First Hello World or Time to Value.

To create a funnel report in Moesif, follow these instructions to add moesif-browser-js and calling the track method in your web app as shown here:

moesif.track(Signed-In', {

button_label: 'Get Started',

sign_up_method: 'Google SSO'

});

1. In the Moesif portal, under Users, choose the submenu item User Funnel.
2. Under Step 1, open the drop down and select Action Name. For the operator, select is. For the value, select Signed-In.
3. Under Step 2, open the drop down and select Event Type. For the operator, select is. For the value, select API Call.
4. Select + Funnel Step to add a third step. Open the drop down and select Event Type For the operator, select is. For the value, select API Call similar to the last step.
   100. Within Step 3, increase the value for At least X Times to 100.

The resulting funnel analysis visualizes the flow from a customer signing up to being fully integrated with the API.

1. The criteria for the first step is a customer sign up. Within the UI, this user action is tracked.
2. The criteria for the second step is a customer’s single payment transaction via the API.
3. The criteria for the third step is a customer’s 100+ transactions which is enough volume for a customer to see value.

The following chart shows my funnel analysis.

1. The top pane shows the three criteria definitions, each defined by Where followed by the criteria.
2. The second pane shows Performed by Users and Grouped By.
3. The third pane is a bar chart showing percentages on the Y axis.
4. The first bar shows all interactions meeting the criteria, totaling 100%.
5. The second is conversion rate for criteria 2, which is 49%. This means that 49% of customers who sign up end up making their first API call, known as Hello World.
6. The third column shows a conversion rate for criteria 3, which is 35%. That means 35% of customers who made at least one API call end up making over 100 payment transactions, presumably seeing value.

Conclusion

In this post, I showed you how to gain observability into your APIs by sending API access logs from your Amazon API Gateway to Moesif for analysis using an Amazon Kinesis Data Firehose as a buffer. I looked at two common use cases including understanding your customers’ API usage and troubleshooting API issues. I also showed you how to create a report on API performance and how to use API analytics to gain insights into user behavior.

Having the right API observability solution can provide your team with the right visibility to make informed decisions. While you can deploy your own API gateway, data processing pipeline, and a data warehouse, these can create a time sink for your engineering team. Using fully managed services such as Amazon API Gateway and Moesif API Analytics can help you scale without the heavy lifting maintenance costs or outdated data infrastructure. Moesif is listed in AWS Marketplace, and you can sign up and pay as you go, including starting with a free tier.

The content and opinions in this post are those of the third-party author, and AWS is not responsible for the content or accuracy of this post.

About the author

Derric Gilling, CEO Moesif

Derric is the Co-Founder and CEO of Moesif, an API Analytics platform, based in San Francisco, CA. Previously, he was Co-Founder and CTO of Trove. After graduating from the University of Michigan, he built award-winning functional and formal verification software for Intel, and later a computer architect on Intel’s Xeon Phi, a manycore CPU for HPC and ML workloads. He focuses on API strategy, platform growth, analytics, machine learning, and computer architecture.