How do I troubleshoot issues when connecting to an API Gateway private API endpoint?

Last updated: 2022-09-22

I'm having issues connecting to my Amazon API Gateway private API endpoint that's in Amazon Virtual Private Cloud (Amazon VPC). How do I troubleshoot the issue?

Short description

AWS resources in Amazon VPC can fail to connect to a private API endpoint for any of the following reasons:

When Amazon CloudWatch logging is activated for your API, an error message that indicates the cause of the error appears in your execution logs.

If the API request doesn't produce any CloudWatch logs after logging is activated, then the request didn't reach the endpoint. If your API requests aren't reaching the endpoint, make sure that the private API's invoke URL is formatted correctly.

Note: Connection issues might be caused by a misconfigured API Gateway resource policy or incorrect DNS names in the private API endpoint’s invoke URL.

Resolution

Confirm the cause of the error

1.    If you haven't done so already, turn on CloudWatch logging for your private REST API. Make sure that you configure execution logging.

Tip: When configuring the logging settings, for Log level, choose INFO. Then, choose Log full requests/responses data.

2.    Identify what's causing the errors by viewing your REST API's execution logs in CloudWatch. If API requests are reaching the endpoint, then an error message similar to one of the following examples appears:

  • "User: anonymous is not authorized to perform: execute-api:Invoke on resource:"
  • "Connection timed out"

If the API request doesn't produce any CloudWatch logs after logging is activated, make sure that your private API's invoke URL is formatted correctly. For instructions, see the If the API request doesn't produce any CloudWatch Logs after logging is activated section of this article.

For more information, see How do I find API Gateway REST API errors in my CloudWatch logs?

Resolve "User: anonymous is not authorized to perform: execute-api:Invoke on resource:" errors

1.    Verify that the private API endpoint's API Gateway resource policy allows traffic from the interface VPC endpoint or source VPC to the API endpoint. For an example resource policy, see Example: Allow private API traffic based on source VPC or VPC endpoint.

2.    Verify that the VPC endpoint policy allows the client access to the private API endpoint. For example VPC endpoint policies, see VPC endpoint policy examples.

For instructions on how to test API Gateway resource policies, see the Test the resource policy section of the following article: How do I allow only specific IP addresses to access my API Gateway REST API?

Important: If you modify your API's resource policy, you must deploy your API to commit the changes.

Resolve "Connection timed out" errors

1.    Verify that an API Gateway execute-api VPC endpoint exists in the VPC. Also verify if the endpoint is in the same AWS Region as the private API. If it doesn't exist, create an interface VPC endpoint for API Gateway execute-api.

2.    Verify that the client invoking the private API endpoint exists in the same VPC or has access to the VPC with the VPC endpoint. You can test this using the traceroute tool.

Note: Private APIs are accessible from clients within the VPC or from clients that have network connectivity to the VPC.

To install traceroute, run the following commands:

Amazon Linux:

$ sudo yum install traceroute

Ubuntu:

$ sudo apt-get install traceroute

To test connectivity using traceroute:

$ sudo traceroute -n -T -p 443 <VPC-endpoint IP-address>

The argument -T -p 443 -n performs a TCP-based trace on port 443.

3.    Verify that the rules for your Amazon VPC security groups are configured correctly.

To test your Amazon VPC security groups:

Run the following command from the client that's making requests to the private API endpoint. Be sure to replace {public-dns-hostname} with the public DNS hostnames containing the VPC endpoint ID for your API. Replace {region} with the AWS Region that your interface VPC endpoint is in.

$ telnet {public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com 443

If the connection times out, then the rules for your Amazon VPC security groups aren't configured correctly.

Verify the following:

  • The requesting resource has a security group rule that allows TCP port 443 outbound traffic to the VPC endpoint's IP address range or security group.
  • The VPC endpoint has a security group rule that allows TCP port 443 inbound traffic from the requesting resource's IP address range or security group.

For more information, see Control traffic to resources using security groups.

If the API request doesn't produce any CloudWatch logs after logging is activated

1.    Verify that the private API endpoint's API Gateway resource policy is configured correctly.

For more information, see the following section of this article: Resolve "User: anonymous is not authorized to perform: execute-api:Invoke on resource:" errors.

2.    Verify that your private API's invoke URL is formatted correctly.

The correct format depends upon having a private DNS activated for the VPC endpoint. If private DNS isn't activated, then you must use endpoint-specific public DNS hostnames to access the private API endpoint. If private DNS is activated, then you must use private DNS names to access the private API endpoint.

For more information, see How to invoke a private API.

Test that the private API endpoint's domain correctly resolves to the VPC endpoint's IP address

1.    Run the following nslookup command from the client that's making requests to the private API endpoint. Be sure to replace {restapi-id} with your private API's ID. Replace {region} with the AWS Region that your private API endpoint is in.

$ nslookup {restapi-id}.execute-api.{region}.amazonaws.com

A successful output shows the VPC endpoint's private IP addresses.

2.    Run the following nslookup command. Be sure to replace {public-dns-hostname} with the public DNS hostnames containing the VPC endpoint ID for your API. Replace {region} with the AWS Region that your interface VPC endpoint is in.

$ nslookup {public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com

A successful output shows the VPC endpoint's private IP addresses.

3.    Compare the IP addresses in the outputs of each command. If the IP addresses from each command output matches, then the setup is working as expected.

Note: You can activate private DNS for your VPC endpoint at any time in the Amazon VPC console by doing the following:
In the Endpoints pane, select your interface VPC endpoint.
Choose Actions.
Choose Modify Private DNS names.
Select the Enable Private DNS Name check box. Then, choose Save Changes.
Choose Modify Private DNS names.