How can I troubleshoot 500 errors with VPC link integration for Amazon API Gateway?

Last updated: 2022-11-01

I configured Amazon API Gateway with VPC Link integrations. When I call my API, I receive a configuration or internal error with an HTTP 500 status code. How do I resolve this issue?

Short description

AWS API gateway can fail to forward traffic to backend resources and return an HTTP 500 status code with VPC links if:

  • The VPC Link is in the Failed state, the endpoint service is deleted, or in the Rejected state.
  • The registered target for the target group is unhealthy or not in use.
    Note: If no traffic is sent over the VPC link for 60 days, then it becomes INACTIVE. For more information, see Working with VPC links for HTTP APIs.
  • Security groups don't allow traffic on specific ports.
  • The network access control list (ACL) is blocking the traffic.
  • The target isn't listening on the target ports.
  • The domain name certificate isn't matching with the TLS enabled Network Load Balancer (NLB) or Application Load Balancer (ALB).

If Amazon CloudWatch logging is activated for your API, then an error message indicating the cause of the error appears in the execution logs.

Example HTTP 500 error messages in CloudWatch logs

Thu Aug 04 17:48:21 UTC 2022 : Execution failed due to configuration error: There was an internal error while executing your request
Thu Aug 04 17:48:21 UTC 2022 : Method completed with status: 500

-or-

Thu Aug 04 19:50:21 UTC 2022 : Execution failed due to configuration error: Host name 'domain.com.com' 
does not match the certificate subject provided by the peer (CN=myinstance.com)
Thu Aug 04 19:50:21 UTC 2022 : Method completed with status: 500

Resolution

Confirm the cause of the error

If you haven't done so already, turn on CloudWatch logging for your REST API. Make sure that you configure execution logging.

Tip: When configuring the logging settings, for Log level, choose INFO. Then, choose Log full requests/responses data.

Identify what's causing the errors by viewing the REST APIs execution logs in CloudWatch.

Check for the error message in the execution logs and follow these resolution steps

"Execution failed due to configuration error: There was an internal error while executing your request"

  • Verify that the VPC Link load balancer exists and hasn't been deleted.
  • Verify that the VPC Link is in the AVAILABLE state. If the VPC Link is in the FAILED state, then you must create a new VPC link and associate it with your API. Note: Be sure to deploy the API after modifying the integration request.
  • Verify that the VPC Link endpoint connection is in the AVAILABLE state.
  • If the VPC ID is referenced with a stage variable, then verify that the VPC Link ID is correct.
  • Verify that the VPC Link load balancer is listening on the HTTP/HTTPS port that the request is configured for. Check that the listeners are configured for the correct port. Check the network ACLs and make sure that they aren't blocking the request.
  • Verify that the target group is accepting the request. Check network ACLs to make sure that traffic is allowed inbound and outbound. Check the security groups to make sure that traffic is allowed inbound for the configured ports.
  • If the request returns a 500 error, the connection might receive TCP RST packets. Verify that the backend server is up and running. Verify that the backend target has a service running on the target port. Verify that the backend target is listening on the target port.

"Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)"

Make sure that the endpoint domain name matches the certificate that's returned by the TLS enabled load balancer target.

"Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

This error means that the API Gateway can't find the root CA. Even if insecureSkipVerification is set to true on the integrations TlsConfig property, API Gateway performs basic certificate validation including:

  • The certificates expiration date.
  • The hostname.
  • The presence of a root certificate authority.

"Execution failed due to configuration error: Cannot verify ECDH ServerKeyExchange signature"

This error means that because a key and the corresponding certificate don't match, the TLS handshake fails. To resolve this, verify the contents of the files used for your configured certificate authorities, certificates, and keys.

"Execution failed due to an internal error"

This error occurs because Amazon API Gateway couldn't connect to the load balancer over the VPC Link due to target resets on the load balancer. To resolve this, set the timeout on the target to be higher than the default timeout (350 seconds) on the load balancer.

"Execution failed due to configuration error: Not a valid protocol version: {Text of string}"

This error means that the integration responded with an invalid HTTP response which doesn't adhere to the HTTP specification. This might indicate that the Integrated backend is sending invalid data back to Amazon API Gateway.

To troubleshoot this error, do the following:

  • Reproduce the request made from Amazon API Gateway. Test the service endpoint used by Amazon API Gateway by creating a VPC endpoint that's linked to the service endpoint ID. You can also run packet captures to review the response that's returned from the integrated backend.
  • Change the load balancers protocol to TLS or TCP depending on the targets response to avoid duplicate encryption.
  • Make sure that insecureSkipVerification is set to true on the integrations TlsConfig property.