How can I troubleshoot connectivity issues over my gateway and interface VPC endpoints?
Last updated: 2022-04-05
How can I troubleshoot connectivity issues over my gateway and interface Amazon Virtual Private Cloud (Amazon VPC) endpoints?
The following resolution covers various parameters that are important for establishing end-to-end connections over VPC endpoints. Steps for troubleshooting connectivity are also included.
Gateway VPC endpoints
Gateway VPC endpoints allow you to connect to Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB privately from your VPC.
When using gateway endpoints, there are four factors that affect connectivity to the respective services:
- The endpoint policy,
- The VPC subnet network access control list (ACL),
- The source subnet route table,
- The Source Amazon Elastic Compute Cloud (Amazon EC2) or AWS Lambda security groups,
When using a custom endpoint policy, make sure that the policy associated with the endpoint allows the necessary access to perform actions against the service. The default endpoint policy allows full access to the service.
For more information, see Endpoint policies for gateway endpoints.
VPC subnet network access control list
The subnet network ACLs must allow both inbound and outbound TCP connections to S3 and DynamoDB CIDRs within the AWS Region. You can view the IP CIDRs for S3 and DynamoDB in a given AWS Region by running the following command from your AWS Command Line Interface (AWS CLI).
Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.
aws ec2 describe-prefix-lists --region <AWS Region>
Note: Public IP address CIDRs for AWS services can change. To make sure that you have all the CIDRs allowed on the network access control list rules, re-run the preceding command intermittently to check for new CIDR ranges.
Source EC2 or Lambda security groups
The security groups associated with the source that's initiating the connectivity to the S3 and DynamoDB endpoints must allow the following:
- Egress (outbound) connections to the public IP CIDRs.
- The Prefix List ID for S3 and DynamoDB in the concerned AWS Region respectively.
Note: When configuring the security group and network ACL rules, refer to the Amazon S3 endpoints and DynamoDB endpoints documents to check for the supported protocols. Use this information to base your traffic restrictions on.
Subnet route table
When creating the gateway endpoint, select the route tables that the routes to the service will be installed on. Make sure that the source route table has routes with the Prefix List ID of the intended service pointing to the gateway VPC endpoint.
Confirming traffic flow over a gateway endpoint
When using gateway VPC endpoints, you can run tcptrace-route to port 80 or 443 to confirm that traffic is flowing over the endpoint:
sudo tcptraceroute s3.us-east-1.amazonaws.com 80 traceroute to s3.us-east-1.amazonaws.com (18.104.22.168), 30 hops max, 60 byte packets 1 * * * 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 s3-1.amazonaws.com (22.214.171.124) <syn,ack> 0.797 ms 1.298 ms 0.821 ms
In the preceding output, there are no hops seen except for the S3 endpoint destination. This shows that traffic is going over the VPC gateway endpoint. If you see multiple public hops on the TCP trace-route, then traffic is going over the internet.
For more information, see Why can't I connect to an S3 bucket using a gateway VPC endpoint?
Interface VPC endpoints (AWS PrivateLink)
When working with interface VPC endpoints, check the following:
DNS Name Resolution
For AWS services:
With private DNS names turned on, you can run AWS API calls against the service endpoints (for example, ec2.us-east-1.amazonaws.com). These resolve to the private IPs of the endpoint interfaces.
If you don't have private DNS names turned on, then you can run the API calls by explicitly specifying the Regional or zonal VPC endpoint DNS name.
For more information, see Why can't I resolve service domain names for an interface VPC endpoint?
Use the dig or nslookup commands to verify the DNS resolution for the interface VPC endpoint name that you're trying to connect to.
Endpoint services (partner or customer-managed services)
Make sure that there are targets in the backend in all of the Availability Zones enabled for the Network Load Balancer on the provider's side. For more information, see Why can't I connect to an endpoint service from my interface endpoint in an Amazon VPC?
For example, assume that the Network Load Balancer is provisioned for serving in two Availability Zones, us-east-1a and us-east-1b. Provision targets on both of the Availability Zones. If there are no targets in the Availability Zones, then turn on cross-zone load balancing to serve all of the requests.
The default policy allows full access to the service. When you're using a custom policy, make sure that the policy allows access to perform the necessary actions against the service.
VPC endpoint security group
With interface VPC endpoints, you can associate security groups to control access. Make sure that the inbound rules for the security group allow communication between the endpoint network interface and the resources (VPC or on-premises) that access the service.
Depending on the ports that the service is accepting connections on, allow connections to that port and protocol.
The endpoint service Network Load Balancer is listening on TCP ports 6169 and 8443. In this case, allow TCP traffic to ports 6169 and 8443 from the appropriate source addresses on the VPC endpoint security group inbound rules.
The outbound rules of the security group aren't evaluated for interface endpoints. So, you can leave these blank or restricted.
Subnet network ACLs
The subnet network ACLs need to allow both inbound and outbound connections to the interface endpoint elastic network interfaces from the source networks when connecting from outside of the VPC.
Interface VPC endpoints can be used to access services privately from within AWS or from an on-premises network.
When connecting from within the same VPC as the interface endpoint, the routing is taken care by the local route in the subnet route tables. So, no additional routing configurations are needed.
If you're connecting to the endpoint from outside of the VPC (cross-Region VPC or on-premises network), make sure that connectivity from one or more source networks to the interface VPC endpoint elastic network interface subnets can be established.
Testing connectivity to interface VPC endpoints
To test whether you can reach the service using an interface endpoint, you can use network connectivity tools against the appropriate ports.
The following is an example of testing connectivity to an interface endpoint for an AWS service:
telnet ec2.us-east-1.amazonaws.com 443
telnet PrivateIPofInterfaceEndpointENI 443
The following is an example of testing connectivity to an interface endpoint, assuming that the provider you're connecting to is listening on port 6169:
telnet vpce-aaaabbbbcccc-dddd.vpce-svc-12345678.us-east-1.vpce.amazonaws.com 6169
telnet PrivateIPofInterfaceEndpointENI 6169
If you're using domain names, make sure that the connection is being made to the correct IP, and whether it's successful. For SSL verification, you can use curl or OpenSSL tools for testing.
For endpoint services, you can contact the provider to troubleshoot the Network Load Balancer.