How do I resolve the ErrorPortAllocation error on my NAT gateway?
Last updated: 2019-07-23
I keep getting the ErrorPortAllocation error on my NAT gateway and several concurrent connections to my destination port are failing. How do I resolve this port allocation error?
NAT gateways support up to 55,000 simultaneous connections to each destination per minute. If this threshold is crossed, new connections to that destination fail and the ErrorPortAllocation metric for the NAT gateway increases in Amazon CloudWatch. Use the following guidance to find the source clients and destinations that are causing these errors. Then, follow the steps to resolve them.
Find source clients and their connection destinations
1. Open the CloudWatch console.
2. In the navigation pane, choose Insights.
3. From the dropdown, choose your log group.
4. To find where traffic is going, use the filter below. Note: Be sure to replace xxx.xxx. with the first two octets of your VPC CIDR. Also, replace NAT gateway Private IP with the private IP of your NAT gateway.
filter (srcAddr like 'NAT gateway Private IP' and dstAddr not like 'xxx.xxx.') | stats count(*) as numaccept by dstAddr | sort numaccept desc | limit 10
The results show the destination IP with the greatest number of Accept responses during the time frame when you received the port allocation errors.
5. To find which source clients are sending traffic to this destination, use the filter below. Note: Be sure to replace xxx.xxx. with the first two octets of your VPC CIDR. Also replace Public IP from above filter with the destination IP you received from the previous query.
filter (dstAddr like 'Public IP from above filter' and srcAddr like 'xxx.xxx.') | stats sum(bytes) as bytesTransferred by srcAddr, dstAddr | sort bytesTransferred desc | limit 10
Take steps to resolve port allocation errors
- Create a NAT gateway in each Availability Zone, and then distribute your clients across Availability Zones. Route traffic to the internet using a NAT gateway in the same Availability Zone as your client to reduce cross Availability Zone data charges.
- If you notice an increase in the IdleTimeoutCount metric in CloudWatch, configure your application or private instance to close idle connections so the NAT gateway can allocate the source port to new connections.
- Limit the number of connections that your clients can make to a single destination.
- If traffic is going to an Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB public IP in the same Region, use a gateway VPC endpoint instead of a NAT gateway. There are no data processing or hourly charges for using gateway VPC endpoints.
- If traffic is going to a public IP for an AWS service that supports interface VPC endpoints, use an interface VPC endpoint instead of a NAT gateway.