How can I determine whether my DNS queries to the Amazon provided DNS server are failing due to VPC DNS throttling?

Last updated: 2019-09-27

My DNS queries to the Amazon provided DNS server are failing. Are the DNS queries from my instance failing because of VPC DNS throttling?

Short Description

Amazon provided DNS servers enforce a limit of 1024 packets per second per elastic network interface (ENI). Amazon provided DNS servers reject any traffic exceeding this limit.

VPC Flow Logs don't capture the traffic your application sends to Amazon provided DNS servers. You can use packet captures or Traffic Mirroring to identify the cause of the DNS query failures.

Resolution

First, use one of the following methods to identify the source of DNS query failures. Then, if you determine that the cause is DNS throttling, use one of the recommended fixes described below.

Option #1: Use tcpdump (Linux only)

1.     Use the following command to take rotating packet captures on your EC2 instance. The following command captures the initial 350 bytes of the packet and saves 20 files of 100 MB each while overwriting the old packet captures.

sudo tcpdump -i eth0 -s 350 -C 100 -W 20 -w /var/tmp/$(curl http://169.254.169.254/latest/meta-data/instance-id).$(date +%Y-%m-%d:%H:%M:%S).pcap

2.    Run the following Linux command to determine the number of DNS queries sent.

tcpdump  -r <file_name.pcap> -nn dst port 53 | awk -F " " '{ print $1 }' | cut -d"." -f1 | uniq -c

3.    If the number of DNS queries is greater than or equal to 1024 per second, any additional queries are throttled.

Option #2: Use Traffic Mirroring

If it's not feasible to take the tcpdump in your use case, you can leverage Traffic Mirroring to identify if DNS queries are throttled.

Note: Traffic Mirroring is only available for Nitro-based instances. Traffic Mirroring charges will apply.

First, capture traffic data:

1.    Complete the Traffic Mirroring prerequisites.
2.    Create a Traffic Mirror target. Confirm that the target elastic network interface or Network Load Balancer allows inbound traffic on port 4789.
3.    Create a Traffic Mirror filter. Under Filter settings, confirm that amazon-dns is enabled for Network services - optional.
4.    Create a Traffic Mirror session. After you've configured Traffic Mirroring, mirrored traffic is gathered and stored on the Traffic Mirror target.

Then, analyze the captured data using Wireshark:

1.    Open the captured traffic in Wireshark.
2.    Choose the Statistics tab.
3.    Select the I/O Graph and clear all options.
4.    (For Linux only) Under Display Filter, add a filter using the VXLAN Network Identifier and DNS query flag. For example, if the VXLAN Network Identifier is 53 and the DNS query flag is 0x0100, the display filter for the graph is (vxlan.vni == 53) && (dns.flags == 0x0100).
5.    Review the graph to check whether it flatlines around 1024 (the Amazon provided DNS server's packet per second limit). If the graph flatlines around this value, DNS throttling is present.

Fixes for DNS throttling issues

If you find that the cause of your DNS failures is DNS throttling, you can:


Did this article help you?

Anything we could improve?


Need more help?