How do I troubleshoot network performance issues between Amazon EC2 Linux instances in a VPC and an on-premises host over an internet gateway?

Last updated: 2019-09-24

I'm experiencing network performance issues between my Amazon Elastic Compute Cloud (Amazon EC2) Linux instances and my on-premises host over an internet gateway. How can I troubleshoot these packet loss or latency issues?

Resolution

Note: Before you begin troubleshooting, identify the source and destination IP addresses. If the destination is a URL, use the dig command to determine the IP address. Be aware that some URLs use dynamic IP addresses, so the IP address might change. Run the command multiple times to see if the IP address is constant.

Check for network issues using traceroute

1.    Run the following command to install traceroute.

Linux:

sudo yum install traceroute

Ubuntu:

sudo apt-get install traceroute

2.    Run the following command to execute traceroute.

ICMP-based traceroute:

sudo traceroute <Public IP of EC2 instance/on-premises host>

TCP-based traceroute:

sudo traceroute -T -p <destination port> <Public IP of EC2 instance/on-premises host>

3.    Review the traceroute output.

  • Determine whether the source of the latency is any of the hops in your internal network.
  • If there's a sudden spike in latency that progresses towards the destination, there might be an issue at the hop where the spike began.
  • If there's high latency in the intermediate hops but no latency at the beginning or end of the output, there might be no issue. The high latency in this case might be the result of an intermediate hop that deprioritized the traceroute traffic.

Troubleshoot packet loss issues using MTR

1.    Run the following command to install MTR.
 
Linux:
sudo yum install mtr

Ubuntu:

sudo apt-get install mtr

2.    Run the following command to execute MTR.

ICMP-based MTR:

mtr -n -c <number of packets> <Public IP EC2 instance/on-premises host>

TCP-based MTR:

mtr -n -t -p <destination port> -c <number of packets> <Public IP EC2 instance/on-premises host>

3.    Review the MTR output. Review hops using a bottom-up approach. For example, check for loss on the last hop or destination and review the preceding hops.

Check for source host or destination host issues

If you have access to the source or destination host:

1.    Run the top command to identify issues with CPU and memory utilization or load average.

2.    Review the top output:

  • The third row in the output indicates the percentage of CPU in use. A high CPU percentage results in degraded performance.
  • The fourth row indicates RAM usage, including total memory in use, total memory free, and total buffers cached.
  • If CPU or memory utilization is high, determine the cause, and then make adjustments as possible.

Check for application issues

If the MTR and traceroute output don't show anything suspicious, the latency might be application-induced. To see if application issues are the cause, use the hping3 tool to send crafted packets to the port on which the application is listening.

1.    Run the following command to install hping3.

Linux:

sudo yum --enablerepo=epel install hping3

Ubuntu:

sudo apt-get install hping3

2.    Run the following command to send TCP SYN packets over the destination port.
Note: By default, hping3 sends TCP headers to the destination host's port 0 with a window size of 64 and no TCP flag.

hping3 -S -c <number of packets> -V <Public IP of EC2 instance/on-premises host> -P <destination port>

3.    Review the hping3 output to see the number of packets transmitted, the number of packets received, and the amount of packet loss, if any. By using this data, you can determine whether the application is catering some or all of the requests made to it.

Take simultaneous packet captures on the source host and destination host using tcpdump

If none of the above steps help you to determine what's causing the performance issue, you can take a packet capture to investigate further. You can use packet captures to determine issues at the host level or with the TCP stack. Packet captures can also show any significant amounts of retransmission, fragmentation, or malformed packets.

1.    Run the following command to install tcpdump.

Linux:

sudo yum install tcpdump

Ubuntu:

sudo apt-get install tcpdump

2.    Run the tcpdump command to test packet capture samples.


Did this article help you?

Anything we could improve?


Need more help?