How do I troubleshoot packet loss on my VPN connection?
Last updated: 2018-07-27
I'm having continuous or intermittent packet loss and high latency issues on my VPN connection. What tests can I run to be sure that the issue isn't taking place inside my Amazon Virtual Private Cloud (Amazon VPC)?
Before beginning performance tests, launch and configure your Amazon Elastic Compute Cloud (Amazon EC2) Linux instances:
- Follow the steps in Launch an Instance to launch Linux instances in at least two different Availability Zones under the same VPC from which you can run network performance testing.
- For the best possible network performance, be sure that the instances support enhanced networking for Linux and launch the instances in the same VPC.
- If you're performing network testing between instances that aren't co-located in the same placement group or don't support jumbo frames, follow the steps to Check and set the MTU on your Linux instance.
- Complete the steps in Connect to your Linux instance to verify that you can access the instances.
Use mtr to check for ICMP or TCP packet loss and latency
Install the mtr network tool on both instances to check for any ICMP or TCP packet loss and latency. mtr provides continually updated output that allows you to analyze network performance over time. It combines the functionality of traceroute and ping in a single network diagnostic tool.
Install mtr on Amazon Linux:
sudo yum install mtr
Install mtr on Ubuntu:
sudo apt-get install mtr
Run the following tests between the private and public IP address of your EC2 instances and your on-premises host bidirectionally. The path between nodes on a TCP/IP network can change when the direction is reversed, and it's important to obtain mtr results in both directions.
The first mtr test is ICMP-based, but the second test has a -T option, which gives you a TCP-based result. The TCP-based result helps you determine if there is any application-based packet loss or latency on the connection. MTR version 0.85 and above has the TCP option.
Private IP tests:
mtr -n -c 200 <Private IP EC2 instance/on-premises host> --report mtr -n -T -c 200 <Private IP EC2 instance/on-premises host> --report
Public IP tests:
mtr -n -c 200 <Public IP EC2 instance/on-premises host> --report mtr -n -T -c 200 <Public IP EC2 instance/on-premises host> --report
Use the Linux traceroute utility to determine latency or routing problems
The Linux traceroute utility identifies the path that is taken from a client node to a specified destination node, as well as the time in milliseconds for each router identified in the path to respond to a request. This utility also calculates and displays the amount of time each hop takes before reaching its destination. If traceroute isn't installed, make sure to install it on your instance.
Install traceroute on Amazon Linux:
sudo yum install traceroute
Install traceroute on Ubuntu:
sudo apt-get install traceroute
Run the following tests between the private and public IP address of your EC2 instances and your on-premises host bidirectionally. The path between nodes on a TCP/IP network can change when the direction is reversed, and it's important to obtain trace route results in both directions.
Private IP tests:
sudo traceroute <private IP of EC2 instance/on-premises host> sudo traceroute -T -p 80 <private IP of EC2 instance/on-premises host>
Public IP tests:
sudo traceroute <public IP of EC2 instance/on-premises host> sudo traceroute -T -p 80 <public IP of EC2 instance/on-premises host>
Note: The arguments -T -p 80 -n perform a TCP-based trace on port 80. Be sure that you have port 80 or the port that you are testing with open in both directions.
The Linux traceroute option to specify a TCP-based trace instead of ICMP is useful because most internet devices deprioritize ICMP-based trace requests. A few timed-out requests are common, so watch for packet loss to the destination or in the last hop of the route. Packet loss that accumulates over several hops can also indicate a problem.
Note: When troubleshooting network connectivity using traceroute, it's helpful to run the command in both directions, from the client to the server and then from the server back to the client.
Use hping3 to determine latency or TCP packet loss problems
hping is a command-line oriented TCP/IP packet assembler/analyzer. In addition to ICMP echo requests, it supports TCP, UDP, and RAW-IP protocols, has a traceroute mode, the ability to send files between a covered channel, and many other features.
If hping3 isn't installed, run the following command on Amazon Linux:
sudo yum --enablerepo=epel install hping3
Then, run the following commands:
hping3 -S -c 50 -V <Public IP of EC2 instance or on-premises host> hping3 -S -c 50 -V <Private IP of EC2 instance or on-premises host>
Note: By default, hping3 sends TCP headers to the target host's port 0 with a winsize of 64 without any tcp flag on.
Packet capture samples using tcpdump
Performing packet captures on your EC2 instances (present in multiple Availability Zones) and your on-premises host when duplicating the issue helps to determine if there are any application or network layer issues on the VPN connection. Install tcpdump on your instance to perform packet captures.
Install tcpdump on Amazon Linux:
sudo yum install tcpdump
Install tcpdump on Ubuntu:
sudo apt-get install tcpdump
Note: Refer to your specific vendor documentation for instructions on how to check network devices for analysis and troubleshooting.