Visualize Amazon EC2 based VPN metrics with Amazon CloudWatch Logs

Organizations have many options for connecting to on-premises networks or third parties, including AWS Site-to-Site VPN. However, some organizations still need to use an Amazon Elastic Compute Cloud (Amazon EC2) instance running VPN software, such as strongSwan.

Gaining insight into Amazon EC2-based VPN metrics can be challenging when compared to AWS native VPN services that feature Amazon CloudWatch integration. This post aims to help surface those important metrics, so that administrators can better monitor the status and performance of their EC2 based VPNs.

Publishing these metrics allow administrators to keep meaningful network metrics in CloudWatch to correlate potential VPN issues with other AWS metrics and logs.

To learn more about running strongSwan on an EC2 instance, take a look at this blog post.

Solution overview

In our scenario, we will export key metrics from an EC2 instance running strongSwan and FRRouting to CloudWatch, including latency to a VPN target and the number of BGP prefixes received. We will also export the actual BGP prefixes that are present in the routing table of the EC2 instance to Amazon CloudWatch Logs. This lets administrators troubleshoot potential VPN and routing issues without the need to log in to the EC2 instance.

FRR works along with strongSwan, and it is used to manage dynamic routing. In this case, we’ll use the BGP protocol, though simple modifications could be made to accommodate other dynamic routing protocols.

Walkthrough

When the AWS CloudFormation template is deployed, it will create an AWS Identity and Access Management (IAM) role that is attached to an EC2 instance that you specify. The permissions include access to put metrics into CloudWatch and to put data into a specific CloudWatch Logs log stream.

Prerequisites

To follow along with this blog post, you should have the following:

• An AWS account
• An EC2 instance running Ubuntu or Amazon Linux 2 with strongSwan and software capable of managing dynamic BGP routing, such as FRR or Quagga
• An active VPN connection from your EC2 instance to another VPN device
• A target IP address on the remote end of the VPN that is capable of receiving ICMP traffic

Steps

As an overview, we will follow these steps:

• Deploy the CloudFormation template
• Attach the IAM instance profile to the EC2 instance
• Copy the bash file to your EC2 instance running strongSwan
• Install the command-line interface (AWS CLI) on the EC2 instance
• Create a cron job to schedule metric delivery to CloudWatch and CloudWatch Logs
• View the CloudWatch dashboard to visualize the metrics and verify metric delivery

Launch the CloudFormation stack

1. Download the CloudFormation template associated with this blog post
2. Log in to the AWS console, and navigate to the CloudFormation console
3. Select the Create stack button
4. Upload the template file that you previously saved, and select Next
5. Enter a name for your CloudFormation stack, and fill in the parameters required by the template

Figure 1: Launching the CloudFormation stack

6. Select Next, and optionally specify any tags to apply
7. Select Next once more, review the details, select the checkbox to acknowledge that the CloudFormation template will create IAM resources, and select Create stack
8. You can monitor the resource creation progress on the Events screen

Attach the instance profile to your EC2 instance

1. In the EC2 console, select the instance running strongSwan
2. From the Actions dropdown in the top right corner, select Security and then select Modify IAM role
3. From the dropdown menu, choose the role called “EC2-CloudWatch-Metrics”, as depicted in Figure 2 (this name may differ if you modified the default options when deploying the CloudFormation template)

Figure 2: Modify the IAM role on your EC2 instance

Create the bash script in your EC2 instance

Figure 3: Creating the vpn-metrics.sh script on your EC2 instance

1. Log in to the EC2 instance running strongSwan
2. Create a new file, and copy the below script into it, as shown above in Figure 3 (the script is also available to download here)

   #!/bin/bash
   # User customizations
   set -e
   source_ip='<EC2 instance IP>'
   target_ip='<Target IP to monitor latency>'
   
   # Instance metadata
   instance_region=$(curl -s "http://169.254.169.254/latest/meta-data/placement/region")
   instance_id=$(curl -s "http://169.254.169.254/latest/meta-data/instance-id")
   
   # Gather latency metrics for target
   latency_output=`ping -I $source_ip $target_ip -c 5 | tail -n 1`
   if [[ `echo $latency_output` == *\/* ]]
   then
       latency_avg=`echo $latency_output | cut -d "/" -f5`
   else
       latency_avg=0   
   fi
   aws cloudwatch put-metric-data --metric-name LatencyAverage --namespace EC2-VPN --value $latency_avg --region $instance_region --dimensions InstanceId=$instance_id,Target=$target_ip
   
   # Gather and format BGP data
   bgp_route_count=`ip route list | grep -c bgp`
   aws cloudwatch put-metric-data --metric-name ReceivedBGPRouteCount --namespace EC2-VPN --value $bgp_route_count --region $instance_region --dimensions InstanceId=$instance_id
   bgp_routes=`ip route list proto bgp | grep "/" | cut -d " " -s -f1`
   epoch_now_ms=`date +%s%3N`
   log_event_body={'"timestamp"':"$epoch_now_ms",'"message":"{''\"aggregate_cidr\"':\\"\"BGP routes received: $bgp_routes\\\"}\"},"
   
   for cidr in $bgp_routes 
   do
       epoch_now_ms=`date +%s%3N`
       log_message={'"timestamp"':"$epoch_now_ms",'"message":"{\"bgp_route\":\"'$cidr\\\"\}"\"}"
       log_event_body=`echo "$log_event_body$log_message"","`
   done
   log_event_body_json="[${log_event_body::-1}]"
   
   echo $log_event_body_json > bgp_json.txt
   
   # Add BGP data to CloudWatch Logs
   DescribeLogStreamsOutput=$(aws logs describe-log-streams --log-group-name EC2-VPN-BGPRoutesReceived --log-stream-name-prefix $instance_id --region $instance_region --output text)
   SequenceToken=`echo $DescribeLogStreamsOutput | cut -d " " -f9`
   
   if [ -z "$SequenceToken" ]
   then
       aws logs put-log-events --log-group-name EC2-VPN-BGPRoutesReceived --log-stream-name $instance_id --log-events file://bgp_json.txt --region $instance_region
   else
       aws logs put-log-events --log-group-name EC2-VPN-BGPRoutesReceived --log-stream-name $instance_id --log-events file://bgp_json.txt --sequence-token $SequenceToken --region $instance_region
   fi

   1. $ nano vpn-metrics.sh
   2. Paste the script into the file
   3. Modify the target_ip variable, and specify a target IP address on the other end of the VPN. Make sure that it accepts ICMP echo-request messages. Additionally, specify a source IP address on your local EC2 instance for the source_ip variable. This is where the ICMP messages will be sourced from.
   4. Save the file
      1. Press ctrl-X to exit the editor
      2. Press Y to save the changes
3. Give the new file execute permissions
   1. $ chmod +x vpn-metrics.sh

Install the AWS CLI

If you’re using an operating system other than Amazon Linux 2, then you will need to install the AWS CLI.

$ sudo apt install awscli

Create a cron job

A cron job creates a scheduled task that runs at a specified time or interval.

Follow these steps for Amazon Linux 2:

1. $ crontab -e
2. Add this line to execute the script every five minutes:
   1. */5 * * * * /home/ec2-user/vpn-metrics.sh

Follow these steps for Ubuntu:

1. $ crontab -e
2. Add this line to execute the script every five minutes:
   1. */5 * * * * /home/ubuntu/vpn-metrics.sh

When the crontab is saved, crontab: installing new crontab appears.

View the CloudWatch dashboard

1. Navigate to the CloudWatch console, and select Dashboards.
2. Select the dashboard titled “EC2-VPN-Dashboard”.
3. If the cron job has already run, then you will see metrics populated in the three dashboard widgets (if you do not see any metrics yet, then wait a few minutes for CloudWatch to populate them).
4. New metrics will continue to appear after the cron job runs and executes the script. Use the refresh icon in the upper right of the CloudWatch console to see new metrics appear in the dashboard widgets. Figure 4, depicts metrics for all three widgets.

Figure 4: Viewing the CloudWatch Dashboard

The Amazon CloudWatch Contributor Insights widget in the bottom left of Figure 4 will show when a BGP prefix is no longer present in the BGP route table of the EC2 instance. This can be very useful in troubleshooting a scenario where routes from multiple BGP sources are present. The dip in the ReceivedBGPRouteCount widget in Figure 4 correlates with a temporary loss of the prefix 10.26.0.0/23, shown in the BGP Prefix Insights widget in Figure 5.

Figure 5: BGP Prefix Insights

The Contributor Insights rule can also be viewed in a standalone fashion by visiting the Contributor Insights link under the CloudWatch service page.

Cleaning up

Disable the cron job running on your EC2 instance to avoid incurring charges. To do this, execute the steps below:

1. $ crontab -e
2. Remove this line: */5 * * * * /home/ec2-user/vpn-metrics.sh (replace ec2-user with ubuntu, if the EC2 instance is running Ubuntu)

To deprovision the CloudWatch dashboard, log group, and IAM role, delete the CloudFormation stack that was deployed.

Conclusion

This post demonstrated how to publish custom CloudWatch metrics from an EC2 instance to a CloudWatch dashboard and custom namespace. Publishing these metrics lets administrators view key performance metrics for an EC2-based VPN, and have the ability to create CloudWatch alarms if desired.

To publish other EC2 metrics not covered in this post, take a look at the CloudWatch agent to publish in-guest, system-level metrics.

About the author