Traffic Mirroring is an Amazon Virtual Private Cloud (VPC) feature you can use to copy network traffic from an elastic network interface of an Amazon Elastic Compute Cloud (EC2) instance and send it to a target storage service for analysis. You can use it for content inspection, threat monitoring, network performance monitoring, and troubleshooting. Through using traffic mirroring, you can analyze specific traffic patterns to identify any vulnerable “blind spots” or “choke points” between application tiers and/or Amazon EC2 instances. VPC Traffic Mirroring also supports filters and packet truncation, so that you can extract only the traffic of interest, using the monitoring tools of your choice.

In our previous posts, Using VPC Traffic Mirroring and Introduction to Traffic Mirroring to GWLB Endpoints as Target, we talked about how to use the native VPC Traffic Mirroring feature and further use AWS Gateway Load Balancer (GWLB) as targets in a centralized way. In this post, we look at how you can use these features and further export the Packet Captures (PCAPs) to a centralized Amazon Simple Storage Service (S3) bucket using Mountpoint for Amazon S3. The Mountpoint for Amazon S3 is used to mount this centralized bucket and access it as a local file system, thus providing an optimized high-throughput performance in storing and accessing the generated PCAPs. The design will involve using AWS GWLB, Amazon EC2 Auto Scaling Groups, Amazon S3 and VPC Traffic Mirroring. Before implementing the design, we recommend reviewing  VPC Traffic Mirroring limitations and quotas.

Traffic Mirroring Solution Architecture

This solution creates persistent Traffic Mirroring sessions on EC2 source instances and forwards them to a GWLB target. The GWLB has a target group of EC2 instances in an Auto Scaling Group that perform Transmission Control Protocol (TCP) Dump and export PCAPs to a centralized S3 bucket. The Agent nodes capture the packets on a regular schedule (for example, every 900 seconds) and dumps them to the configured S3 bucket that acts as centralized storage.

Our sample solution creates EC2 instances in two different Spoke accounts for sources, and creates the mirroring session. The centralized GWLB, fleet of TCP Dump agents, and S3 bucket to store PCAPs reside in the Hub account. This is shown in the following diagram (figure 1).

Figure 1: Architecture for Packet Captures using VPC Traffic Mirroring

Deploying solution modules

Our solution is based on Hashicorp Terraform and consists of key modules for Traffic Mirroring components and GWLB in the Hub account, source EC2 instances and GWLB endpoints in Spoke accounts.

The GWLB Endpoint Service and Endpoints can be deployed in different architecture patterns:

• Method-1: Allowing all principals and controlling the Acceptance feature. For simplicity and automation purposes, our solution follows this method. This can be changed to require manual acceptance by updating the variable “enable_manual_acceptance” in terraform.tfvars.
• Method-2: Allowing only certain principals (through variables) and turning on the Auto Acceptance.

Hub account components: (Module: gwlb_target)

This includes GWLB, GWLB Endpoint Service, Auto Scaling Group with agents installed on the EC2 instances to perform TCP Dump, and S3 bucket to store PCAPs.

Spoke accounts components: (Modules: ec2_src, tm_session)

This includes source EC2 instances, GWLB Endpoints and Traffic Mirror components, like session, filter, and rules.

Deployment steps

1. Clone the project from the GitHub source.
2. Update the providers.tf file with the necessary Assume roles for the both Hub and Spoke accounts.
3. Update terraform.tfvars with the required values.
4. Run the following Terraform commands in your code terminal to deploy the required components.

Prepare your working directory: terraform init

Check whether the configuration is valid: terraform validate

Show changes required by the current configuration: terraform plan

Create or update infrastructure: terraform apply --auto-approve

5. Verify the successful deployment of components (listed as follows) in the Hub and Spoke accounts.

Key entities deployed

• A Traffic Mirroring target for the GWLB endpoint (Target A) in Spoke VPC1
• A Traffic Mirroring target for the GWLB endpoint (Target B) in Spoke VPC2
• A Traffic Mirroring filter with a Traffic Mirroring rule for the TCP inbound traffic (Filter 1) for the GWLB endpoint
• A Traffic Mirroring session for Spoke VPC1 that has the following:
  • A Traffic Mirroring source
  • A Traffic Mirroring target (Target A) for the GWLB endpoint
  • A Traffic Mirroring filter (Filter 1) with a Traffic Mirroring rule for the TCP inbound traffic
• A Traffic Mirroring session for Spoke VPC2 that has the following:
  • A Traffic Mirroring source
  • A Traffic Mirroring target (Target B) for the GWLB endpoint
  • A Traffic Mirroring filter (Filter 1) with a Traffic Mirroring rule for the TCP inbound traffic
• The GWLB is deployed in a Hub VPC with multiple appliances behind an Auto Scaling Group as targets. The targets have a TCP Dump service installed that generates PCAPs (every 15 minutes).
• A centralized S3 bucket that stores the PCAP files. We mount an S3 bucket using Mountpoint for Amazon S3 to write back the generated PCAPs. Details around this new feature are discussed here.

Viewing PCAP files

• Once you have deployed the components, the Traffic Mirroring sessions (Spoke account) should start mirroring traffic and the targets would forward it to the Endpoint Service (Hub account), as shown in the following screenshots (figures 2 and 3).

Figure 2: Traffic Mirroring sessions created in Spoke account

Figure 3: Traffic Mirroring targets in Hub account

• Now, log in to the Hub account where you deployed the TCP Dump agents and verify the PCAP generation. The following services should be in the running state:

trafficmirror-tcpdump.service: Service that performs TCP Dump operation and generates PCAP every 900 seconds (configurable in the run_tcpdump.sh) (figure 4)

systemctl status trafficmirror-tcpdump.service

Figure 4: Traffic Mirror TCP Dump systemd service status

trafficmirror-svc-alerts.service: Service that sends Amazon Simple Notification Service (SNS) email alert if TCP Dump service fails (figure 5)

systemctl status trafficmirror-svc-alerts.service

Figure 5: Traffic Mirror alerts systemd service status

• The centralized S3 bucket that was created is mounted to the packet-captures directory (figure 6)

Figure 6: S3 Mountpoint on the packet captures directory

• The packet-captures directory has a folder with the current instance ID that stores all the generated PCAPs. The PCAPs are stored in the YYYY-MM-DD-HH-MM format (figure 7).

Figure 7: Generated PCAPs stored in S3 Mountpoint directory

• The PCAPs are also available in the configured S3 bucket (Hub account) as shown in the following screenshot (figure 8):

Figure 8: PCAPs stored in Amazon S3 bucket

• You can use tools like Wireshark to access these PCAPs from Amazon S3 (without the need for local storage) and analyze them.

Cleaning up

Leaving resources that you no longer need in your AWS account may incur charges. Run the following command to delete the infrastructure that was created as part of deploying the solution discussed in this post:

Destroy the infrastructure: terraform destroy --auto-approve

Conclusion

In this post, we have shown how to perform event driven packet captures and store them in Amazon S3. We also leveraged the Mountpoint for Amazon S3 to take advantage of the elastic throughput. The generated packet captures can be downloaded and viewed using third-party tools, like Wireshark, for troubleshooting or identifying security threats. The script executed on agents installed on the EC2 instances can be changed to adjust the packet size or have the captures done in a time window.

About the Authors