Mirror production traffic to test environment with VPC Traffic Mirroring

Many organizations want to replay production traffic to a test environment, with no impact on the end user’s experience. This is known as traffic mirroring or traffic shadowing. Testing the new version of a workload with production traffic is a key step for a successful release. Some tests use scripted requests, but real traffic is often a better or complementary approach because, in many cases, it is not possible to anticipate and script all potential client behaviors.

In this blog post, you first learn how to use VPC Traffic Mirroring to replay HTTP requests to a test environment. Then, you set it up with the code that is provided. This approach does not add any proxy or software on your production environment. Therefore, it minimizes risk and does not affect performance.

Problem statement

There are two ways to test with production traffic: splitting and mirroring.

Figure 1 – traffic splitting and traffic mirroring

With traffic splitting, you route some requests to the new version and the remaining requests to the base version. This approach is used in canary releases, A/B testing, and blue/green deployments. It is usually easier to implement, because it does not generate duplicate data. But it exposes some clients to the new version being tested, so the risk is that problems are discovered only when the clients experience them.

With traffic mirroring, all requests are routed to the base version and replayed to the new version. The client receives responses only from the base version; responses from the new version are discarded. Mirroring is safe and low impact, because all clients are still handled by the base version. One use case is a workload where zero requests can be lost or degraded. For example, an e-commerce platform or the acquisition of critical banking data. Compared to traffic splitting, traffic mirroring has the additional requirement of preventing data duplication. For example, an idempotent application is needed when requests must only be processed once even if duplicate requests are received, such as for bank payments or e-commerce orders.

AWS has several tools for traffic splitting: at the DNS level, Amazon Route 53 has weighted policies; when load balancing, Application Load Balancer has weighted target groups; for serverless applications, Amazon API Gateway has canary release deployments; and many others.

For traffic mirroring, open-source solutions exist. For example, NGINX mirroring capability, or for microservices, Envoy request mirror policy. But the existing solutions require that you either add a proxy in front of your production workload or install an agent on your servers. The addition of these components to a production environment might not be feasible, might be risky, or might impact performance.

Proposed solution

VPC Traffic Mirroring enables a new approach for mirroring HTTP requests. VPC Traffic Mirroring replicates traffic out-of-band with no performance impact and without the need to add any components on, or in front of, your servers.

The following diagram (figure 2) shows the architecture of a solution using VPC Traffic Mirroring.

Figure 2 – architecture diagram

This architecture consists of three components:

1. Production environment: The sources of VPC traffic mirroring are the elastic network interfaces (ENIs) of the Amazon Elastic Compute Cloud (EC2) instances in this environment. You can have one or more source EC2 instances, including instances that are added dynamically, such as an Auto Scaling group (see section “Working with multi-instance environments”).
2. Replay handler: this is where the mirrored packets are processed. The target of the mirrored packets is a Network Load Balancer, which in turn distributes them to a handler script that runs on EC2 instances. You can use multiple handler EC2 instances for high availability and scalability. VPC traffic mirroring encapsulates packets with a VXLAN header. Therefore, the script decapsulates them, reassembles them into HTTP requests, and replays the requests to the test environment. Responses from the test environment are sent to the handler, which ignores them.
3. Test environment: this is your test or pre-production environment. The entry point of this environment can be either a single IP address or a DNS name. The replay handler sends the replicated requests to this entry point.

The replay handler is provided as an open-source CloudFormation template on GitHub. You can launch it with one click or you can customize it.

Walkthrough

In this section, you deploy the proposed solution.

Prerequisites

You first need a VPC with private subnets that route internet-bound traffic to a NAT Gateway (NAT Gateway enables instances of the replay handler, which do not have a public IP, to download the handler script from the internet). If you don’t have such a VPC yet, you can launch it using this CloudFormation template.

Configuring the production and test environments

Launch two sample environments: production and test. This step is needed for demonstration purposes only. If you already have existing environments, you can skip this step and go to the next step (“Launching the replay handler”).

1. Launch two identical EC2 instances. Use the Amazon Linux 2 AMI (see launch an instance) and a security group that allows TCP traffic on port 80 from within the VPC.
2. Change the Name of the instances: call one “Production”, and the other “Test”.
3. Connect to both EC2 instances using SSH or one of the methods to connect to Linux instances.
4. Configure Apache Web Server on both instances by running sudo yum install -y httpd and sudo systemctl start httpd.

Figure 3 – newly launched EC2 instances

Launching the replay handler

Start by using the AWS CloudFormation template to configure the replay handler.

1. Select “Launch Stack” by clicking on the button below:
   
2. Set the parameters appropriately. For this demonstration, do the following:
   1. For “HandlerVpcId” specify the VPC for the replay handler. To keep this demonstration simple, use the same VPC where you have created the EC2 instances “Production” and “Test” (but you could use a separate VPC, as long as it has connectivity to the VPCs of the production and test environments).
   2. For “HandlerSubnetIds” specify the subnets for the replay handler. Use subnets that have the following features: a route to and from the subnets of the EC2 instances “Test” and “Production”; a route to a NAT Gateway for internet-bound traffic (so that instances of the replay handler, which do not have a public IP, can download the handler script from the internet). For high availability use at least two subnets in separate availability zones.
   3. For “RequestsForwardDestination” use the HTTP protocol and the IP address of the EC2 instance “Test”. Use the following format: “http://<ip-address-here>”.
   4. For “TrafficMirroringVNI” use an integer between 1 and 16777215. For example, “1234”.
3. Select the capability “I acknowledge that AWS CloudFormation might create IAM resources”.
4. Choose Create Stack.
5. Wait until the status of the stack transitions to “CREATE_COMPLETE”. Then choose Outputs and take note of the output values.

Figure 4 – outputs of the CloudFormation stack

Configuring traffic mirroring on the production environment

Now configure a VPC traffic mirroring session on the production environment.

1. Go to the VPC Console.
2. Under Traffic Mirroring, choose Mirror Sessions and Create traffic mirror session.
3. For Mirror source choose the elastic network interface (ENI) of the “Production” instance
4. For Mirror target choose the value of the output “TrafficMirrorTarget” of your CloudFormation stack (see point 5 in the previous step).
5. For Session number use “1”.
6. For VNI use the value of the parameter “TrafficMirroringVNI” of your CloudFormation stack (see point 2.e from the previous step).
7. For Filter use the value of the output “TrafficMirrorFilter” of your CloudFormation stack (see point 5 from the previous step).
8. Keep the rest as default and choose Create.

Figure 5 – create a traffic mirror session

Testing the solution

When HTTP requests are sent to your production environment, they are automatically replicated to your test environment. To see that happening, you can visualize the Amazon CloudWatch metric “NetworkIn” of the production and test instances in a metric graph.

Figure 6 – CloudWatch graph for “NetworkIn” metric

Cleaning up

To clean up the resources we used in this example, delete the CloudFormation stack that you deployed and the two EC2 instances that you created. If you created a VPC in the prerequisites section, delete that VPC.

Additional capabilities

In this section, some additional capabilities of this solution are described. A more comprehensive list of parameters and capabilities is available in the documentation for the replay handler on GitHub. This is an open-source project, so if you need an unsupported feature, you can request it or contribute via a pull request (see contributing).

Working with multi-instance environments

In the walkthrough section, you have configured the replication from and to a single EC2 instance. But in the real world, the production and test environments would probably have multiple instances. For example, an Auto Scaling group fronted by Elastic Load Balancing. You can easily use this solution in a multi-instance environment. To do that, you must configure the sources and the target appropriately.

For multi-instance sources, you could use this source automation to configure traffic mirroring from multiple and newly launched EC2 instances in your production environment. The automation sets up the sources based on tags, VPCs, or subnets. For example, you can specify a tag that identifies the instances in the Auto Scaling group for the application layer. The source automation sets up traffic mirroring on all the existing instances in the group. When new instances are added by the Auto Scaling group, the automation sets up traffic mirroring also on them.

For multi-instance destination, you simply specify a DNS name as the target when you create the CloudFormation stack. For example, you provide the DNS name of the load balancer for the application layer of your test environment. The stack parameter to use is “RequestsForwardDestination”. You must specify a DNS name preceded by the protocol (HTTP or HTTPS). For example: “https://load-balancer-address.example.com”.

Replicating a fraction of the requests

When you deploy the CloudFormation stack for the replay handler, you can optionally specify additional parameters.

For example, you can use the parameter “ForwardPercentage” to define the percentage of requests that are replicated (by default, it is 100%). You can even choose to replicate only the requests that come from a percentage of header values or remote addresses – for example, to mirror all requests from some users (rather than some requests from all users). To do that, you set the stack parameter “PercentageBy” to “header” or “remoteaddr”. When “PercentageBy” is set to “header”, you must provide the header name in the parameter “PercentageByHeader”.

Conclusion

Traffic mirroring is a safe and effective technique to test new versions with production traffic with no impact on the client. In this blog post, you have learned how to mirror HTTP requests from a production environment to a test or pre-production environment using VPC Traffic Mirroring. This approach does not add any components to the production servers or in front of them. Therefore, it minimizes risk and does not affect performance. For more information on VPC Traffic Mirroring, see the documentation and this webinar.