Inspecting network traffic between Amazon VPCs with AWS Cloud WAN

AWS Cloud WAN is a managed service used to build wide area networks that connect data centers, branch offices, and Amazon Virtual Private Clouds (VPCs) using AWS global infrastructure. When building with Cloud WAN, many organizations employ next-generation firewalls (NGFW) and intrusion prevention systems (IPS) to inspect network traffic as part of a defense-in-depth strategy. This is often done using a separate and centralized security VPC where security appliances are set up, and traffic is routed in and out using Cloud WAN—creating a hub-and-spoke model. Having a separate security VPC provides a simplified and centralized way to manage security inspection. In this blog post, we take a closer look at centralized architectures for native East-West (VPC-to-VPC) inspection both within and across Regions with Cloud WAN.

The architectures we present in this post is commonly used in East-West (VPC-to-VPC) inspection. Inspecting egress/ingress, or North-South, traffic is a valid used case and was covered in an earlier blog post: Centralized outbound inspection architecture in AWS Cloud WAN.

Here, we will focus on the following four scenarios:

1. Inspecting traffic between VPCs in the same segment in the same Region (Isolated attachments)
2. Inspecting traffic between VPCs in different segments in the same Region
3. Inspecting traffic between VPCs in the same segment across different Regions (Isolated attachments)
4. Inspecting traffic between VPCs in different segments across different Regions

Concepts

Network segments play an important role in how traffic is routed through the core network and between AWS Cloud WAN attachments.

• Segments are dedicated routing domains. Hence, by default, only the attachments within the same segment can communicate. Using network segments, you can divide your global network into separate isolated networks. For example, isolate traffic between production and development, or between business units.
• While segments allow you to isolate the networks, using the core network policy parameter segment-actions, you can share/advertise segment routes with other segments, thus allowing attachments in isolated segments to communicate with attachments in the shared segment. For example, you have a centralized firewall service hosted in an inspection segment and you want to allow communication between production and development segments, such that traffic is first routed to a firewall in the inspection segment for inspection (the premise of this blog).
• Resources in the same segment can be isolated from each other, and only access shared services.

Security appliances are typically stateful and must process both forward and return traffic for a network flow. AWS Cloud WAN appliance mode allows to achieve this symmetric routing. When you create an attachment, you can turn on the appliance mode support for a VPC attachment where the appliance is located. This ensures that Cloud WAN uses the same Availability Zone for that VPC attachment for the lifetime of the flow of traffic between a source and destination. It also allows Cloud WAN to send traffic to any Availability Zone in the VPC as long as there is a subnet association in that zone.

While we briefly covered network segments and appliance mode, to get the most out of this post, you should be familiar with the following AWS Cloud WAN components:

• Global network
• Core network
• Core network policy
• Attachments
• Core Network Edge
• Network segment

Refer to the Cloud WAN documentation to learn more about Cloud WAN concepts. The Introducing AWS Cloud WAN blog post also provides a great overview of these concepts in action.

Architectures

Scenario 1 – Inspecting traffic between VPCs in the same segment in the same Region (Isolated attachments)

The diagram that follows (Figure 1) show scenario 1, where we have created two network segments, Production Segment and Inspection Segment 1. Even if two VPCs belong to the Production segment, you might want to inspect traffic traveling between them. To do this, we first isolate the VPC attachments mapped to the Production Segment and, using the core network policy parameter segment-action, share it with Inspection Segment 1.

Prod VPC 1 and Prod VPC 2 are located in the same Region 1, and we map them to the Production Segment. We mark these (Prod VPC 1 and Prod VPC2) attachments as isolated to prevent any direct communication between them. To allow them to communicate through a firewall, we share Inspection Segment 1 with the Production Segment. We show this as (1) Segment Sharing in Figure 1.

Figure 1: Inspecting traffic between VPCs in the same segment in the same Region

Packet walkthrough

The following steps describe a packet walkthrough when instance 1 in Prod VPC 1 communicates with instance 3 in Prod VPC 2:

• (A) When Instance 1 in Prod VPC 1 starts a connection to Instance 3 in Prod VPC 2, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and the packet gets routed to the Core Network.
• (B) When the packet arrives at the core network, because Prod VPC 1 is associated with the Production Segment, it does a Production Segment Route Table lookup. The packet matches the default entry with Inspection VPC A attachment as the target and the packet gets routed to Inspection VPC A.
• (C) When the packet arrives at the Inspection VPC A attachment, it does a VPC (CWAN Subnet) route table lookup. The packet matches the default route with Firewall Endpoint 1 as the target and the packet gets routed to a firewall, through the firewall’s endpoint, for inspection.
• (D) The firewall inspects the traffic, compares it to its security policy, and allows it through. The firewall routes the packet back to the firewall’s endpoint, where it does a VPC (Firewall Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target, and the packet gets routed to the Core Network.
• (E) When the packet arrives at the core network, because Inspection VPC A is associated with the Shared Inspection Segment 1, it does a Shared Inspection Segment 1 Route Table lookup. The packet matches the Prod VPC 2 CIDR entry with Prod VPC 2 Attachment as the target and the packet gets routed to Prod VPC 2.
• (F) When the packet arrives at Prod VPC 2, it does a VPC (CWAN Subnet) route table lookup. The packet matches the VPC CIDR entry with local as the target and the packet gets routed to Instance 3.

Return traffic (not shown) follows the same path in the opposite direction.

Scenario 2 – Inspecting traffic between VPCs in different segments in the same Region

The diagram that follows (Figure 2) show scenario 2, where you might need to allow attachments in different segments to communicate with each other, but only through an inspection appliance. For example, when your development team must push an engineering change into production. As described in Concepts section, you can meet this requirement by sharing the segments with each other. Because you must inspect the traffic flowing between these two segments, instead of sharing the Production Segment and Development Segment with each other, we share the Inspection Segment 1 with the Production Segment and the Development Segment, respectively.

Prod VPC 1 and Dev VPC 1 are in the same Region 1 but mapped to different global segments – Production and Development segments, respectively. To allow communication between these attachments through firewall, we share Inspection segment 1 (1) with both these segments – Production and Development.

Figure 2: Inspecting traffic between VPCs across different segments in the same Region.

Packet walkthrough

The following steps describe a packet walkthrough when instance 1 in Prod VPC 1 communicates with instance 3 in Dev VPC 1 in the same Region 1:

• All the steps (A) thru (F) remain the same as mentioned in scenario 1, except for the return path. When the return packet (not shown) arrives at the core network, because Dev VPC 1 is associated with the Development Segment, it does a Development Segment Route Table (G) look-up instead of a Production Segment Route Table look-up (B).
• The packet matches the default entry with Inspection VPC A attachment as the target and the packet get routed to Inspection VPC A.
• Once the packet gets inspected, it gets routed back to the core network, where it does Inspection Segment 1 (shared) Route Table lookup. The packet matches the Prod VPC 1 CIDR entry with Prod VPC 1 attachment as the target and the packet gets routed to instance 1 in Prod VPC 1.

Scenario 3 – Inspecting traffic between VPCs in the same segment across different Regions. (Isolated attachments)

Scenario 3 is similar to scenario 1, except that the Prod VPCs are in different Regions. Prod VPC 1 and Prod VPC 3 are in the different Regions 1 and 2 respectively and mapped to the same global Production Segment.

Apart from the need to inspect traffic between attachments in the same segment, you might also need to treat each AWS Region as a separate protected environment. To achieve this, we inspect traffic in each local Region as shown in the following diagram (Figure 3).

Note: Since we share the inspection segment with the production segment, Prod VPC 1 and Prod VPC 3 CIDR route entries get propagated in to the inspection segment’s route table (for segment route table view refer to #8 under Considerations section). With one global inspection segment, today, we don’t have a way to force traffic to route through firewalls in a respective Regions. For our scenario, in order to meet the requirement of inspecting the traffic in respective Regions and to maintain symmetric routing, we create two global shared inspection segments. Separate inspection segments allow to create a static route for Region specific CIDRs pointing to Region specific inspection VPCs in the respective inspection segment route tables. In a core network, since static route takes the precedence over propagated routes, the static routes for Region specific CIDRs ensures traffic gets routed through firewalls in respective Regions and avoids asymmetric routing. For more details, refer to Route evaluation section in the documentation.

Figure 3: Inspecting traffic between VPCs in the same segment across different Regions

Packet walkthrough

The following steps describe a packet walkthrough when instance 1 in Prod VPC 1, Region 1 communicates with instance 3 in Prod VPC 3, Region 2:

• (A) When Instance 1 in Prod VPC 1 starts a connection to Instance 3 in Prod VPC 3, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and the packet gets routed to the Core Network.
• (B) When the packet arrives at the core network, because Prod VPC 1 is associated with the Production Segment, it does a Production Segment Route Table lookup. The packet matches the default entry with Inspection VPC A attachment as the target and the packet gets routed to Inspection VPC A. There are two next hops, and it prefers the one local to the Region.
• (C) When the packet arrives at the Inspection VPC A attachment, it does a VPC (CWAN Subnet) route table lookup. The packet matches the default route with Firewall Endpoint 1 as the target and the packet gets routed to the firewall, through the firewall’s endpoint, for inspection.
• (D) After the inspection, the firewall allows the traffic through because of the security policy configured. The firewall routes the packet back to the firewall endpoint where it does a VPC (Firewall Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and the packet gets routed to the Core Network.
• (E) When the packet arrives at the core network, because Inspection VPC A is associated with the Shared Inspection Segment 1, it does a Shared Inspection Segment 1 Route Table lookup. The packet matches the Prod VPC 3 CIDR entry with Inspection VPC B Attachment as the target and the packet gets routed to Inspection VPC B. The core network routes the traffic across the Region to Inspection VPC B.
• (F) When the packet arrives at the Inspection VPC B, it does a VPC (CWAN Subnet) route table lookup. The packet matches the default route with Firewall Endpoint 1 as the target and the packet gets routed to the firewall, through the firewall’s endpoint, for inspection.
• (G) After inspection, the firewall allows the traffic based on the security policy configured. The firewall routes the packet back to the firewall’s endpoint, where it does a VPC (Firewall Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and the packet gets routed to the Core Network.
• (H) When the packet arrives at the core network, because Inspection VPC B is associated with Shared Inspection Segment 2, it does a Shared Inspection Segment 2 Route Table lookup. The packet matches the Prod VPC 3 CIDR entry with Prod VPC 3 Attachment as the target and the packet gets routed to Prod VPC 2.
• (I) When the packet arrives at Prod VPC 3, it does a VPC (CWAN Subnet) route table lookup. Packet matches the VPC CIDR entry with local as the target and the packet gets routed to Instance 3.

Return traffic (not shown) follows the same path in the opposite direction.

Scenario 4 – Inspecting traffic between VPCs in different segments across different Regions

Scenario 4 is similar to scenario 2, except the Prod VPC 1 and Dev VPC 3 are in different Regions 1 and 2 and mapped to different global segments – a Production Segment and a Development Segment, respectively.

Similar to scenario 3, since each Region is treated as a separate protected environment, we inspect traffic in each local Region as shown in the following diagram (Figure 4).

Figure 4: Inspecting traffic between VPCs across different segments across different Regions

Packet walkthrough

The following steps describe a packet walkthrough when instance 1 in Prod VPC 1, Region 1 communicates with instance 3 in Dev VPC 3, Region 2:

• All the steps (A) thru (I) remain the same as mentioned in scenario 3, except for the return path. When the return packet arrives at the core network, because Dev VPC 3 is associated with Development Segment, it does a Development Segment Route Table look-up (J) versus Product Segment Route Table look-up (B). Packet matches default entry with Inspection VPC B attachment as the target and the packet gets routed to Inspection VPC B.
• Once the packet gets inspected by Inspection VPC B, it gets routed back to the core network, where it does Inspection Segment 2 (shared) Route Table lookup. The packet matches the Prod VPC 1 CIDR entry with Inspection VPC A attachment as the target and the packet gets routed to Inspection VPC A in Region 1.
• Once the packet gets inspected by Inspection VPC A, it gets routed back to the core network, where it does Inspection Segment 1 (shared) Route Table lookup. The packet matches the Prod VPC 1 CIDR entry with Prod VPC 1 attachment as the target and the packet gets routed to instance 1 in Prod VPC 1.

Considerations

1. Core Network Edge (CNE) inherits many of AWS Transit Gateway‘s characteristics, including throughput per VPC attachment. Refer to quotas section in the Cloud WAN documentation for more details.
2. Cloud WAN supports both IPv4 and IPv6.
3. As a best practice, use a separate small CIDR, for example /28, for each core network attachment so that you have more addresses available to use with other VPC resources. Keep the inbound and outbound network ACLs associated with the core network attachment subnets open.
4. Cloud WAN Appliance Mode is only supported for VPC attachments, in all AWS Regions where Cloud WAN is available.
5. To configure routing on Cloud WAN network segments, you must use core network policies (not covered in this post). To learn more about how network policies work, refer to the Create a core network policy version section in the documentation.
6. When inspecting traffic across the Regions, there are two options:
   • Consider each Region as the protected environment, thus inspecting everything coming in and out of each Region. We show this in Figure 3 and Figure 4. Be mindful of the Cloud WAN data processing cost, as traffic traverses the core network to access the firewalls in each Region.
   • Decide to inspect the traffic only in one Region. While this approach helps reduce the data processing cost, it also introduces a Regional dependency. Note: Avoid inter-Region dependencies.
7. You will have to create a static route in Cloud WAN’s Inspection Segment route table for each new VPC that gets added. You can use Infrastructure as Code (IaC) to automate the setup. Refer to Route evaluation section in the documentation to understand how Cloud WAN evaluates the routes.
8. Core network routes – You can search for and view core network routes for specific segment and edge location. The segment route table depicted in the preceding diagrams is for visualization purpose only. In reality, the Destination and Route Type visualization differ based on the edge location selected for that segment.
9. Although the CNE is represented by a single endpoint per subnet/Availability Zone (AZ) in our diagrams, it is highly available and based on AWS Hyperplane.
10. Security group (SG) referencing can’t be used to allow the source to connect to your destination. You can use IP addresses of your source in the SGs to permit the incoming traffic.
11. Inspection architectures shown in this post equally apply to security appliances deployed behind Gateway Load Balancer (GWLB) or AWS Network Firewall. For simplicity, the diagrams in the post only show the endpoints.

Conclusion

In this blog post, we discussed security inspection architecture patterns for East-West (VPC-to-VPC) traffic within and across Regions using native Cloud WAN appliance mode. Cloud WAN makes it easy to build and operate wide area networks that connect your data centers and branch offices, as well as your Amazon (VPCs). With Cloud WAN, you connect to AWS through your choice of local network providers, then use a central dashboard and network policies to create a unified network that connects your locations and network types.

For more information about AWS Cloud WAN, refer to the following resources:

• Cloud WAN documentation.
• AWS Cloud WAN and AWS Transit Gateway migration and interoperability patterns blog post.