Achieve optimal routing with AWS Cloud WAN for multi-Region networks

In multi-Region AWS Cloud WAN deployments, customers deploy centralized architectures for use cases such as traffic inspection, hybrid connectivity (using AWS Transit Gateways and AWS Direct Connect), and internet egress. For internet egress, you may use a single egress VPC per AWS Region or per geography (collection of AWS Regions), for example, Americas (AMER GEO), Europe (EMEA GEO), where you deploy firewalls and NAT Gateways for egress traffic. Since a Cloud WAN segment can be configured as a global or a Regional object, designing your routing so that every AWS Region has a Regional attachment for a particular destination CIDR gives you control over the exit point for egress traffic. This is because, if multiple routes within a segment are available to the same destination, including a local Region route, then the default behavior is to always prefer the local Region’s route over a remote Region’s route as long as the BGP AS-Path length of learned prefixes are the same. Regions that do not have attachments in this destination list receive a propagated version of this route through another Region’s Core Network Edges (CNEs, Regional connection endpoints managed by AWS). At the time of writing this blog, if multiple attachments with similar attributes are received from multiple CNEs, then the attachment that forwards the traffic is chosen at random. This may result resulting in less than desirable routing behavior.

In this post, we take a closer look at how to achieve optimal routing in multi-Region

architectures with Cloud WAN. We focus on the following two scenarios:

1. Multi-Region centralized internet egress with inspection
2. Multi-Region hybrid connectivity using Direct Connect

AWS Cloud WAN concepts

Before we look at these scenarios, it is important to understand some key Cloud WAN concepts:

• Core Network Edge (CNE) – All CNEs in the core network create full-mesh peering with each other to form a highly resilient network. They use dynamic routing (BGP) to exchange routing information. For hybrid connectivity through Direct Connect, Cloud WAN always prefers the Transit Gateway in the same Region (if it exists) as long as the BGP AS-Path lengths of learned prefixes are the same when deciding where to send the packet destined for on-premises resources. Refer to the Route evaluation section in the Cloud WAN documentation for more on this topic.
• Network segments are dedicated routing domains that play an important role in how traffic is routed through your core network and between Cloud WAN attachments. Therefore, by default, only the attachments within the same segment can communicate. Using network segments, you can divide your global network into separate isolated networks.

Refer to the Cloud WAN documentation to learn more about these concepts. If you are just getting started with Cloud WAN, the Introducing Cloud WAN blog post provides a great overview of these concepts in action.

Example architectures used in this post

In both scenarios one and two, shown in the subsequent figures, there are four Regions (A, B, C, and D). Prod VPCs A, B, C, and D are deployed in each respective Region:

• Prod VPC A in Region A: 10.1.0.0/16
• Prod VPC B in Region B: 10.2.0.0/16
• Prod VPC C in Region C: 10.3.0.0/16
• Prod VPC D in Region D: 10.4.0.0/16

In the diagrams that follow, we have labeled each Prod VPC with (E). In the respective Region, these VPCs advertise their local VPC CIDR to Cloud WAN CNEs (CNE A, B, C, and D) using their respective VPC attachment.

Similarly, in both scenarios one and two, the four Regions are virtually grouped into two GEOs: GEO 1 and GEO 2. Regions A and B are grouped under GEO 1 and Regions C and D are grouped under GEO 2.

Scenario 1: Achieve optimal routing in multi-Region centralized internet egress with inspection

Let’s look at the setup in this scenario as shown in the following diagram (figure 1).

Figure 1: Multi-Region Cloud WAN setup with centralized internet egress and inspection.[Click the image to see a larger version in a new tab.] 

Label (F): Regions B and C have Egress VPCs B and C respectively:

• Egress VPC B in Region B: 100.65.0.0/16
• Egress VPC C in Region C: 100.66.0.0/16

Each Egress VPC has three subnets:

• CWAN Subnet – used for attaching the Egress VPC to the Cloud WAN Core Network
• Firewall (FW) Subnet – has a Firewall endpoint deployed to help with traffic filtering/inspection
• Public Subnet – has a NAT Gateway deployed to enable outbound internet connectivity from the VPC

Each Egress VPC also has an Internet Gateway (IGW) attached. Optionally, you can just have centralized Egress VPCs without inspection similar to the architecture as shown in the preceding figure.

Label (G): Cloud WAN has two global segments that span all four Regions. The Prod VPCs are associated with the Production Segment. All Prod VPC Route tables have a default route pointing to the Cloud WAN Amazon Resource Name (ARN) as the next hop destination. The Production Segment route table has propagated the routes of all Prod VPCs, which enables intra and inter-Region connectivity between Prod VPCs. It also has statically added default routes pointing to the Egress VPCs B and C as the next hop destination.

Label (H): The Egress VPCs are associated with the Egress Segment. Due to segment-sharing from Production Segment into the Egress Segment (represented with label (O) in the preceding figure), the Egress Segment route table has propagated the routes of all the Prod VPCs.

In this use case, in GEO 1, internet traffic originating from Prod VPCs A and B should traverse the Cloud WAN network and exit out of Region B after inspection/filtering at the firewall in Egress VPC B. Similarly, in GEO 2, internet traffic originating from Prod VPCs C and D should exit out of Region C after inspection/filtering at the firewall in Egress VPC C. This is done to optimize the cost of deployment of firewall endpoints and NAT Gateways for internet traffic from multiple Regions within a GEO.

Scenario 1: Packet walkthrough

The following steps describe a packet walkthrough of traffic initiated from Prod VPC A toward an internet destination (see the following diagram, figure 2). In this scenario, we only show packet flow using VPC A, which does not have a local Egress VPC in the same Region, but the same steps apply for VPC D.

Figure 2: Traffic flows in multi-Region Cloud WAN setup with centralized internet egress and inspection. [Click the image to see a larger version in a new tab.]

Label (1): When instance 2 in Prod VPC A starts a connection to a destination on the internet, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and gets routed to the Core Network. Since the Prod VPC A is associated with the Production Segment, a Production Segment Route Table lookup is performed for the packet arriving at the Core Network. The packet matches the default route entry with two next hops: Egress VPC B attachment (GEO 1, Region B) and Egress VPC C attachment (GEO 2, Region C). Since Region A does not have local Egress VPC, one of the two available next hops is chosen at random by Cloud WAN. In this example, the route received by CNE A from CNE C (through full-mesh peering) is chosen and the packet gets routed through CNE C to the Egress VPC C. This behavior stays consistent for all internet egress traffic flowing out of Region A as shown in the preceding figure, until you modify the core network policy to remove CNE C from the Core Network.

Label (2): Next, when the packet arrives at the Egress VPC C attachment, it does a VPC (CWAN Subnet) route table lookup. The packet matches the default route with Firewall Endpoint 1 as the target and gets routed to a firewall for inspection through the firewall’s endpoint. The firewall inspects the traffic and, if allowed, routes it back to the firewall’s endpoint, where it does a VPC (FW Subnet) route table lookup. The packet matches the default route entry with the NAT Gateway as the target and gets routed to the NAT Gateway.

Label (3): At the NAT Gateway, the traffic is source NATed to its Elastic IP Address (EIP) and sent out toward the internet destination through the VPC’s IGW.

Labels (4), (5), (6): Return traffic (shown with Red dotted lines in the preceding figure) follows the same path in the opposite direction.

The optimal behavior is for traffic from Regions A and B to always exit out of Egress VPC B (in the same GEO 1) and traffic from Regions C and D to always exit out of Egress VPC C (in the same GEO 2). However, as seen in the preceding packet walkthrough, when a local Egress VPC and corresponding default route are not available for a particular Region, the routing behavior of Cloud WAN could result in the traffic exiting out of a remote Region’s Egress VPC chosen at random. This remote Region may belong to a completely different GEO compared to the Prod VPC’s GEO where the traffic originated. This may not be the routing behavior you desire.

Scenario 1: Controlling which Region is used for internet egress

The recommended way to control the Region that Cloud WAN uses for internet egress is by creating an Egress VPC in every Region where you have workloads running. However, there may be reasons where you may decide not to have an Egress VPC in every Region and still would want to have control over the Cloud WAN routing.

Let’s now look at the architecture pattern where you can control the routing behavior and achieve optimal routing of internet egress traffic with Cloud WAN. Before we dive into the solution details, let’s first look at the setup in this example. We show this in the following diagram (figure 3).

Figure 3: Multi-Region Cloud WAN setup with centralized internet egress and inspection using multiple GEOs.[Click the image to see a larger version in a new tab.] 

The key difference in the preceding architecture is with the Cloud WAN segmentation for Prod VPCs. In figures 1 and 2, we had single Production and Egress Segments spread across all Regions in the Core Network. In this new architecture, we have split the Production Segment into multiple GEO-based segments:

• Production Segment 1 (labeled (G) in figure 3) now spans GEO 1
• Production Segment 2 (labeled (I) in figure 3) now spans GEO 2

Production Segment 1 and Production Segment 2 are associated with Production Segment 1 Route Table and Production Segment 2 Route Table, respectively. There are no changes to the Egress Segment (label (H) in figure 3), which remains global. Production Segment 1 and Production Segment 2 share their routes with the Egress Segment to enable connectivity between the Prod VPCs and the internet (label (O) in figure 3). Optionally, the Production Segment 1 and Production Segment 2 can be shared with each other to enable inter-GEO connectivity between the Prod VPCs (not shown in figure 3). Finally, a static default route is added per GEO to the Production Segment route tables, pointing to the Egress VPC attachment of that GEO as the next-hop.

Scenario 1: Packet walkthrough

The following steps describe a packet walkthrough of traffic initiated from Prod VPC A toward an internet destination (see the following diagram, figure 4). In this scenario, we only show packet walkthrough using VPC A, which does not have a local Egress VPC in the same Region, but the same steps apply for VPC D as well.

Figure 4: Traffic flows in multi-Region Cloud WAN setup with centralized internet egress and inspection using multiple GEOs.[Click the image to see a larger version in a new tab.] 

Label (1): When instance 2 in Prod VPC A starts a connection to a destination on the internet, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and gets routed to the Core Network. Since the Prod VPC A is associated with the Production Segment 1, a Production Segment 1 Route Table lookup is performed for the packet arriving at the Core Network. The packet matches the default route entry with a single next hop: Egress VPC B attachment (GEO 1, Region B) and it gets routed to Egress VPC B.

Label (2): Next, when the packet arrives at the Egress VPC B attachment, it does a VPC (CWAN Subnet) route table lookup. The packet matches the default route with Firewall Endpoint 1 as the target and gets routed to a firewall for inspection through the firewall’s endpoint. The firewall inspects the traffic and, if allowed, routes it to the firewall’s endpoint, where it does a VPC (FW Subnet) route table lookup. The packet matches the default route entry with the NAT Gateway as the target and gets routed to the NAT Gateway.

Label (3): At the NAT Gateway, the traffic is source NATed to its EIP and sent out toward the internet destination through the VPC’s IGW.

Labels (4), (5), (6): Return traffic (shown with red dotted lines in the figure 4) follows the same path in the opposite direction.

This solution makes sure that the internet traffic from any Region within a particular GEO always uses the GEO’s local Egress VPC as an internet exit point and for any security inspection/filtering requirements. This gives you optimal routing through Cloud WAN.

Scenario 2: Achieve optimal routing in multi-Region hybrid connectivity using Direct Connect

Before we dive into the details, let’s first look at the setup in the scenario shown in the following diagram (figure 5).

Figure 5: Multi-Region Cloud WAN setup with hybrid connectivity through Transit Gateway and Direct Connect.[Click the image to see a larger version in a new tab.] 

Label (F): Cloud WAN has two global segments that span all four Regions. All Prod VPCs are associated with the Production Segment, and all Prod VPC Route tables point to Cloud WAN as the next hop destination.

Label (G): Corp Data Center B is connected to Direct Connect Location B and Corp Data Center C is connected to Direct Connect Location C. Both are advertising a default route (0.0.0.0/0) to the Direct Connect Gateway (DXGW). No BGP traffic engineering is applied to either Direct Connect connection. We covered BGP traffic engineering scenarios with Cloud WAN and hybrid connectivity through Direct Connect in a previous blog post, Hybrid security inspection architectures with AWS Cloud WAN and AWS Direct Connect.

Label (H): Transit Gateways B and C are peered with respective Cloud WAN CNEs (CNE B and C) in their respective Regions. Both Transit Gateways B and C have their route table attachments associated with the Hybrid Segment. Regions A and D do not have a local Transit Gateway.

Label (I): Production and Hybrid Segments are shared with each other through segment-sharing. As a result, both segments learn all routes from the other segment.

Label (J): Since the Production segment is shared with the Hybrid segment that Transit Gateways B and C are attached to, both Transit Gateways learn all Prod VPC CIDRs using their respective local Region Cloud WAN peering/Route Table-to-Cloud WAN segment attachment.

Label (K): Both Transit Gateways B and C learn the on-premises prefix 0.0.0.0/0 through DXGW over BGP.

Label (L): DXGW Allowed Prefix List for Transit Gateway B during association is set up to include Prod VPC A (10.1.0.0/16) and Prod VPC B (10.2.0.0/16). Similarly, DXGW Allowed Prefix List for Transit Gateway C during association is set up to include Prod VPC C (10.3.0.0/16) and Prod VPC D (10.4.0.0/16).

Label (M): DXGW advertises CIDRs of Prod VPCs A, B, C, and D to on-premises resources through BGP.

In this use case, traffic from Prod VPCs A and B (GEO 1) that is destined to on-premises resources should traverse the Cloud WAN network and exit out of Transit Gateway B toward Corp Data Center B only. Similarly, traffic from Prod VPCs C and D (GEO 2) destined to on-premises resources should traverse the Cloud WAN network and exit out of Transit Gateway C toward Corp Data Center C only.

Scenario 2: Packet walkthrough

The following steps describe a packet walkthrough (see the following diagram, figure 6). In this scenario, we only show packet walkthrough using VPC D, which does not have a local Transit Gateway in the same Region, but the same steps apply for VPC A as well:

Figure 6 : Traffic flows in multi-Region Cloud WAN setup with hybrid connectivity through Transit Gateway and Direct Connect [Click the image to see a larger version in a new tab]

Label (1): When an instance in Prod VPC D starts a connection to an on-premises resource in Corp Data Center C, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and the packet gets routed to the Core Network.

Label (2): When the packet arrives at the core network, because Prod VPC D is associated with the Production Segment, it does a Production Segment Route Table lookup. The packet matches the default route entry with two next hops: Transit Gateway B (Region B) and Transit Gateway C (Region C). Since Region D does not have a local Transit Gateway, one of the two available next hops is chosen at random. Here in our example, Transit Gateway B is chosen and the packet is routed to Transit Gateway B.

Label (3): When the packet arrives at the Transit Gateway B, it does a Transit Gateway B Hybrid Route Table lookup. The packet matches the default route with DXGW as the target and it gets routed to the DXGW.

Label (4): When the packet arrives at the DXGW, there are two next hops: Direct Connect Location B and Direct Connect Location C. Direct Connect Location B is preferred because it is local to the Region (Associated Home Region).

Labels (5), (6), (7), (8): Return traffic (shown in Red dotted lines in the preceding figure) traverses TGW C via DXGW. This is due to DXGW Allowed Prefix List configured for Transit Gateway C that includes Prod VPC D CIDR. Note that in the current scenario, since we have not considered a stateful appliance in network path to inspect traffic, the return traffic via a different TGW (Transit Gateway C) would not be affected due to asymmetric routing. However, with a stateful firewall appliance, asymmetric routing path could cause traffic to be dropped.

The optimal behavior is for traffic from Regions A and B to always exit out of Transit Gateway B toward Corp Data Center B and traffic from Regions C and D to always exit out of Transit Gateway C toward Corp Data Center C. However, as seen in the preceding packet walkthrough, when a local Transit Gateway and corresponding default route are not available for a particular Region, the routing behavior of Cloud WAN results in the traffic exiting out of a remote Region’s Transit Gateway chosen at random. Therefore, it may result in sub-optimal routing. In this example, the route received by CNE D from CNE B (through full-mesh peering) is chosen and the packet gets routed through CNE B to the Transit Gateway B. This behavior stays consistent for all on-premises traffic flowing out of Region D as shown in the preceding figure, until you modify the core network policy to remove CNE B from the Core Network. Note that in the example shown in the preceding figure, as long as the PATH lengths of the prefixes learned from on-premises resources are the same, any other BGP traffic engineering applied on Direct Connect has no effect on the Cloud WAN routing behavior.

Scenario 2: Controlling which Transit Gateway is used for egress to on-premises

The recommended way to control the aforementioned Cloud WAN behavior is to have a Transit Gateway in every Region where you have workloads running and connected to on-premises through Direct Connect (as shown in the Hybrid security inspection architectures with Cloud WAN and AWS Direct Connect blog post). However, there may be reasons where you decide not to have a Transit Gateway in every Region and still want control over the Cloud WAN routing. Let’s now look at architecture pattern where you can achieve optimal routing with Cloud WAN.

Before we dive into the details, let’s first look at the setup in the following diagram, figure 7.

Figure 7: Multi-Region Cloud WAN setup with hybrid connectivity through Transit Gateway and Direct Connect using multiple GEOs. [Click the image to see a larger version in a new tab.] 

The key difference in this architecture (shown in figure 7) is with the Cloud WAN segmentation for Prod VPCs and Transit Gateways. In figures 5 and 6, we had single Production and Hybrid Segments that spread across all the Regions of the Core Network. In this new architecture, we have split the segments into multiple GEO-based segments: Production Segment 1 (label (F) in figure 7) and Hybrid Segment 1 (label (H) in figure 7) spans across GEO 1 (Regions A and B) and Production Segment 2 (label (N) in figure 7) and Hybrid Segment 2 (label (O) in figure 7) (spans across GEO 2 (Regions C and D).

Production Segment 1 and Production Segment 2 are associated with Production Segment 1 Route Table and Production Segment 2 Route Table, respectively. Similarly, Hybrid Segment 1 and Hybrid Segment 2 are associated with Hybrid Segment 1 Route Table and Hybrid Segment 2 Route Table, respectively. The Production Segment 1 and Production Segment 2 share their routes with the Hybrid Segment 1 and Hybrid Segment 2 respectively to enable connectivity between the Prod VPCs and Corp Data Centers (label (I) in figure 7). Optionally, the Production Segment 1 and Production Segment 2 can also be shared with each other (not shown in the preceding figure), to enable inter-GEO connectivity between the Prod VPCs. Note that in this scenario, we are splitting Hybrid Segment into multiple GEO-based segments since Transit Gateway peering attachments do not support static routing.

Scenario 2: Packet walkthrough

The following steps describe a packet walkthrough (as shown in the following figure). In this scenario, we only show a packet flow using VPC D, but the same steps apply for VPC A as well:

 Figure 8: Traffic flows in multi-Region Cloud WAN setup with hybrid connectivity through Transit Gateway and Direct Connect using multiple GEOs.[Click the image to see a larger version in a new tab.] 

Label (1): When an instance in Prod VPC D starts a connection to an on-premises resource, it does a VPC (App Subnet) route table lookup. The packet matches the default route entry with the Core Network ARN as the target and is routed to the Core Network.

Label (2): When the packet arrives at the core network, because Prod VPC D is associated with the Production Segment 2 in GEO 2, it does a Production Segment 2 Route Table lookup. The packet matches the default entry with a single next hop: Transit Gateway C attachment (GEO 2, Region C). The packet gets routed to Transit Gateway C.

Label (3): When the packet arrives at the Transit Gateway C, it does a Transit Gateway C Hybrid Route Table lookup. The packet matches the default route with DXGW as the target and gets routed to the DXGW.

Label (4): When the packet arrives at the DXGW, there are two next hops: Direct Connect Location B and Direct Connect Location C. Direct Connect Location C is preferred because it is local to the Region (Associated Home Region).

Labels (5), (6), (7), (8): Return traffic (shown in red dotted lines in the figure 8) follows the same path in the opposite direction.

With this approach, you can make sure that the traffic to Corp Data Centers from any Region within a particular GEO always uses the GEO’s local Transit Gateway as an exit point. This results in achieving optimal routing through Cloud WAN.

Considerations

• As a best practice
  • Make sure you have an Egress VPC in each Region where you have workloads and need centralized internet egress to avoid taking dependencies on remote Regions.
  • Use a separate small CIDR, for example /28, for each core network attachment so that you have more addresses available to use with other VPC resources. Keep the inbound and outbound network ACLs associated with the core network attachment subnets open.
• Make sure you have a Transit Gateway in every Region where you have hybrid connectivity through Direct Connect with Cloud WAN to avoid taking dependencies on any Region.
• The architectures shown in this post use Cloud WAN segment splitting to achieve the desired optimal routing behavior. There are quotas on the number of segments you can have per Core Network. Refer to quotas section in the Cloud WAN documentation for more details.
• Cloud WAN supports both IPv4 and IPv6.
• We always recommend creating redundant Direct Connect connections from at least two separate colocation facilities for high resiliency. Refer to AWS Direct Connect Resiliency Recommendations for details.
• The egress architectures shown in this post equally apply to security appliances deployed behind Gateway Load Balancer (GWLB) or AWS Network Firewall. For simplicity, the diagrams in this post only show firewall endpoints.
• You do not need additional Transit Gateways or Cloud WAN CNEs per Region for high availability. This is because they are highly available by design built on AWS Hyperplane.
• Cloud WAN segment sharing is bi-directional by default. Sharing a segment with another segment will automatically propagate routes into both sides.

Conclusion

In this post, we discussed architecture patterns for achieving optimal routing with Cloud WAN for multi-Region networks. We covered two scenarios, centralized internet egress with inspection, and hybrid connectivity using Direct Connect.

Visit the AWS Cloud WAN, AWS Transit Gateway, and AWS Direct Connect documentation for additional information.

About the authors