Expanding VMware Cloud on AWS Multi-Region Connectivity Using AWS Cloud WAN

By Sheng Chen, Sr. Specialist Solutions Architect – AWS

As customers continually adopt VMware Cloud on AWS, it’s important to provide a scalable global network to seamlessly support their business demand.

VMware Cloud on AWS customers have requirements to interconnect their software-defined data centers (SDDCs) across different AWS regions, using Amazon Web Services (AWS) networking services and constructs.

However, the existing network design patterns require you to build a full-mesh topology with static routing to provide multi-region connectivity for your SDDCs. This leads to engineering challenges with increased network configuration complexity and management overhead.

For enterprise customers deploying SDDCs in three or more regions, it’s imperative to address additional requirements such as centralized network management and monitoring, dynamic routing, traffic segmentation, and network automation.

In this post, I will look at how AWS Cloud WAN can help address your VMware Cloud on AWS multi-region connectivity challenges and requirements. I’ll walk through a reference architecture with a real example for integrating VMware Cloud on AWS with AWS Cloud WAN.

AWS Cloud WAN Overview

In July 2022, AWS announced the general availability of AWS Cloud WAN, a managed wide-area networking (WAN) service. It provides a central dashboard for customers to build hybrid connectivity across on-premises branch offices, data centers, and Amazon Virtual Private Clouds (Amazon VPCs) over the AWS global network.

With AWS Cloud WAN, you can use simple network policies to centrally configure and automate network management, and build a global network in minutes.

Let’s review the core components of AWS Cloud WAN as illustrated in the diagram below:

• AWS Network Manager: The user interface (UI) in the AWS Management Console and associated APIs to centrally manage your global network.
• Global network: A single private network that acts as the root-level container for your network objects. A global network can contain both transit gateways and a core network.
• Core network: The part of your global network managed by AWS.
• Core network policy: A single, versioned policy document which defines all aspects of your core network.
• Attachments: Any connections or resources you want to add to your core network. Supported attachment types include VPCs, virtual private networks (VPNs), Connect (SD-WAN/GRE) and Transit Gateway route table attachments.
• Core Network Edge (CNE): A regional connection point for your attachments as defined in the policy.
• Network segments: Routing domains that by default only allow communication within a segment, consistently throughout the global network.

Figure 1 – AWS Cloud WAN components.

AWS Cloud WAN Architecture Advantages

VMware Cloud on AWS customers can utilize VMware Transit Connect (vTGW) with inter-region peering capability to expand to multiple regions, as discussed in a previous AWS blog.

Comparing to VMware Transit Connect with inter-region peering, AWS Cloud WAN provides the following advantages:

• Scale beyond three regions: Currently, VMware Transit Connect (SDDC group) is limited to a maximum of three regions, and it supports up to three AWS Transit Gateways (TGWs) in different regions. These limits can be bypassed by connecting to AWS Cloud WAN.
• Network segmentation: AWS Cloud WAN supports end-to-end, cross-region network segmentation, which is not currently available over the Transit Gateway inter-region peering.
• Dynamic routing: Dynamic routing is available in AWS Cloud WAN, and routes are automatically propagated over the Cloud WAN attachments.
• Built-in automation for attachments: With AWS Cloud WAN, you can use tags to automatically map attachments to segments through core network policy.
• Remove Transit Gateway full-mesh requirements: AWS Cloud WAN automatically builds full-mesh topology across all CNEs in different regions. This removes the management overhead for building cross-region full-mesh connectivity between Transit Gateways.
• Centralized global network management and monitoring: AWS Cloud WAN uses the AWS Network Manager console to provide a complete view of your global networks and help you monitor network health, security, and performance.

Integration with VMware Cloud on AWS

There are currently two options to connect VMware Cloud on AWS SDDCs into AWS Cloud WAN.

For the production environment or large-scale SDDC deployment, you can leverage VMware Transit Connect and AWS Transit Gateway to connect SDDCs to the Cloud WAN core network via Transit Gateway route table attachment. This option provides high-bandwidth connectivity but requires static routing over the intra-region peering link.

For the test/dev environment or smaller SDDCs, you could connect your SDDCs directly to Cloud WAN CNEs via site-to-site VPN attachments. This option provides end-to-end dynamic routing and is suitable for customers with traffic encryption requirements. However, with this option each SDDC is limited to a total maximum VPN throughput of 4~5 Gbps due to the per VPN tunnel bandwidth limit of 1.25 Gbps.

It’s important to note that in both cases, all cross-region traffic over Cloud WAN stays on the AWS secure backbone and is transparently encrypted at the physical layer.

The diagram below illustrates a reference architecture for integrating VMware Cloud on AWS with AWS Cloud WAN, covering both connectivity options.

Figure 2 – Integrating AWS Cloud WAN with VMware Cloud on AWS.

This reference architecture can be used as the basis for a proof of concept (PoC) by following the highlighted steps:

1. Create an AWS Cloud WAN core network with multiple network segments across all regions where customers’ SDDCs are deployed.
2. Within the Cloud WAN core network, a CNE is automatically provisioned in each associated region.
3. Deploy an AWS Transit Gateway with multiple route tables following the same segmentation strategy.
4. For each environment (such as prod/dev), create a dedicated SDDC group to deploy VMware Transit Connect (vTGW) which interconnects SDDCs with other networking resources.
5. Build an intra-region peering from each SDDC group to a customer-managed Transit Gateway. Associate each peering attachment to its corresponding TGW route table and configure relevant static routes.
6. Configure a TGW peering connection to the local CNE in the AWS Cloud WAN core network.
7. Create separate TGW route table attachments mapped to their respective Cloud WAN segments.
8. Configure a route-based site-to-site VPN connection from the NSX edge in an SDDC to the local CNE, and then create a VPN attachment to the corresponding Cloud WAN segment.

For more details, please refer to the full reference architecture.

Setup

In the following example, we’ll deploy an AWS Cloud WAN core network to provide cross-region connectivity for two SDDCs running in the Sydney and Singapore regions.

We’ll leverage VMware Transit Connect and AWS Transit Gateway to connect SDDC-01 to the Cloud WAN core network via Transit Gateway route table attachment. On the other hand, SDDC-02 will be connected to the same core network via a site-to-site VPN attachment. Both attachments will be placed into a core network segment “development” for end-to-end traffic segmentation.

Figure 3 – Demo setup overview.

As illustrated in the above table, I have provisioned and prepared the following items as prerequisites to this lab:

• 2x SDDCs (SDDC-01 in Sydney and SDDC-02 in Singapore).
• An AWS Transit Gateway is created in the Sydney region with two separate route tables (prod/dev) for traffic segmentation.
• For SDDC-01, a SDDC group (VMware Transit Connect) is deployed with intra-region peering established to the above Transit Gateway. Refer to this VMware blog post for more details on how to setup VMware Transit Connect intra-region peering.

For this example, I’ll walk through the configurations covering the following aspects:

• Creating a Cloud WAN core network.
• Setting up Cloud WAN Transit Gateway route table attachment for SDDC-01.
• Setting up Cloud WAN VPN attachment for SDDC-02.
• Verification.

Part 1: Create a Cloud WAN Core Network

To set up an AWS Cloud WAN core network, go to AWS Network Manager within the AWS console and perform the following tasks:

• Under Connectivity, navigate to Global Networks, and on the right section click Create global network.
• Provide a Name and Description for the global network, and click Next.
• You will now create a Cloud WAN core network. Provide a name and description for the core network and make sure to select “Add core network in your global network.”
• On the same page, provide basic settings for the initial core network policy. Select the required edge locations and supply a unique autonomous system number (ASN) range for CNE deployment. Here, we are also adding a default network segment called development, as shown in Figure 4 below.
• On the next page, review all configurations for the global network and core network. When finished, click Create global network.

Figure 4 – Deploy a Cloud WAN core network.

At this stage, a Cloud WAN core network is being deployed with an initial core network policy. However, we’ll want to make a few modifications to enable attachment automation, so that the attachments for both SDDCs are automatically mapped into the development segment and routes are automatically propagated.

To do so, click into the Global network you just created, navigate to Core network and then Policy versions. Select Policy version – 1 and click Edit. Add the following changes to the policy:

• Under the Segments tab, add a production segment across both edge locations.
• Also under the Segments tab, select the development segment and disable/uncheck “Require attachment acceptance.”
• Under the Attachment policies tab, add a new policy to automatically map any attachment with a “development” tag to the development segment.

Figure 5 – Update core network policy to enable attachment automation.

Once you’ve made above changes, click Create policy and you’ll see a new policy version in the list showing “Ready to execute.” Select the new policy, and click View or apply change set to apply the new policy.

Part 2: Set Up Cloud WAN TGW Route Table Attachment for SDDC-01

Before we create our first AWS Cloud WAN attachment, let’s review the current route table of VMware Transit Connect at SDDC-01.

Within the VMware Cloud on AWS SDDC console, navigate to the SDDC group for SDDC-01 and go to the Routing tab.

Figure 6 – VMware Transit Connect route table.

As shown above, apart from the SDDC management routes, I’m using the Route Aggregation feature to summarize all SDDC-01 workload subnets into a single 10.101.0.0/16 prefix.

Similarly, I have added a summarized route of 10.102.0.0/16 towards the AWS Transit Gateway, as I know all workload segments from SDDC-02 are within this /16 range. You could further optimize the Transit Connect route table by pre-configuring all RFC1918 routes (or a default route) if the peered TGW is the only egress point within your environment.

In addition, you’ll need to ensure the intra-region peering from VMware Transit Connect is associated with the TGW development route table, since we will soon attach this route table to the respective Cloud WAN segment.

Figure 7 – Associating SDDC intra-region peering to TGW route table.

Now, let’s switch to the Cloud WAN configuration. Within the AWS Network Manager, click into your Global Network and go to Core network > Peerings. We’ll first create a Cloud WAN peering connection from the AWS Transit Gateway to the local CNE within the Sydney region.

Figure 8 – Create Transit Gateway peering to Cloud WAN.

Next, navigate to Core network > Attachments, we will create a TGW route table attachment based on the previous peering connection, and make sure to select the Transit Gateway development route table.

Note that I’m also adding a tag as a trigger condition for attachment automation.

Figure 9 – Create TGW route table attachment.

While the TGW route table attachment is getting created, its tag is recognized by the previously configured policy and the attachment is automatically associated with the development segment.

Figure 10 – Attachment is automatically associated with Cloud WAN segment.

Part 3: Set Up Cloud WAN VPN Attachment for SDDC-02

Next, we will create a VPN attachment from SDDC-02 to the Cloud WAN CNE within the Singapore region.

Follow this documentation to create a VPN connection. Make sure to select Not associated under the Target gateway type, as we’ll be attaching the VPN connection directly to the local CNE.

Now, switch to AWS Network Manager, locate your Global Network and go to Core network > Attachments. Create a VPN attachment as shown below. Again, we are using a tag as a condition trigger to map the SDDC-02 VPN attachment into the development segment.

Figure 11 – Create VPN attachment.

Finally, follow this VMware guide to create corresponding route-based VPN connections at SDDC-02.

Part 4: Verification

By now, you should have two Cloud WAN attachments established, and both should be automatically placed into the development segment.

To verify routes on the SDDC-01 side, navigate to the TGW development route table. You should see all SDDC-02 routes are automatically propagated from Cloud WAN core network. At the SDDC-02 VPN route table, you can also see the two routes for SDDC-01 learned from Cloud WAN via BGP.

Figure 12 – Verify route table updates.

Back at the Core network, under the Topology tree you can easily get an overview of the global network topology, including all CNEs, segments, and core network attachments.

Notice there are two VPN connections (from SDDC-02) under the VPN attachment, since by default Cloud WAN supports Equal-Cost Multi-Path (ECMP) over VPN.

Figure 13 – Cloud WAN core network topology tree.

Additional Considerations

For this walkthrough, I have only used the AWS console but you can also use the AWS Command Line Interface (AWS CLI) or AWS SDK for Cloud WAN configurations. In addition, core network policy is expressed as a JSON template and can be modified using the AWS Console JSON editor, graphical user interface (GUI), or supplied with an API call.

AWS Cloud WAN uses eBGP over peering as well as communications between CNEs within the core network. Therefore, ensure your CNE ASNs are unique and not in use by any existing TGWs or SDDCs.

Cloud WAN integration with existing customer VPCs and AWS Direct Connect connections were outside the scope of this post. Refer to this AWS blog post for more details on these topics.

Cloud WAN pricing details are available on the product page. Make sure to only enable CNEs in AWS regions required, as each CNE has an associated hourly cost. Also, it’s important to note there is no data processing charges on traffic sent over peered connections between a CNE and AWS Transit Gateway.

Summary

In this post, we took a closer look on how AWS Cloud WAN can help VMware Cloud on AWS customers connect SDDCs across multiple AWS regions at scale.

I discussed a reference architecture of integrating AWS Cloud WAN with VMware Cloud on AWS, and went through a step-by-step example based on this architecture.

To learn more, we recommend you to review these additional resources:

• VMware Cloud on AWS hybrid network design patterns
• Introducing AWS Cloud WAN
• AWS Cloud WAN and AWS Transit Gateway migration and interoperability patterns