AWS Partner Network (APN) Blog

Connectivity Options for VMware Cloud on AWS Software Defined Data Centers

By Haider Witwit, Senior Solutions Architect at AWS
By Aarthi Raju, Partner Solutions Architect at AWS

VMware Cloud on AWSVMware Cloud on AWS enables customers to have a hybrid cloud platform by running their VMware workloads in the cloud while having seamless connectivity to on-premises and Amazon Web Services (AWS) native services.

Customers can use their existing AWS Direct Connect or Virtual Private Network (VPN) solutions to connect to their VMware Software Defined Data Center (SDDC) clusters.

In this post, we dive deep into SDDC networking and how it connects to different local and remote customer networks. We also explore options for connecting from your SDDC to Amazon Virtual Private Cloud (Amazon VPC), on-premises networks, and multiple VPCs.

SDDC Networking

With the SDDC deployed on AWS bare metal hardware provided by Amazon Elastic Compute Cloud (Amazon EC2), the lowest network level to start with is the Amazon EC2 underlay network or the Infrastructure Subnet. This network hosts the vmkernal adapters of the ESXi hosts like vMotion, vSAN, and host management networks. VMware uses NSX to control access to this network as part of the SDDC management model, and limits access to only remote traffic required to support features like cross-cluster vMotion.


Figure 1 – System diagram view in the VMware Cloud on AWS portal.

On the top of the underlay, NSX builds an overlay networks for logical VMware connectivity. Each SDDC has two types of overlay networks:

  • Default VMware-managed Appliance Subnet for SDDC management components like vCenter and NSX managers. This network is created during cluster provisioning and cannot be changed by the customer. Access to this network is controlled by the NSX Management Gateway (MGW) through firewall rules and IPsec tunnels.
  • One or more customer-managed logical networks for VM traffic. Those can be either routed locally within the cluster or stretched from remote on-premises clusters with remote gateway for L3 routing. Access to this network is controlled by the NSX Compute Gateway (CGW) through firewall rules and IPsec capabilities to enable customers to connect securely to their remote workloads and the Internet.


Figure 2 – SDDC network topology.

Connectivity to Customer VPC

VMware Cloud on AWS enables customers to seamlessly integrate their SDDC cluster to AWS services. This is achieved by connecting the SDDC cluster to the customer’s VPC of choice. During the onboarding process, customers have the ability to choose a VPC and the subnets they want to connect to their SDDC cluster.

Customers will also run an AWS CloudFormation template, which grants VMware Cloud Management Services across account roles with a managed policy. This managed policy allows VMware to perform operations like creating Elastic Network Interfaces (ENIs) and route tables. A detailed description of what the policy allows can be seen in the AWS IAM Console under AWS Managed Policies. The policy’s Amazon Resource Name (ARN) is arn:aws:iam::aws:policy/AmazonVPCCrossAccountNetworkInterfaceOperations.

Once the role has been created and assigned, VMware Cloud Management Services assumes a role in the customer account and creates ENIs in the subnet the customer chooses. These ENIs are directly attached to the ESXi hosts in the VMware SDDC account.

Of all the attached ENIs, only one of them is in use. The Compute Gateway lives on a single ESXi host, and this decides the ENI that’s in active state. This ENI allows for connectivity between the SDDC cluster and customer VPC. In the event of a host failure, VMware vMotions the Compute Gateway to a new host and the customer route table is updated to point to the new active ENI. Customers will be able to see these ENIs in their account with a description set to ‘VMware VMC Interface.’

VMware also makes it easy to update customer route tables based on the logical networks that are created. The default route table has updated routes to all customer logical networks, and this paves the way for services running in the VPC to communicate to the logical networks. Customers need to have the right security group and Network Access Control List (ACL) rules to allow traffic from the logical network.

Customers can view details about the connected account from their VMware Cloud on AWS console. Connectivity between the SDDC and VPC will not incur any data egress costs if the ENI created for this service and SDDC are in the same Availability Zones (AZs). If they are in different AZs, you will incur standard data transfer charges.


Figure 3 – Connectivity model between SDDC and customer VPC.

Connectivity to Other Customer VPCs

There are two approaches to enable connectivity between SDDC and other customer-managed VPCs. Choosing the right approach depends on the number of VPCs and SDDC logical networks they need to communicate with.

The first approach is to create multiple tunnels from the NSX CGW to the different VPCs using the AWS-managed VPN. Because NSX edge only supports policy-based VPNs today, and the AWS-managed VPN is limited to a unique Security Association (SA) pair per tunnel, this model enables a single logical SDDC network to communicate over each VPN tunnel at the same time.


Figure 4 – Approach 1: Direct VPN tunnels from CGW to customer VPCs.

If multiple SDDC logical network communication are needed, a different solution using the AWS Transit VPC model can be used to address the caveat in the first approach. This model suggests the use of third-party software VPN on Amazon EC2 inside a shared/transit VPC. The software VPN works as a hub and spoke model to enable connectivity between SDDC logical networks and customers VPCs.


Figure 5 – Approach 2: Transit VPC model

Connecting to On-Premises

SDDC supports integration with AWS Direct Connect service to provide persistent, reliable, and low latency connection to customers’ on-premises environments. To support a variety of connection options for different use cases, the separate NSX edges for management and compute enable customers to have easy traffic segregation to remote sites through dedicated IPsec tunnels. These tunnels can be established over Internet or AWS Direct Connect Public Virtual Interface (VIF).

One of the most popular and core feature of VMware vSphere is vMotion—the ability to live migrate workloads from one physical host to another with no downtime. vMotion capability was later extended to support cross-cluster, cross-vCenter, and long-distance live migrations. These features pushed infrastructure capabilities to its limits to support and maintain reliable migration operations.

In this section, we will explain how to use AWS Direct Connect to meet network requirements for long distance vMotion between on-premises and VMware Cloud on AWS.

AWS Direct Connect uses virtual interfaces to enable access to AWS services. Private virtual interface is used to connect to resources inside Amazon VPC. Public virtual interface connects to public AWS services outside VPC boundaries, such as Amazon Simple Storage Service (Amazon S3), Amazon Glacier, and EC2 Elastic IP addresses (EIP).

AWS Direct Connect Service is a multi-account service. Connection owners can create and share virtual interfaces with different AWS accounts in a hosted virtual interface model. VMware Cloud on AWS supports the hosted virtual interface model. SDDC customers can create private virtual interfaces from AWS Direct Connect connections they own or request from AWS Partner Network (APN) Partners, and then attach to the VMware managed VPC where the SDDC resides.

Requirements for the private and public interfaces vary depends on use case. Private VIF enables connectivity to SDDC underlay or the infrastructure subnet, while Public VIF enables connectivity to the two Management Gateway (MGW) and CGW edge gateways since both use public IPs for external connectivity.

Network Requirements for Data Center Extension

Let’s take data center extension as one common use case and touch on how to meet networking requirements for vMotion with AWS Direct Connect today.

VMware has defined three network requirements for reliable virtual machine migrations between on-premises and SDDC:

  • L3 connectivity between source and destination vMotion and host management networks. This requires Private VIF between on-premises and the SDDC VPC. The Private VIF will enable L3 connectivity between the SDDC underlay and related on-premises networks.
  • L3 connectivity between source and destination vCenter Servers. Since the SDDC vCenter Server operates on the overlay network, it needs L3 VPN between the MGW and edge router of the on-premises vCenter Server network. This tunnel can be established over AWS backbone using Public VIF or an Internet connection. With Public VIF, the MGW public address will be included in the AWS prefixes customers receive on their on-premises edge router. We recommend you use a firewall filter to limit accepted prefixes to only targeted endpoints and ports when possible. In this case, the two public addresses of the MGW and CGW only.
  • L2 stretch to maintain addressing during VM migration. This requires an L2 tunnel between on-premises NSX edge VPN client (free download for VMC customers) and CGW L2 VPN server.


Figure 6 – Network requirements for vMotion between on-premises and SDDC.


VMware Cloud on AWS has different connectivity options available to remote networks. The VMware endpoints in the connected customer VPC provides low latency, high throughput, and seamless connectivity to VPC workloads and other AWS services. SDDC integration with AWS Direct Connect provides reliable, dedicated, and low latency connection between VMware SDDC and on-premises VMware environments. This optimized data transfer speed is required for reliable vMotion operations.

SDDC customers who need to connect to multiple VPCs can create tunnels over AWS backbone using an AWS-managed VPN with multiple options based on SDDC logical networks and connectivity requirements.

Additional Resources

Contact VMware Cloud on AWS

If you would like to see how VMware Cloud on AWS can help transform your business, please contact VMware >>