Hybrid cloud architectures using AWS Direct Connect gateway

In April 2023, AWS increased several AWS Direct Connect quota limits, as you have asked for increased scale and capacity for hybrid cloud connectivity. With the new limits, you can now create up to four Transit Virtual interfaces (VIFs) per AWS Direct Connect dedicated connection. The maximum number of prefixes has increased to 200 for a single AWS Transit Gateway association to a Direct Connect gateway. You can now associate up to six Transit Gateways or up to 20 Virtual Private Gateways (VGWs) to a single Direct Connect gateway. By using multiple Direct Connect gateways and Transit VIFs on a single Direct Connect dedicated connection, you can implement new designs to achieve several outcomes:

• Expand Direct Connect gateway connectivity out to six Transit Gateways
• Scale out global hybrid connectivity with a single Direct Connect dedicated connection up to 24 AWS Regions when using Transit Gateways
•  Create multiple, logical, private hybrid connections for ensuring the isolation of segmented AWS networks to on-premises networks
• Consolidate connectivity into a single Direct Connect dedicated connection when establishing logical networks into AWS commercial and AWS GovCloud (US) Regions, as well as VMware Cloud (VMC) on AWS and VMC on AWS GovCloud Transit Gateways

In this post, we look at the history of the Direct Connect service and its major feature launches. We also review Direct Connect gateway functionality and purpose in building private, hybrid cloud architectures. Then we introduce four scenarios where the new limits and design patterns can help you optimize the cost of your hybrid cloud architectures. Last, we provide a list of additional considerations to be aware of when building hybrid cloud connectivity to AWS.

The history of AWS Direct Connect

AWS launched Direct Connect in 2011 to help you bypass the internet and create dedicated connections to AWS. Over time, the service has expanded to over 130 global Direct Connect locations and has consistently added features and capabilities to meet your needs for private hybrid cloud connectivity. These features include Direct Connect gateway, multi-account association support, and private connectivity to support AWS Transit Gateway. The latest Direct Connect quota increases will help you meet your scaling needs while optimizing your costs through reducing physical infrastructure, as shown in the following timeline (figure 1).

Figure 1: AWS Direct Connect launch timeline

AWS Direct Connect gateway best practices

Let’s review what Direct Connect gateway is, along with some common best practices and guidance.

Direct Connect gateway is globally available and distributed as a logical construct. It is inherently highly available, so we recommend using a single Direct Connect gateway to achieve resiliency in your Direct Connect hybrid connection patterns.

You can create a Direct Connect gateway through Regional APIs and use it within all other AWS Regions in your account, excluding AWS China Regions.

There is no additional cost associated with using a Direct Connect gateway.

When using Direct Connect gateway, your traffic takes the shortest path to and from your Direct Connect location to the destination AWS Region, regardless of the associated home AWS Region of the Direct Connect location where you are connected. Direct Connect gateway is a control plane construct – it does not sit in the data path.

Direct Connect SiteLink sends traffic between Direct Connect locations using the shortest path routing possible, when two or more VIFs are connected to the same Direct Connect gateway. This is useful when building transit networks using AWS services.

You must configure a Direct Connect gateway with a different BGP Autonomous System Number (ASN) than any associated Transit Gateways, virtual private gateways (VGWs), or customer gateways.

Consider using MACsec for high-speed encryption to a Direct Connect dedicated connection from your corporate data center. All data flowing across the AWS global network that interconnects with data centers and AWS Regions is automatically encrypted at the physical layer before it leaves the data center.

If you need end-to-end encryption, or control over which cryptographic algorithms are used, consider using Private IP VPN or Public VIFs with AWS Site-to-Site VPN, as a Direct Connect gateway is only a logical separation construct.

We recommend Direct Connect gateways for establishing hybrid cloud connectivity when using Direct Connect Private VIFs.

AWS Direct Connect gateway data flows

We created Direct Connect gateway for private communications between on-premises networks and AWS cloud resources (that is, north-south). Use AWS Transit Gateway Peering, Amazon VPC Peering, or AWS Cloud WAN for intra- and inter-Region (that is, east-west) communication between your Amazon Virtual Private Clouds (VPCs). Direct Connect gateway does not enable BGP prefix propagation between associated Transit Gateways or VGWs. This indirectly denies east-west traffic between associations, except in one scenario, as described in our Direct Connect gateway documentation. We show these data flows in the following diagram (figure 2).

Figure 2: Permitted data flow (green), data flow with optional SiteLink enabled (blue) and not permitted data flow (red) with DirectConnect gateway.

AWS Direct Connect gateway BGP prefix propagation

Direct Connect gateway functions like a BGP route reflector. By default, Direct Connect gateway advertises learned BGP prefixes from on premises only to AWS associated resources (Transit Gateways and Amazon VPCs, for example)—and likewise AWS associated resources learn prefixes from on premises resources only.  Amazon VPCs and Transit Gateways only learn the prefixes advertised from each attached VIF. Similarly, the customer gateway connected to the VIF learns prefixes from all associated Direct Connect gateway resources configured, as shown in the following diagram (figure 3).

Figure 3: BGP AWS advertised (green) and received (blue) prefix limits

Direct Connect gateway doesn’t filter prefixes, so extending network segmentation for security isolation or customized routing decisions becomes burdensome at scale. When using local preference BGP community tags to create preferred routing paths to on premises, a singular routing domain is created and enforced by the global Direct Connect gateway object. This singular Direct Connect gateway and routing domain limits the ability to route traffic from specific AWS Regions to specific Direct Connect locations when you are using multiple Direct Connect connections for resiliency.

Although you have been able to use multiple Direct Connect gateways for Private VIFs and VPCs to scale beyond ten VPCs per Direct Connect gateway, many of you use Transit Gateway to interconnect thousands of VPC and on-premises hosted systems. This reduces the number of interfaces and BGP peers to configure and maintain per dedicated connection. Historically, there was only one Transit VIF for a single Direct Connect dedicated connection. Therefore, you would have to buy more Direct Connect connections if you needed additional Transit VIFs to scale beyond three Transit Gateways or three AWS Regions. There are alternate designs that you can use to work around these limits by using features of Transit Gateway, such as longest prefix match (LPM) routing or implementing Transit Gateway Connect attachments as overlay networks to extend network segmentation.

Now, with the new quota increases, the ability to have four Transit VIFs along with six Transit Gateway associations per Direct Connect gateway, you can simplify your configurations to implement your desired network segmentation. Simultaneously, you can establish a global scale network (when needed) in a cost-optimized fashion when building hybrid connections to AWS.

AWS Direct Connect gateway – New design patterns

Let’s explore the following scenarios to understand how the new quota increases are used to increase scale and add additional layers of segmentation:

• Scenario 1: Expand AWS Region footprint with connectivity up to six Transit Gateways and increase allowed prefixes through one Direct Connect gateway
• Scenario 2: Increase usage of existing Direct Connect dedicated connections with four Transit VIFs
• Scenario 3: Extend logically segmented networks to on-premises networks for separation between Line of Business (LOB) or tenants
• Scenario 4: Create isolation between AWS Commercial, AWS GovCloud (US), VMware Cloud on AWS, and VMware Cloud on AWS GovCloud (US) on a single Direct Connect dedicated connection

Note that we show connections between your premises and two Direct Connect locations in all scenarios to reflect the high resilience model, as noted in the Direct Connect resilience recommendations.

Scenario 1: Expand AWS Region footprint with connectivity up to six Transit Gateways
and increase allowed prefixes through one Direct Connect gateway

As shown in the following diagram (figure 4), you can use a single Direct Connect gateway and connect up to six Transit Gateways to any Regions globally, excluding China Regions. The number of listed prefixes allowed for a Transit Gateway association to Direct Connect gateway has increased to 200 (combined total for IPv4 and IPv6). These increased capacities allow you to scale your existing Direct Connect gateway implementations with up to six different AWS Regions, including AWS GovCloud (US) Regions. For those of you who implement more than one Transit Gateway within a single Region, this pattern gives you the capacity to expand into a multi-Region strategy on a single Direct Connect gateway and Transit VIF. Additionally, you can now advertise up to 1,200 prefixes to your premises over a single Direct Connect gateway and respective Transit VIF.

Note that it is not possible to increase a Transit Gateway’s allowed prefix limit beyond 200 by adding another Transit Gateway association that uses less than the 200-prefix limit.

Figure 4: Using one Direct Connect gateway and six Transit Gateways

Scenario 2: Increase usage of existing Direct Connect dedicated connections with four
Transit VIFs

Over time, how you use your network and the capacity you require changes. For hybrid cloud private connectivity using Transit Gateways, you previously had to buy multiple connections to support multiple Transit VIFs. Direct Connect now offers a cost optimized capacity that supports up to four Transit VIFs per Direct Connect dedicated connection. This allows you to consolidate your connectivity needs for up to four Transit VIFs across a singular dedicated connection to provide expansive AWS Regional connectivity. It also provides increased capacity for the total number of network prefixes to be advertised from AWS to on premises over a single physical connection without overlay networks up to 4,800 prefixes (for example, Scenario 1 defines up to 1,200 prefixes multiplied by four Transit VIFs).

Note that AWS Cloud WAN is a managed wide area network (WAN) that lets you create your own global network between AWS Regions with built-in automation, segmentation, and configuration management features designed specifically for building and operating global networks. You can peer Transit Gateways to Cloud WAN as part of your hybrid cloud design. If you would like to learn more, the Deploying hybrid networks using AWS Cloud WAN with AWS Direct Connect blog post may be useful.

Figure 5: Multiple Transit VIFs over single Direct Connect dedicated connection

Scenario 3: Extend logically segmented routing to on-premises networks for separation between Line of Business (LOB) or tenants

In this scenario, as depicted in the following diagram (figure 6), a customer has configured four Virtual Routing and Forwarding (VRFs) domains on the on-premises routers while using one Direct Connect connection and four Direct Connect gateways with a dedicated Transit Gateway. This creates end-to-end routing segmentation for each LOB. By provisioning separate Direct Connect gateways with their own Transit VIF, logical segmentation can be extended from a single Transit Gateway up to the customer gateway through the Virtual Interface.

Figure 6: Multiple Direct Connect gateways and Transit VIFs for traffic isolation

Scenario 4: Create isolation between AWS Commercial, AWS GovCloud (US), VMware Cloud on AWS, and VMware Cloud on AWS GovCloud (US) on a single Direct Connect dedicated connection

You may need to connect to multiple AWS hosted offerings because of compliance or service availability. Separate Direct Connect hosted or dedicated connections may have been implemented. Now, you can use a single Direct Connect dedicated connection for logically isolated, private connectivity to the following four AWS hosting environments: AWS commercial Regions, AWS GovCloud (US) Regions, VMC on AWS, and VMC on AWS GovCloud. As seen in the following diagram (figure 7), this design pattern uses shared physical network connectivity while enabling a broader scale of adoption within each environment. If you would like to learn more about these connectivity options with Direct Connect, the AWS Direct Connect Integration with VMware Cloud on AWS and AWS Hybrid Connectivity: Sharing AWS Direct Connect with AWS GovCloud (US) and commercial Regions blog posts may be helpful.

Figure 7: Shared physical connectivity to AWS commercial, AWS GovCloud (US), VMC on AWS and VMC on AWS GovCloud (US)

Considerations

Here are some more things to consider when implementing Direct Connect after the latest quota increase:

• Review the AWS Whitepaper on hybrid connectivity when beginning to explore your hybrid cloud architecture.
• To access critical workloads over Direct Connect, we strongly recommend configuring maximum resiliency with redundant Direct Connect connections to separate devices in more than one colocation facility. Use the AWS Direct Connect Resiliency Toolkit to get started in building resilient hybrid cloud architectures.
• Transit Gateway Connect and Private IP VPN attachment quotas allow for more prefixes to be advertised both from on-premises and from Transit Gateway where the prefix needs may be higher than Direct Connect VIFs support today.
• Transit Gateway route evaluation order:  Transit Gateway prefers Direct Connect gateway propagated prefixes over equivalent propagated prefixes from AWS Site-to-Site VPN or Transit Gateway connect attachments, even when those prefixes have more favorable BGP attributes.
• Creating active/passive BGP connections over AWS Direct Connect shows how BGP best path selection algorithms local preference and AS Path prepending can be configured in a data center to create active/passive connections when using Direct Connect.
• Engage with an AWS Well-Architected Partner Program or an AWS Solutions Architect to review the Well-Architected Framework Hybrid Networking Lens for existing deployments.
• Review the service quota documentation for the most accurate information

Conclusion

In this post, we shared how AWS has enabled hybrid cloud connectivity through the Direct Connect service since 2011. We shared best practices and guidance when using Direct Connect gateway, and we introduced several new design patterns that you can use when building hybrid cloud architectures. Last, we shared some additional points to consider when designing hybrid cloud network connectivity.