Using AWS Transit Gateway connect to extend VRFs and increase IP prefix advertisement

Overview

You can solve advanced network use-cases encountered by Service Providers extending AWS cloud hosted services to their customers. Doing this often requires advertising thousands of IP prefixes into the AWS cloud, while maintaining separation of unavoidable conflicting IP address space. This can be accomplished by increasing IP prefix advertisement and extending Virtual Routing and Forwarding (VRF), using AWS Transit Gateway Connect. Two use-cases presented in this post use Transit Gateway Connect, a Transit Gateway attachment type that supports Generic Routing Encapsulation (GRE).

You can use the native Transit Virtual Interface (Transit VIF) of AWS Direct Connect to propagate a limited number of IP prefixes per AWS Transit Gateway, from AWS to on-premise networks. Refer to the Direct Connect quotas page for the currently supported number of IP prefixes. However, this may not be enough if you cannot use summarization or other methods to stay within the quota. This post will show the use of Transit Gateway Connect, which lets you increase these propagated routes to to a much higher number over the GRE tunnel. Refer to the Amazon VPC Transit Gateways Quotas page for the currently supported value of route advertised from and to connect peers..

Service Providers may want to extend Virtual Routing and Forwarding (VRF) from on-premises networks into AWS. They often use VRFs, which let you connect to them for hosted services, while enabling isolation of network traffic. Before Transit Gateway Connect, this was a daunting task, as many customers of Service Providers were using overlapping RFC 1918 CIDR ranges. Here, we show how Service Providers can use Transit Gateway Connect to extend VRFs into AWS. This lets them provide hosted services, while maintaining network traffic isolation in AWS, regardless of the IP schemas used by their customers.

Advertising Routes at scale using Transit Gateway Connect Attachments over Direct Connect

Transit Gateway is commonly used to achieve transitive routing between Virtual Private Clouds (VPCs) and connections to your premises over Direct Connect. We show this design pattern in the following figure (figure 1), where a Transit Gateway is between a single Amazon VPC and on-premises networks. The Transit Gateway associates with a Direct Connect gateway. You can advertise a limited number of IP prefixes per Transit Gateway across a Direct Connect transit virtual interface (Transit VIF). Refer to the Direct Connect quotas page for the currently supported number of IP prefixes.

Figure 1 – Customer Design Pattern with Transit Gateway and Single VPC

As customers grow the number of VPCs or VPC CIDRs, they can use route summarization when configuring the association between Transit Gateway and Direct Connect gateway. Summarizing routes is one strategy that can help you remain under the quota threshold for Direct Connect transit VIFs. In some scenarios, aggregating IP prefixes is not an option, possibly because of building networks without an IP address plan. To take advantage of route summarization, you would have to re-architect your networks, which could result in downtime. The following figure (figure 2) demonstrates a scenario with a single Transit Gateway and minimal options for route summarization because of the discontinuous CIDRs per VPC.

Figure 2 – Customer Design Pattern with Transit Gateway and Multiple VPCs

Transit Gateway Connect attachments establish a GRE tunnel between a Transit Gateway and a third-party virtual appliance. BGP peering is configured during the Transit Gateway Connect peer creation. You can grow to any number of VPCs without re-architecting your network to use route summarization with Transit Gateway Connect attachments. In the following figure (figure 3) , your third-party virtual appliances are shown in the respective on-premises data centers.

Figure 3 – Customer Design Pattern using Transit Gateway Connect Attachment and Multiple VPCs

In the following diagram (figure 4), the Direct Connect connections and transit Virtual Interfaces (VIFs) compose the underlay. We configured VLANs on the transit VIF, which is shown as VLAN 110. We configured BGP peers on transit VIFs, shown as 169.254.0.0/31 and 169.254.0.1/31. The BGP peering session is established between the Direct Connect gateway and the on-premises router.

Figure 4 – Transit Gateway with Transit VIF architecture

The Transit Gateway Connect attachment is shown as the overlay in the following figure (figure 5). The Peer IP and Transit Gateway addresses in the following figure are the only required IP prefixes to be advertised across the transit VIF BGP peering session. The Transit Gateway Connect attachment is configured between the Transit Gateway and the customer router.

Figure 5 – Transit Gateway Connect attachment architecture

Transit Gateway Connect peers must support GRE and specify a /29 CIDR block from the 169.254.0.0/16 range for IPv4 or fd00::/8 range for IPv6. The Transit Gateway address requires a unique IP address from the transit gateway CIDR block. Customers should follow the Direct Connect high resiliency recommendations. Lastly, there’s no need for additional Transit Gateways to meet high availability requirements, because transit gateways are highly available by design.

Extending VRFs into AWS with Direct Connect and Transit Gateway Connect

Service Provider organizations use VRFs to maintain isolated connectivity to their services. This provides their customers with access to services on their premises. However, it maintains the separation of multi-customer traffic for security. This saves on costs for both providers and their customers, as dedicated infrastructure is not needed. Service providers that want to extend traffic isolation into AWS using VRFs can use Transit Gateway Connect.

The following diagram (figure 6) shows a scenario where a Service Provider extends two VRFs dedicated to two distinct customers into AWS. The customer networks that the Service Provider has extended into AWS are displayed in red and green. In this example, the same CIDR ranges (i.e. 10.1.0.0/16 – on premises / 172.16.0.0/16 AWS cloud) are used by both customers. AWS recommends using non-overlapping CIDR ranges, but many service providers cannot dictate the IP schemas used by their customers. To address the overlapping address ranges, separate Transit Gateway Connect attachments are used to extend the VRFs between the Service Provider customer router and the Transit Gateway. GRE tunnels are formed via the Transit Gateway Connect attachments. This establishes separate BGP peering sessions for each customer VRF between the Transit Gateway and the Service Provider’s router. Customer network traffic remains isolated as it travels between AWS and the Service Provider’s on-premises location.

In AWS, VPCs connected to the Transit Gateway via Transit Gateway attachments send traffic to and from resources within their respective VPCs. This is done using separate route tables on the Transit Gateway. Each Transit Gateway route table is also associated to a different connect attachment. Traffic coming from the VPC destined to on-premises IPs can be routed through the respective GRE tunnels. This makes it possible for Service Providers to extend their customers network traffic into AWS using VRFs, as well as keep it separated. Furthermore, this allows for possible overlapping CIDR ranges among customers connected to a Service Provider’s network.

Figure 6 – VRF Extension from On-Premise to AWS Using Distinct GRE Tunnels

There is a quota of four Transit Gateway Connect peers (GRE tunnels) per Transit Gateway Connect attachment. Customers must also account for the hard quota of CIDR blocks per Transit Gateway, which is used by Transit Gateway Connect Attachments and Transit Gateway Connect peers. Currently, this quota cannot be increased. You can find these quotas documented at the Transit Gateway quotas site. Lastly, there are also quotas to consider regarding EC2 instance network bandwidth.

We have covered a scenario involving Service Providers, but we can use the same principle for other organizations that want to extend VRFs. For example, some enterprise organizations separate production and non-production network traffic to allow for testing, research, and development. If they want to extend their on-premises services into AWS, and let them remain isolated, then they can take advantage of this architecture.

Summary

AWS Transit Gateway Connect attachments can address advanced network challenges. The AWS Transit Gateway Connect attachment provides an option for advertising more prefixes than when using only a single Direct Connect transit VIF. AWS Transit Gateway Connect attachments help you obtain the separation of network traffic over common physical infrastructure. Consult your Technical Account Manager (TAM) if you subscribe to AWS Enterprise Support or AWS Solutions Architect for help.

An update was made on September 11, 2024: An earlier version of this post included quota information for AWS Direct Connect and AWS Transit Gateway that have been increased since. The post has been updated to point to the relevant quota pages within the AWS documentation instead.

About The Authors