Integrate SD-WAN devices with AWS Transit Gateway and AWS Direct Connect

Many AWS customers like to use their existing Software Defined Wide Area Network (SD-WAN) devices when connecting their on-premises networks to an AWS Transit Gateway. When doing this, a large number of prefixes must be advertised to and from AWS Transit Gateway. In this post, we show how to use the Transit Gateway Connect feature along with AWS Direct Connect to advertise a higher number of prefixes, increase bandwidth, and use multi-protocol BGP (Border Gateway Protocol) support for route exchange. We also look at ways to extend and segment your SD-WAN traffic into your AWS network.

AWS Transit Gateway Connect terminology

Because Transit Gateway Connect was launched fairly recently, let’s review a few terms before we dive in to the integration process:

• Transit Gateway Connect attachment: This type of attachment supports up to four GRE tunnels between a Transit Gateway and a network appliance (routers, firewalls, and virtual appliances) located inside AWS, or in your on-premises network. A Transit Gateway Connect attachment requires a Transport, or underlay, attachment. That attachment can be either an Amazon Virtual Private Cloud (VPC) or a Direct Connect attachment.
• Transit Gateway Connect peer: This is created on the Connect attachment that connects your Transit Gateway and your third-party appliance. You establish GRE and BGP sessions to the Transit Gateway Connect peer to exchange routing information.

For more on the fundamentals of Transit Gateway Connect, the blog post, Simplify SD-WAN connectivity with AWS Transit Gateway Connect, is a helpful resource.

This blog post provides step-by-step guidance for integrating Direct Connect with Transit Gateway Connect and looks at how to integrate third-party appliances into an AWS environment. It also presents reference architectures for dividing a Direct Connect connection for production and non-production environments.

Integrating AWS Direct Connect with AWS Transit Gateway Connect

You can establish private network connectivity between your on-premises network to Transit Gateway Connect by following these six steps:

Step 1: Create AWS Direct Connect Gateway

Navigate to the AWS Direct Connect console and create a new Direct Connect Gateway, or choose an existing Direct Connect Gateway. This is shown in the following graphic (Figure 1).

Figure 1: Creating new Direct Connect Gateway

Step 2: Create or modify an existing AWS Transit Gateway

The Transit Gateway you are connecting to can have an IPv4 or IPv6 address, but it must be in the same IP address family as the peer IP address. For this post, we modify an existing Transit Gateway and add both IPv4 and IPv6 Classless Inter-Domain Routing (CIDRs) to it as shown in the following image (Figure 2).

Figure 2: Modify Transit Gateway Console

Transit Gateway CIDRs can be any public or private IP address ranges, except for addresses in the 169.254.0.0/16 range, and ranges that overlap with the addresses of your VPC attachments and on-premises networks. We are creating an IPv4 peer, but you also have the option to create an IPv6 peer (outer IP) with your Transit Gateway.

An IP address within the added CIDR block is allocated for the Transit Gateway side of the Transit Gateway Connect peer (GRE tunnel), and is configured as the GRE outer IP address. Modifying your Transit Gateway to add CIDRs does not affect your existing attachments or traffic flow over the Transit Gateway.

Step 3: Associate AWS Direct Connect Gateway with AWS Transit Gateway

Associating these gateways creates the underlying transport attachment that is used by Transit Gateway Connect for network connectivity. Under Allowed Prefixes, entire Transit Gateway IPv4 or IPv6 CIDR (If you plan to have multiple Transit Gateway Connect Peers), or you can just add the Transit Gateway Connect peer IPs you plan to assign to the GRE tunnels. We use an IPv4 CIDR to create the GRE tunnels, but you have the option assigning IPv6 CIDRs for Transit Gateway Connect GRE outer addresses.

Figure 3: Associating Transit Gateway with Direct Connect Gateway

Step 4: Create AWS Transit Gateway Connect attachments and AWS Transit Gateway Connect peers

Navigate to the Transit Gateway Attachment tab and create a new Transit Gateway Attachment. In the field next to Attachment type, select the options for Connect. Then, select Transport Attachment as the Direct Connect Gateway Attachment created from previous steps.

Figure 4: Creating Transit Gateway Connect attachment with Transport as Direct Connect Gateway attachment

Select the Transit Gateway Connect Attachment once it is available. Then you can create a Transit Gateway Connect peer as shown in the following screenshot (Figure 5).

Figure 5: Create a Transit Gateway Connect peer

A quick look at tunnel options:

• • Transit Gateway GRE address – This is the outer IP address of the GRE tunnel assigned to the Transit Gateway side of the tunnel.
  • Peer GRE address – This is the outer IP address that is assigned to the on-premises appliance/firewall/router that’s creating the GRE tunnel with AWS Transit Gateway.
  • BGP Inside CIDR Blocks IPv4 – These are inside peer IPs assigned to the first IP from the block is assigned to your appliance, firewall, or router. The second and third IPs are allocated to the GRE tunnel on the AWS side of the connection. (See the Transit Gateway Connect peers documentation for the complete list of allowed CIDRs.)
  • BGP Inside CIDR blocks IPv6 – These are IPv6 addresses that are used to configure next-hop routing when running Multi-Protocol Border Gateway Protocol (MP-BGP). This CIDR is not used to established IPv6 BGP sessions.
  • Peer ASN – An Autonomous System Number (ASN) is used by the appliance, firewall, or router running the GRE tunnel. If you do not specify an ASN, then the ASN allocated to your Transit Gateway is used. In other words, we assume that you plan to run iBGP session between the Transit Gateway and your appliance.

Step 5: Create a Transit Virtual Interface (Transit VIF)

To connect your AWS Direct Connect connection to your Transit Gateway, you must create a transit virtual interface (Transit VIF). To provision a Transit VIF, follow the steps in the Create a transit virtual interface to the Direct Connect gateway documentation.

Step 6: Establish GRE tunnel

Once your Transit Virtual Interface is established, start exchanging prefixes with Direct Connect. The prefixes announced to Direct Connect must include Peer GRE Address IP to allow connectivity between the appliance and the Transit Gateway. This, however, does not require any changes to the Transit Gateway’s route table. Adding the Transit Gateway CIDRs to the route table associated with Direct Connect Gateway attachment causes traffic flow to fail. In this example, we are advertising 10.0.0.10/32 (Peer GRE IP address) over our Transit Virtual Interface.

At this point, you have everything you need to start configuring a GRE tunnel on your appliance. We’ll use a MP-BGP configuration on the on-premises appliance that’s running a GRE tunnel with AWS Transit Gateway.

We’ll use the architecture shown in the following diagram (Figure 6) to configure GRE tunnel and MP-BGP.

Figure 6: Transit Gateway Connect with Direct Connect Gateway attachment

In this example, we establish connectivity to one VPC (VPC B, as shown in the preceding diagram (Figure 6). However, you can establish connectivity to multiple VPCs by propagating VPC CIDRs into Transit Gateway route table associated with Transit Gateway Connect attachment if you prefer.

Configuration walkthrough

Step 1: GRE configuration

Start building the GRE tunnel once you have a Transit Virtual Interface established and are exchanging BGP prefixes. The following configuration creates a GRE tunnel between your appliance and the Transit Gateway. The tunnel is established over IPv4 IP address space (Outer IP) and not IPv6.

We are using a Cisco appliance vendor configuration in this example, but there are many other options available. The exact commands used will vary based your specific appliance.

We start by creating a Virtual Tunnel Interface and assigning inner IPv4 and IPv6 addresses that we selected in “Step 4: Create AWS Transit Gateway Connect attachments and AWS Transit Gateway Connect peers.”

Reduce the IP MTU to accommodate for GRE overhead, and set the tunnel source and destination IPs using outer IP addresses defined earlier in Step 4. In this example, we are establishing a GRE tunnel over IPv4 outer IP addresses. However, you can create GRE tunnel over IPv6 outer IP address as well.

Transit Gateway supports an MTU of 8500 bytes for traffic between Transit Gateway Connect (includes GRE header) and your appliance. If needed, adjust the tunnel MTU if you are not using jumbo frames for your configuration.

interface Tunnel1
ip address 169.254.100.1 255.255.255.248
ip mtu 8476
ipv6 enable
ipv6 address fde2:aa1a:d66:443b:d9d8:30c9:9674:6379/125
tunnel source 10.0.0.10
tunnel destination 172.16.0.10

#ping 169.254.100.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.100.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 74/75/76 ms

#ping 169.254.100.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.100.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 74/75/76 ms

!
route-map IPV6_NEXT_HOP permit 10
set ipv6 next-hop fde2:aa1a:d66:443b:d9d8:30c9:9674:6379
end
!

router bgp 65002
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 169.254.100.2 remote-as 64512
neighbor 169.254.100.2 ebgp-multihop 2
neighbor 169.254.100.3 remote-as 64512
neighbor 169.254.100.3 ebgp-multihop 2
!
address-family ipv4
network 192.168.1.0 mask 255.255.255.0
neighbor 169.254.100.2 activate
neighbor 169.254.100.3 activate
exit-address-family

!
address-family ipv6
network 2001:db8:156:3b01::/64
neighbor 169.254.100.2 activate
neighbor 169.254.100.2 route-map IPV6_NEXT_HOP out
neighbor 169.254.100.3 activate
neighbor 169.254.100.3 route-map IPV6_NEXT_HOP out
exit-address-family
!

#show ip bgp summary

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
169.254.100.2 4 64512 182 191 5 0 0 00:29:32 1
169.254.100.3 4 64512 182 195 5 0 0 00:29:36 1

#show bgp ipv6 summary

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
169.254.100.2 4 64512 182 192 3 0 0 00:29:36 1
169.254.100.3 4 64512 182 195 3 0 0 00:29:39 1

#show ip bgp

Network Next Hop Metric LocPrf Weight Path
* 172.31.0.0/16 169.254.100.2 100 0 64512 i
*> 169.254.100.3 100 0 64512 i
*> 192.168.1.0 0.0.0.0 0 32768 i

#show bgp ipv6 unicast

Network Next Hop Metric LocPrf Weight Path
*> 2001:DB8:156:3B01::/64
:: 0 32768 i
* 2001:DB8:C0D:7E00::/56
FDE2:AA1A:D66:443B:D9D8:30C9:9674:637A
100 0 64512 i
*> FDE2:AA1A:D66:443B:D9D8:30C9:9674:637B
100 0 64512 i

[ec2-user@ip-172-31-4-155 ~]$ ping 192.168.1.10 -c 2
PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
64 bytes from 192.168.1.10: icmp_seq=1 ttl=254 time=76.2 ms
64 bytes from 192.168.1.10: icmp_seq=2 ttl=254 time=75.9 ms

--- 192.168.1.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 75.942/76.085/76.229/0.310 ms

[ec2-user@ip-172-31-4-155 ~]$ ping6 2001:db8:156:3b01::1 -c 2
PING 2001:db8:156:3b01::1(2001:db8:156:3b01::1) 56 data bytes
64 bytes from 2001:db8:156:3b01::1: icmp_seq=1 ttl=63 time=75.9 ms
64 bytes from 2001:db8:156:3b01::1: icmp_seq=2 ttl=63 time=75.7 ms

--- 2001:db8:156:3b01::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 75.789/75.867/75.945/0.078 ms

In your Transit Gateway route table, you’ll see that routes received through Transit Gateway Connect. You do not have to make any routing changes in your Transit Gateway route table in order to route between your Appliance/Firewall/router and Transit Gateway Connect Peer IP addresses. As long as you advertise your peer GRE outer IP address, Transit Gateway is able to reach the appliance, router, or firewall.

Figure 7: Transit Gateway Route Table entries

If you have same prefix propagated into your Transit Gateway route table coming from VPN, Direct Connect, and Transit Gateway Connect attachments, we evaluate the best path in the following order:

Priority 1 – Direct Connect Gateway attachment

Priority 2 – Transit Gateway Connect attachment

Priority 3 – VPN attachment

The two Connect BGP Peers provided by Transit Gateway Connect are there for High Availability (HA). When you create Transit Gateway Connect peer, we ensure that Transit Gateway Connect peers land in different Availability Zones. This provides fault-tolerance, so even if one AZ goes down, you’ll not notice any change in your traffic flow. Your Transit Gateway can use ECMP between Transit Gateway Connect peers of the same Transit Gateway Connect attachment. It might also use ECMP between your Transit Gateway Connect attachments on the same Transit Gateway. Transit Gateway cannot use ECMP between the BGP peerings of the same Transit Gateway Connect peer.

To achieve true high availability on both sides, we recommend the uses of two different Transit Gateway Connect peers and establishing GRE tunnels with two peer appliances. This is shown in the following diagram (Figure 8).

Figure 8. Transit Gateway Connect and Direct Connect High Availability Design

You can use Transit Gateway Connect to segment the network traffic running on top of a single Transit Virtual Interface. Network segmentation can be done at two levels:

1. By using a Transit Gateway routing domain and associating a Transit Gateway Connect attachment with a separate route table. This is shown in the following diagram (Figure 9).

Figure 9: Network segmentation using Transit Gateway Connect and Direct Connect

2. By attaching Transit Gateways (until you hit the limit of three) to your Direct Connect Gateway and creating a Transit Gateway Connect. This is shown in the following diagram (Figure 10).

Figure 10: Segmentation using Transit Gateway Connect and Direct Connect

Note: These example diagrams (Figure 9 and Figure 10) show a single Direct Connect connection to simplify the diagram in order to increase clarity. This architecture is only recommended for non-production workloads. We recommend using the Maximum or High Resiliency Direct Connect models for critical production workloads. For detailed information, review the AWS Direct Connect Resiliency Recommendations page.

In order to separate network traffic, you can create multiple Transit Gateway Connect attachments and associate them with different Transit Gateway route tables. Adding multiple connect peers on the same Transit Gateway Connect attachment allows you to scale for bandwidth. If you would like to segment the network, then you must create multiple Transit Gateway Connect attachments.

In addition, if you want control plane segmentation, attach multiple Transit Gateways to a Direct Connect Gateway. When doing this, remember that you can only have three Transit gateways attached per AWS Direct Connect gateway.

Conclusion

In this blog, we walked through how to integrate Direct Connect with Transit Gateway Connect. With these reference architectures, you can extend and segment your SD-WAN into AWS. Transit Gateway Connect will make your transition from IPv4 to IPv6 simpler by providing hassle-free network connectivity with MP-BGP support.

To learn more about setting up Transit Gateway Connect attachment with VPC as transport, please see our documentation.