Building a global network using AWS Transit Gateway Inter-Region peering

Global companies have historically dedicated considerable financial and engineering resources to building wide-area-networks (WANs) so people in different geographies could stay connected and operate as a single entity. Those WANs are typically implemented using Carrier Ethernet (CE) services delivered by leading telecommunications providers, commonly referred to as MPLS services.

In recent years, the IT “center of gravity“ of these enterprises has – however – rapidly moved from on-premises data centers and branch offices to the cloud. Today, businesses predominantly choose the cloud to deploy their applications.

Because of this momentous shift, these global businesses are now looking at novel ways of making sure that staff around the globe is connected and has access to the vital business critical applications they need to succeed.

In this blog post, I describe how a fictitious global company, Example Corp., can architect a global network that relies on AWS Transit Gateway, AWS Direct Connect, and AWS Accelerated Site-to-Site VPN to build a modern, secure, scalable, and cost-efficient WAN on top of the AWS global network.

Building blocks

At re:Invent 2019 we announced that AWS Transit Gateway now supports inter-Region peering. This means that all the Amazon VPCs, Site-to-Site VPN connections, and Direct Connect gateways attached to an AWS Transit Gateway hosted in one AWS Region can exchange traffic with resources deployed in other AWS Regions. The inter-Region traffic is encrypted, traverses the AWS global network, and is not exposed to the public internet – thus reducing the attack surface.

The AWS global network infrastructure is designed and operated in a highly available way that minimizes latency between locations and is built using private network capacity owned and controlled by AWS.

At re:Invent 2019 we also announced Accelerated Site-to-Site VPN. Accelerated Site-to-Site VPN uses AWS Global Accelerator to optimize the experience of customers who connect to their Amazon VPCs through IPsec VPNs.

With Accelerated Site-to-Site VPN, customers terminate their IPsec VPN tunnels to an endpoint in an AWS edge location as close as possible to their customer gateway. This ensures the traffic is handed-over to the AWS global network as soon as possible.

Architecture

Requirements

Example Corp. is a global enterprise headquartered in London, United Kingdom, where they have a data center and their HQ office. They also have branch offices all around the world. The biggest ones are located in Los Angeles and Seoul, which I use to illustrate the architecture.

Example Corp. has the majority of their applications hosted in the eu-west-2 Region (London) and ap-northeast-2 (Seoul), and some legacy applications that are in the process of being migrated, hosted in the London data center.

Their branch offices require a network capacity of 500 Mbps to and from the AWS Regions and 10 Mbps to and from their data center. Routing between branches must not be allowed. The data center requires 5 Gbps network capacity to and from the AWS Regions to handle the migration of data-heavy workloads to AWS. This set of requirements is summarized by the following diagram.

Design

The design is shown in the following picture, which I am going to explain in further detail in the next sections of this post.

Data center

Given the throughput requirements (5 Gbps), the on-premises data center is connected to the eu-west-2 AWS Region via a resilient pair of 10 Gbps AWS Direct Connect dedicated connections. I create two Transit Virtual Interfaces (T-VIFs), one per Direct Connect connection and associate these to a Direct Connect Gateway (dxgw-eu-west-2). The Direct Connect Gateway is attached to an AWS Transit Gateway (tgw-eu-west-2) configured in the eu-west-2 region.

I specify which routes Direct Connect announces via the BGP routing protocol to customer gateways by adding them to the allowed prefixes list. Specifically, I announce the routes for the network the data center must reach (all VPCs, all branch offices ranges).

I also configure our Customer Gateways (one per Direct Connect connection) to establish BGP sessions with the Direct Connect Gateway and advertise the on-premises route for the 10.0.0.0/8 network on both Transit VIFs.

Branch offices

All branch offices are connected to AWS using Accelerated Site-to-Site VPN. By doing so, we ensure the customer data is taken onboard the AWS global network as close as possible to the customer location, so the benefits of using the AWS backbone are maximized.
I help the customer by making sure they have an internet connection at those branches capable of handling the required traffic volume (500 Mbps full-duplex) and a customer gateway (preferably more than one) that supports IPsec VPN technology. See this page for a full list of requirements.

I create three Accelerated Site-to-Site VPN connections for all the branches:

• the Seoul branch connects to a VPN connection configured in the ap-northeast-2 Region (vpn-icn);
• the London branch connects to a VPN connection configured in the eu-west-2 Region (vpn-lhr);
• the Los Angeles branch connects to two VPN connections configured in the eu-west-2 Region (vpn-lax).

Note: each VPN connection includes two VPN tunnels that can be used for high availability and increased throughput (if terminated on an AWS Transit Gateway and ECMP is enabled).

Note: to connect the Los Angeles branch you could enable a VPN connection to a closer AWS Region. However, as we are relying on Accelerated Site-to-Site VPN there is no performance benefit in doing so, as that Accelerated VPN connection is already terminated in an AWS edge location close to the customer (possibly closer than the closest AWS Region). I have configured the VPN connection in eu-west-2 because the business applications deployed in that AWS region are considered ‘primary’ for the Los Angeles branch IT systems configuration.

AWS Transit Gateway and inter-Region connectivity

So far, I have described how connectivity is provided for all the physical (branches and data center) and virtual (Amazon VPC) resources of our customer. We now have to think about how we enable the network connectivity requirements between all these locations that I have described in the previous section.

This is made possible by AWS Transit Gateway. An AWS Transit Gateway is a regional construct that can be seen as a highly scalable cloud router, which also consolidates inbound connectivity into an AWS Region (VPN, VPC, Direct Connect, inter-Region peering).

Both AWS Regions in this architecture have an AWS Transit Gateway configured. AWS Transit Gateway supports different types of attachments, such as an Accelerated Site-to-Site VPN connection, a VPC, an inter-Region peering or a Direct Connect Gateway.

The two AWS Transit Gateways are peered with one another and the appropriate routing is configured so the required network connectivity between the Regions is enabled.

Note: I created the two AWS Transit Gateways with different Autonomous System Numbers (ASN) numbers. The Autonomous System Number (ASN) is a parameter used by the BGP routing protocol to uniquely identify a BGP router. You have to specify an ASN when you create an AWS Transit Gateway, and you cannot change that later.

Today, BGP route propagation between AWS Transit Gateways in different Regions is not available, but choosing unique ASNs is a good practice as it can help head off the need for changes down the road.

The following image shows the list of AWS Transit Gateway attachments in the eu-west-2 AWS Region.

The following image shows the list of AWS Transit Gateway attachments in the ap-northeast-2 AWS Region.

The connectivity we must provide to all the locations of our customer is not any-to-any. The branch offices, the data center, and the Amazon VPCs attached to the AWS Transit Gateways in the two AWS Regions have different network reachability requirements.

An AWS Transit Gateway can have multiple route tables that can be associated to different attachments. This means that when network traffic comes into AWS Transit Gateway from a certain attachment, AWS Transit Gateway makes a forwarding decision (decides where to send the packet next) using the route table associated to that attachment. This feature of AWS Transit Gateway allows customer to implement network communication patterns such as the one we are addressing in this design.

In this architecture we create the following route tables:

• rt-eu-west-2-vpn: this is associated to the London and Los Angeles branch office VPN attachment;
• rt-eu-west-2-dxgw: this is associated to the Direct Connect Gateway attachment;
• rt-eu-west-2-xrp: this is associated to the inter-Region peering attachment in theeu-west-2 AWS Region;
• rt-eu-west-2-vpc: this is associated to the Amazon VPCs in the eu-west-2 AWS Region;
• rt-ap-northeast-2-vpn: this is associated to the Seoul branch office VPN attachment;
• rt-ap-northeast-2-vpc: this is associated to the Amazon VPCs in the ap-northeast-2 AWS Region;
• rt-ap-northeast-2-xrp: this is associated to the inter-Region peering attachment in the ap-northeast-2 AWS Region.

Like any normal IP router, AWS Transit Gateway decides the next hop for a packet by doing a lookup operation in one of its route tables.

Entries can be added to an AWS Transit Gateway route table in two ways:

• manually: we can add static route table entries to an AWS Transit Gateway route table using the API, SDK, CLI, or AWS Management Console;
• dynamically: by propagating other attachments into a route table. Each attachment comes with routes that can be installed to one or more AWS Transit Gateway route tables. When an attachment is propagated to an AWS Transit Gateway route table, these routes are installed in the route table.
  Note: propagation is currently not available for inter-Region peering attachments, therefore routes for these attachments must be manually added to an AWS Transit Gateway route table.

The following tables represent which attachments are propagated to which AWS Transit Gateway route tables.

I will now show some details about the rt-eu-west-2-vpn route table as an example. For brevity, I omit the other route tables in either AWS Region.

The rt-eu-west-2-vpn route table is associated to the VPN attachments in the eu-west-2 Region. 

In the following screenshot, we see that the dxgw-eu-west-2, vpc-a-eu-west-2, and vpc-b-eu-west-2 attachments are propagated into the rt-eu-west-2-vpn route table.

The following image shows the content of the rt-eu-west-2-vpn route table.

Some of the routing entries in rt-eu-west-2-vpn have been automatically added via propagation (route type: propagated), whereas some have been entered manually (route type: static).

The static routes are required by the VPN attachments in eu-west-2 to reach the VPCs in the ap-northeast-2 Region. You may also notice that there are no entries to reach the VPN attachments in the ap-northeast-2 Region. This is because networking between branch offices must not be allowed.

Visibility and monitoring

In this previous blog post, we talked about how a global network such as the one I designed today can be monitored using AWS Transit Gateway Network Manager. With Network Manager, you have a centralized and aggregated view of all the AWS resources that are part of your global network. You can monitor things like VPN or AWS Transit Gateway metrics or react to events that affect your global network such as topology changes, routing, or status updates.

Conclusion

In this post I have shown how an enterprise customer can build a global network on top of the AWS backbone using AWS services such as AWS Transit Gateway, Accelerated Site-to-Site VPN and Direct Connect. This architecture can scale up to thousands of branch offices worldwide and can be deployed in a matter of minutes (except for Direct Connect) via the AWS Management Console, CLI, API or SDK.

You can get started right now on the AWS Management Console. If you want more details on how AWS Transit Gateway or Accelerated-Site-to-Site VPN work please head to our documentation website.

A correction was made on February 6, 2024: An earlier version of this post misstated route table names in a table representing which attachments are propagated to which AWS Transit Gateway route tables. The post has been updated to include the correct route table names. 

About the Author