Field Notes: Working with Route Tables in AWS Transit Gateway

An AWS Transit Gateway enables you to attach Amazon VPCs, AWS S2S VPN and AWS Direct Connect connections in the same Region, and route traffic between them. Transit Gateways are designed to be highly scalable and resilient. You can attach up to 5000 VPCs to each gateway and each attachment can handle up to 50 Gbits/second of bursty traffic.

In this post,  I explain the packet flow if both source and destination network are associated to the same or different AWS Transit Gateway Route Table. An AWS Transit Gateway Route Table includes dynamic routes, static routes and blackhole routes. This routing operates at layer 3, where the IP packets are sent to a specific next-hop attachment, based on the destination IP addresses. You can create multiple route tables to separate network access. AWS Transit Gateway controls how traffic is routed to all the connected networks using route tables.

Architecture overview

To illustrate how AWS Transit Gateway route tables work, consider the following architecture with resources in multiple AWS accounts (Prod, Pre-Prod, Staging, Dev, Network Service) and all the accounts are under the same AWS Organization.

Figure 1 – How different AWS accounts are connected via AWS Transit Gateway

Overview

I have created a transit gateway in Network Service account and shared the Transit Gateway with other AWS accounts (Prod, Pre-Prod, Staging, Dev) in an AWS Organization with AWS Resource Access Manager (RAM).

AWS RAM enables you to easily and securely share AWS resources with any AWS account, or within your AWS Organization.  I created three Transit Gateway route tables: Prod Route table, Network Service Route table and Staging Route table.

To attach VPCs, VPN and Direct Connect connections in the same region to Transit Gateway, you need to create Transit Gateway attachments. The process is as following:

• When you attach VPC to Transit Gateway, you must specify one subnet in each Availability Zone to be used by Transit Gateway to route traffic.
• Create a connectivity subnet in all VPC and define connectivity subnets for the Transit Gateway attachment.
• Transit Gateway places a network interface in the connectivity subnet using one IP address from the subnet. Specifying one subnet for an Availability Zone enables traffic to reach resources in other subnets in that Availability Zone.
• If an Availability Zone is not associated when you create Transit Gateway attachments to attach the VPC, resources in that Availability Zone cannot reach the Transit Gateway.

Note: – We can have 20 TGW route tables per Transit Gateway and 10,000 routes per Transit Gateway.

Resources in AWS accounts and on-premises

We have AWS S2S VPN that connects on-premises and TGW in Network Service account. AWS S2S VPN connection is attached to the TGW attachment (TGW Attachment-4) in the Network Service account. Following is the network information for the on-premises network.

The following table shows VPC CIDR blocks in different AWS accounts and the respective Transit Gatewayattachments. It also has one test instance in each VPC in every AWS account.

VPCs, VPN/Direct Connect connections can dynamically propagate routes to the Transit Gateway route table. You can enable or disable route propagation for each Transit Gateway attachment. For a VPC attachment, the CIDR blocks of the VPC are propagated to the Transit Gateway route table. For a VPN/Direct Connect connection attachment, routes in the Transit Gateway route table propagate to your on-premises router/firewall using Border Gateway Protocol (BGP). The prefixes advertised over BGP session from on-premises router/firewall are propagated to the Transit gateway route table.

Transit Gateway Route Table Association

Transit Gateway attachments are associated to a Transit Gateway route table. An attachment can be associated to one route table. However, an attachment can propagate their routes to one or more Transit Gateway route tables.

The following table shows route table and associated Transit Gateway attachments.

To understand how the packet flow within or different Transit Gateway route table. In this post, I have explained it in three different scenarios.

Important:  For all scenarios, I have summarized the network 10.0.0.0/8 and added static route in VPC’s route table, so that we don’t need to make changes on the subnet’s route table every time you want to access resources in different AWS accounts. However, we are able to control the traffic from TGW route table. This reduces operational overhead and gain the ability to centrally manage all the connectivity.

Scenario 1: Packet flow between Prod account and Pre-Prod Account

You are trying to do SSH to Instance-2 (pre-prod) from Instance-1 (Prod). The following image is the packet flow for first two steps of TCP 3-Way handshake between the instances.

Figure 2 – Packet flow between Prod and Pre-Prod account

AWS Security Groups and NACLs are configured to allow communication for both the instance.

The source IP address of instance-1 is 10.1.10.10 and destination IP address of  instance-2 is 10.2.10.10.

1. SYN packet is validated against VPC route table associated to the subnet that has instance-1 and VPC route table has route for the network 10.0.0.0/8 points to Transit Gateway attachment. Packet is forwarded to TGW attachment.
2. Transit Gateway receives the traffic on the Prod TGW route table since Transit Gateway attachment (TGW Attachment-1) is associated to the Prod TGW route table. Prod TGW route table has route for the network 10.2.0.0/16.
3. SYN packet is evaluated against Prod Transit Gateway route table and forwarded to Pre-Prod VPC. Then it is evaluated against VPC route table of the subnet of TGW attachment and NACL of the connectivity subnet.
4. Packet is forwarded from the Transit Gateway attachment ENI to instance-2(10.2.10.10) after the NACL and Security groups are evaluated. For the return traffic (SYN/ACK), source IP is 10.2.10.10 and destination IP is 10.1.10.10
5. SYN/ACK packet is evaluated against VPC route table associated to the subnet that has instance-2 and VPC route table has route for the network 10.0.0.0/8 points to TGW attachment. Packet is forwarded to TGW attachment.
6. TGW receives the traffic on the Prod TGW route table since the Transit Gateway attachment (TGW Attachment-2) is associated to the Prod TGW route table. Prod TGW route table has route for the network 10.1.0.0/16.
7. SYN/ACK packet is evaluated against Prod TGW route table and forwarded to Prod VPC. Then it is evaluated against VPC route table of the subnet of Transit gateway attachment and NACL of the same.
8. SYN/ACK packet is forwarded from TGW attachment ENI to instance-1 after the NACL and Security group inbound states are evaluated.

Result:  Instance-1 is able to receive SYN/ACK packet from instance-2 successfully.

This is the packet flow when both source and destination networks are associated to the same Transit Gateway route table.

Scenario 2: Packet flow between Staging Account and Pre-Prod account

Pre-Prod account’s Transit Gateway attachment (TGW Attachment-2) is associated to Prod TGW Route table and Staging account’s TGW attachment (TGW Attachment-5) is associated to Staging Transit Gateway route table in Network Service account.

On the Transit Gateway route table, you can either add static route or create propagation to attach Transit Gateway attachment and routes will be propagated in the Transit Gateway  route table.  In this post, I chose the option to create propagation to propagate the route in the Transit Gateway route table.

On the Prod Transit Gateway route table, you must create propagation and choose Transit Gateway attachment (TGW Attachment-5) and the VPC CIDR block 10.4.0.0/16 is propagated in Prod TGW route table. Similarly, you must create propagation in Staging Transit Gateway route table and choose TGW attachment (TGW Attachment-2), the VPC CIDR block 10.2.0.0/16 is propagated in the Staging route table.

If you are trying to ping Instance-4 (Staging) from the Instance-2 (Pre-Prod),  the ICMP echo reply doesn’t reach Instance-2 from Instance-4 . Below is the packet flow for ICMP echo request and echo reply between the instances.

Figure 3. Packet flow between Pre-Prod and Staging account

AWS Security Groups and NACLs are configured to allow communication for both the instances.

Source IP address of Instance-2 is 10.2.10.10 and destination IP address of  Instance-4 is 10.4.10.10.

1. Echo-Request packet is validated against VPC route table associated to the subnet that has instance-1 and VPC route table has route for the network 10.0.0.0/8 points to the Transit Gateway attachment. Packet is forwarded to the Transit Gateway attachment.
2. TGW receives the traffic on the Prod TGW route table since the Transit Gateway attachment (TGW Attachment-2) is associated to the Prod TGW route table. Prod TGW route table has route for the network 10.4.0.0/16.
3. Echo-Request packet is evaluated against Prod TGW route table and forwarded to the Staging VPC. Then it is evaluated against VPC route table of the subnet of Transit Gateway attachment and NACL of the connectivity subnet.
4. Packet is forwarded from TGW attachment ENI to instance-4(10.4.10.10) after the NACL and Security Groups are evaluated. For the return traffic (Echo-Reply), source IP is 10.4.10.10 and destination IP is 10.2.10.10
5. Echo-Reply packet is evaluated against VPC route table associated to the subnet that has instance-4 and VPC route table has route for the network 10.0.0.0/8 points to the Transit Gateway attachment. Packet is forwarded to the Transit gateway attachment.
6. The Transit Gateway receives the traffic on the Staging Transit gateway route table since the Transit Gateway attachment (TGW Attachment-5) is associated to the Staging Trnasit Gateway route table. Staging Transit Gateway route table does not have route for the network 10.2.0.0/16
7. Echo-Reply packet is evaluated against the Staging Transit Gateway route table and it does not  have route for the destination network 10.2.0.0/16. The packet is not forwarded to Pre-Prod VPC.

Instance-2 is not able to receive echo-reply packet from Instance-4 since Staging TGW route table does not have route for Instance-2 network.

To have successful communication between the Transit Gateway route table, we must propagate routes on both Prod and Staging Transit Gateway route table.

Scenario 3: On-premises server needs to access resources in Dev AWS account.

S2S VPN connection is configured as an attachment on the TGW (TGW attachment-4) and it  is associated to Network Service route table. VPC in Network Service Account was also associated (TGW attachment-3) to the Network service route table and it propagated VPC CIDR block(10.3.0.0/16) in Network Service TGW route table.

VPN tunnel is up and Border Gateway Protocol (BGP) session is established.  BGP neighbors exchange routes via update messages. TGW advertised the network 10.3.0.0/16 to on-premises and received the network 172.16.0.0/16 via BGP update messages. Network Service route table has two propagated routes 10.3.0.0/16 (VPC attachment) and 172.16.0.0/16 (S2S VPN attachment).

Note: TGW advertised only 10.3.0.0/16 to on-premises, since that was only route propagated in the Network service Transit Gateway route table.

If you are trying to SSH to the instance-5 (Dev) from the on-premises server 172.16.10.10. Below is the packet flow for first two steps of TCP 3-Way handshake between the instances.

Figure 4. Packet flow between on-premises and Dev account

On the Network Service TGW route table, you must create propagation and choose Transit Gateway attachment (TGW Attachment-6) and the route 10.5.0.0/16 propagated in Network service Transit Gateway route table. BGP neighbour advertised the network 10.5.0.0/16 to on-premises via BGP update messages.

On the Staging TGW Route table, you must create propagation and choose TGW attachment (TGW Attachment-4) and the route 172.16.0.0/16 is propagated in Staging TGW route table.

You must update the VPC route table in Dev account.  VPC Console → Route Tables → Select the route table belonging to the VPC and add the network 172.16.0.0/16 with Target as TGW.

Below is the packet flow from on-premises workstation to instance-5 in Dev account. AWS Security group and NACL are allowed on the instance-5 in Dev Account and necessary TCP port is allowed on the on-premise router/firewall to communicate each other. Source IP address server is 172.16.10.10 and destination IP of Instance-5 is 10.5.10.10

1. SYN packet validated against On-premise (router/Firewall)  route table and it has received the route for 10.5.0.0/16 via BGP.  Packet is forwarded to TGW attachment over S2S tunnel.
2. TGW receives the traffic on the Network Service TGW route table since TGW attachment (TGW Attachment-4) is associated to the Network Service TGW route table. Network Service TGW route table has route for 10.5.0.0/16
3. SYN packet is evaluated against Network Service TGW route table and forwarded to Dev VPC. Then it is evaluated against VPC route table of the subnet of TGW attachment and NACL of the connectivity subnet.
4. Packet is forwarded from TGW attachment ENI  to instance-5(10.5.10.10) after the NACL and Security groups are evaluated. For the return traffic, source IP is 10.5.10.10 and destination IP is 172.16.10.10
5. SYN/ACK packet is evaluated against VPC route table associated to the subnet that  has instance-5 and VPC route table has route for the network 172.16.0.0/16 point to the Transit Gateway Transit Gateway attachment. Packet is forwarded to Transit Gateway attachment.
6. Transit gateway receives the traffic on the Staging TGW route table since TGW attachment (TGW Attachment-6) is associated to the Staging TGW route table. Staging Transit Gateway route table has route for the network 172.16.0.0/16.
7. SYN/ACK packet is forwarded to on-premises router/firewall over S2S VPN tunnel. Router/Firewall forwards the packet to the workstation.

SYN/ACK packet returned successfully.

AWS Transit Gateway also has an option to add blackhole static route to the TGW route table. It prevents the source attachment from reaching a specific route. If the static route matches the CIDR of a propagated route, the static route will be preferred than the propagated route.

Conclusion

In this post, you learnt the packet flow if both source and destination network are associated to the same or different TGW route table. AWS Transit Gateway simplifies network architecture, reduces operational overhead, and centrally manages external connectivity. If you have a substantial number of VPCs, it makes it easier for the network team to manage access for a growing environment.

For further reading, review the AWS Transit Gateway User guide.

If you have feedback about this post, submit comments in the Comments section below.

Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.