VPC Routing Enhancements and GWLB Deployment Patterns

At re:Invent 2020, AWS introduced  Gateway Load Balancer (GWLB), an AWS service that helps you deploy, scale, and manage third-party virtual network appliances, such as firewalls, intrusion detection and prevention systems, and others. GWLB is a type of load balancer under the Elastic Load Balancing (ELB) family. Other load balancers within the ELB family include Application Load Balancer (ALB) and Network Load Balancer (NLB), which are reverse proxies where traffic is routed to. On the other hand, GWLB is a non-proxy construct, as it doesn’t terminate the client connections, and traffic is transparently routed through the GWLB. Using Amazon Virtual Private Cloud (Amazon VPC) routing, traffic from source to destination is routed to GWLB, making GWLB act as a gateway and/or a route target on the frontend. When the traffic arrives at GWLB on the backend, it acts as a load balancer that distributes traffic and scale virtual appliances acting as targets on demand.

Your VPC contains one or more route tables. A route table contains a set of rules, called routes, that determine where network traffic from your subnet or gateway is directed. When the source is trying to reach to the destination, it does a route table look up and, based on the routes configured, routes the traffic to the next hop. Your VPC has an implicit router and each VPC route table, default and/or custom, has a pre-configured local route pointing to this implicit router. Any traffic originating from a source within a VPC with a destination within the same VPC will match this local route, and traffic gets routed.

Until August 2021, you couldn’t add a route that was more specific than the VPC’s Classless Inter-Domain Routing (CIDR). To address this, in August 2021, AWS launched Amazon Virtual Private Cloud (VPC) routing enhancements. VPC routing enhancements make it possible to configure routes more specific than the VPC CIDR at a subnet route table level. Furthermore, this lets you edit the target of the pre-configured local route from “local” (implicit router) to a middle box, such as GWLB Endpoint. Therefore, you can add a virtual network appliance between subnets for inter-subnet traffic inspection.

We’ve published several articles on GWLB since its launch. In this post, we discuss different deployment models made possible through the VPC routing enhancements. In particular, we focus on the following three sections:

1. Internet Ingress traffic inspection: Inbound traffic coming in from the Internet to applications hosted inside of the VPC.
2. Internet Egress traffic inspection: Outbound traffic originated from a resource within your private subnet in a VPC to a resource on the Internet.
3. East-West (inter-subnet) traffic inspection: Traffic originating from a resource in one subnet in the VPC to a resource in another subnet in the same VPC.

For this post, we focus on intra-VPC routing, as VPC routing enhancement only applies to routing within the VPC.

Security Groups (SGs) and Network ACLs (NACLs) and reasons to go beyond them

Security Groups (SGs) and Network ACLs (NACLs) are both mechanisms to control traffic. SGs operate at an elastic network interface (ENI) level regardless of the subnet. SGs are stateful in nature, so responses to inbound traffic can flow out of the ENI regardless of outbound SG rules, and vice versa. NACLs allow or deny specific inbound or outbound traffic at the subnet level. NACLs are stateless, which means that responses to allowed inbound traffic are subject to the rules for outbound traffic and vice versa.

For many use cases, SGs and NACLs satisfy the security requirements. That being said, for some use cases, additional security constructs like network firewall might be necessary. Here are some of the main reasons why additional firewalling could be required:

• Scale: By default, you can have 20 rules per NACL, up to a maximum of 40 rules per NACL. Similarly, by default, you can have 60 rules per Security Group, up to a maximum of 1000 rules per Security Group. If you have a use case that requires going beyond this quota, then you will need an additional control point.

• Deep packet inspection: SGs and NACLs operate at the network (Layer 3) and transport (Layer 4) layers respectively. This means that they can apply controls based on IP addresses, protocol, and ports, but they can’t apply rules on application parameters. If your use case requires packet inspection at an application layer (Layer 7), then you must use additional security constructs such as AWS Network Firewall and/or AWS Web Application Firewall (AWS WAF).

• Centralized management: SGs/NACLs/Firewalls provide mechanisms to provide Network Security for your deployment. To centrally manage and audit your SGs, or your AWS Network Firewall’s rules across multiple accounts, you can use AWS Firewall Manager Service. Alternatively, you can use a third-party firewall that brings its mechanism to centrally manage the firewall rules and policies across your deployment.

Internet Ingress traffic inspection

You must carefully design network architectures to inspect and protect Internet-facing applications. The blog, Design your firewall deployment for Internet ingress traffic flows, discusses in detail various deployment models to inspect and protect Internet-facing applications. In this section, we focus on how GWLB, combined with VPC routing enhancements, lets you inspect distributed ingress architecture. In the distributed ingress architecture, each VPC has its own path to and from the Internet through a dedicated Internet Gateway (IGW). Setup can be replicated across multiple VPCs, as the data path remains the same. The distributed nature limits the blast radius and makes troubleshooting easier.

ALB and/or NLB play an important role in architecting these Internet-facing applications. Before VPC routing enhancements, you could use gateway route tables and insert an inline firewall between the IGW and your load balancer. VPC routing enhancement combined with GWLB lets you inspect the traffic between ALB/NLB and their targets. A device such as a firewall can now be inserted seamlessly between ALB/NLB and its targets by adding a more specific route for the application/backend-targets’ subnet with target as Gateway Load Balancer Endpoint (GWLBE). We show this deployment model in Figure 1. The benefits of inserting an inline firewall appliance between the ALB/NLB and its targets are:

• The firewall appliances will only see the traffic that is allowed by the ALB/NLB. Any traffic that doesn’t match the ALB’s listener/NLB’s listener setting will be dropped at the ALB/NLB level. This makes sure that the firewall appliances’ resources aren’t spent inspecting unwanted traffic.
• For encrypted traffic, the firewalls don’t need to host any public certificates. This is especially beneficial if you’re using AWS Certificate Manager (ACM) for generating the public certificates, as certificates can’t be exported to non-AWS resources, such as a third-party firewall. You can continue to use the public certificates generated by ACM with ALB/NLBs.

With the deployment model shown in Figure 1, the firewalls won’t see the original IP address of the clients. The firewalls will only see the ALB’s source IP address.

Although we show an ALB-based example in Figure 1, note that the guidance provided applies even if you host your applications behind an NLB.

Figure 1: Internet Ingress traffic inspection – Distributed Architecture

GWLBEs are zonal in nature. Therefore, the GWLBE that processed the originating traffic should also process the return traffic. As shown in Figure 1, with cross-zone load balancing enabled:

• (A) When traffic lands on ALB, it can land on an ALB node residing in either of the Public Subnets. By default, ALB does cross-zone load balancing, which means that ALB can route traffic to targets in either of the application subnets. For example, the ALB node in Public Subnet 2 in Availability Zone AZ-B, can route traffic to an EC2 instance in Application Subnet 1 in AZ-A.
• (B) Public Route Tables associated with Public Subnets where ALB is hosted have more specific routes for each Application Subnet with targets as the respective GWLBE in the respective AZ.
• (C) GWLBE routes traffic to GWLB which then distributes traffic to firewalls (targets). Firewall is a stateful construct. The firewall node that processes the originating traffic should also process the return traffic, otherwise the return traffic is dropped. This scenario is known as asymmetric routing.
• (D) Application Route Tables associated with Application Subnets have more specific routes for each Public Subnet with targets as the respective GWLBE in the respective AZ. (B) and (D) make sure that the traffic is symmetrically routed in both directions to the same GWLBE, which is a requirement for inline firewalls.
• (E) ALB supports HTTPS listeners, and you can use ACM to associate a certificate with the ALB. For the flow depicted in the following figure, the SSL session will get terminated on the ALB.
• (F) As an additional security measure, you can attach AWS WAF to the ALB. This lets you secure web applications against layer-7 exploits. This is a design best practice, as the WAF blocks malicious layer-7 traffic before the traffic hits the inline firewalls.

If you have compliance requirements that mandate pushing all internet inbound traffic through a firewall even before traffic hits the ALB, then you can place the GWLB endpoints between the IGW and the ALB’s subnets. In this scenario, you must upload the correct certificates on the firewalls, as the firewalls will perform the necessary SSL termination. Note that this design option doesn’t utilize VPC more specific routing enhancement. You can influence the path of Internet inbound traffic by using Gateway Routing tables associated with the IGW.

Considerations for Internet ingress use case

The following considerations are useful for the Internet ingress use case:

• Traffic between the NLB and backends of the instance target type don’t follow VPC route table routes. If you want to inspect traffic between the NLB and targets, then use the IP target type.
• Client IP Preservation:
  • (i) When using ALB, the firewalls will see the ALB’s IP address as the traffic source. You can use the X-forwarded-for header in case the firewalls must see the original source IP address.
  • (ii) When using NLB along with GWLB, client IP preservation isn’t supported. The firewalls will always see the NLB’s IP address as the source IP.

Internet Egress traffic inspection

You have resources configured in a private subnet that must communicate with the Internet for 1/downloading a security patch, 2/software management, and 3/accessing public API endpoints or some other reasons. A NAT gateway lets you achieve that. A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that resources in a private subnet can connect to services outside of your VPC, but external services can’t initiate a connection with those resources.

Similar to Internet Ingress, you must carefully design network architectures to inspect and protect traffic originating from the private subnet within your VPC to services outside of your VPC. When talking to customers, we learn that they have a need to preserve the original source IP address. If you place the inline firewall between the NAT gateway and the IGW, the firewalls would always see the NAT gateway’s IP address as the source of all traffic.

The VPC routing enhancements feature lets you insert a security appliance between a private resource and the NAT gateway. This lets the security appliance see the original source IP address versus the IP address of the NAT gateway. We show this deployment model in Figure 2.

Figure 2: Internet Egress Traffic Inspection – Distributed Architecture

• (A) Application Route Tables associated with Application Subnets have a default route pointing to GWLBE in the respective AZ. When the source sends the traffic, it does a route table lookup and routes the traffic to GWLBE in the respective AZ.
• (B) GWLBE routes traffic to GWLB which then distributes traffic to firewalls (targets).
• (C) GWLBE Route Tables associated with GWLBE Subnets has a default route pointing to the NAT gateway in the respective AZ.
• (D) Public Route Tables associated with Public Subnets has a default route pointing to IGW.
• (E) To avoid asymmetrical routing, the return traffic must be routed to the same firewall node that processed the originating traffic. This is made possible by editing the Public Subnets’ Route Table’s pre-configured local route’s target from local to GWLBE in the respective AZ.

Similar to distributing the ingress architecture, in the distributed egress architecture, each VPC has its own path to and from the Internet through a dedicated NAT gateway and IGW. Setup can be replicated across multiple VPCs, as the data path remains the same. The distributed nature limits the blast radius and makes troubleshooting easier.

East-West (inter-subnet) traffic inspection

The other use case that we come across is inspecting traffic between resources residing in two different subnets of the same VPC. Either resource can be the source and/or destination of the flow. Similar to Internet Ingress and Internet Egress, the concept remains the same. Using VPC routing enhancement, we insert a security appliance between two subnets, which allows the traffic inspection. Since either of the resources can originate the traffic (act as a source), one must carefully design the network architecture to avoid asymmetrical routing.

Within the same AZ

When two subnets are in the same AZ, you edit the pre-configured local route’s target from ‘local’ to AZ-specific GWLBE in the route table associated with these subnets. This makes sure that the originating traffic and return traffic are processed by the same GWLBE. We show this setup in Figure 3.

Figure 3: Internet Subnet Traffic Inspection – Same AZ

Across different AZs

Things become interesting when these subnets are spread across multiple AZs. Since GWLBEs are zonal in nature, you must pick an endpoint in an AZ to inspect the inter-AZ traffic. This makes sure that the traffic is symmetrically routed in both directions to the same GWLB endpoint, which is required for inline firewalls. We show this setup Figure 4. For AZ-A and AZ-B, we select GWLBE-A to process the traffic in either direction. Similarly for AZ-B and AZ-C, we select GWLBE-B. For AZ-C and AZ-A, we select GWLBE-C. This design lets us reduce the blast radius to the affected AZ. For example, if AZ-A experiences an issue, then traffic between AZ-B and AZ-C continues to operate normally.

Figure 4: Internet Subnet Traffic Inspection – Same AZ

Shared VPC

To inspect traffic between two applications hosted in two different VPCs, we recommend deploying a third-party firewall in a centralized deployment model where GWLB, GWLB endpoints, and third-party firewalls are deployed in a centralized inspection VPC. Application traffic that must be inspected is routed to a centralized inspection VPC using an AWS Transit Gateway. For many use cases, the third-party firewall deployed in a centralized model satisfies the requirements. But for the use case where you’re using VPC sharing for hosting different applications, you can use VPC routing enhancements to inspect the traffic between two applications hosted in two different subnets in the same VPC.

VPC sharing lets you share subnets with another AWS account within the same AWS Organization. VPC sharing is a powerful concept, and it comes with its own share of benefits and considerations. For more details, refer to documentation or this post that discusses considerations for VPC peering. In this section, we focus on the VPC routing enhancement feature to inspect the application traffic between two subnets and related considerations.

When using VPC routing enhancement with VPC sharing, it follows the same concept as the one described above in the East-West (inter-subnet) traffic inspection section. We show this in Figure 5. That being said, you must carefully design the network architecture. As you start scaling by adding multiple subnets, note the following considerations.

• (A) Managing route tables configuration: The routing tables can become tough to manage as you create more subnets within the VPC. This is because when adding routes that are more specific than the VPC’s CIDR, the destination must match a subnets’ CIDR. The following figure shows a two AZ deployment, with the requirement that all inter-subnet traffic must be inspected by inline firewalls. The routing tables shown in Figure 5 provide this routing structure:
  1. Traffic between subnets within AZ-A is sent to GWLB endpoint in AZ-A
  2. Traffic between subnets within AZ-B is sent to GWLB endpoint in AZ-B
  3. All Inter-AZ traffic is sent to GWLB endpoint in AZ-A

Figure 5: VPC Sharing – Inter-subnet inspection with three subnets per-AZ

In case your deployment grows beyond two AZs, you should keep tight control on the routing table configuration and make sure that traffic is always symmetrically routed to the same GWLBE.

You can alleviate the routing table management by separating out your workloads into different VPCs. After putting workloads in multiple VPCs, see this post on how to insert firewalls with GWLB between multiple VPCs.

(B) Cross-zone data transfer charges: In Figure 5, consider the traffic flow from Database Subnet 1 going to Application Subnet 1. Even though this traffic originates in AZ-B, the routing table configuration sends all of the data to the GWLBE in AZ-A. This traffic accrues cross-AZ data transfer charges.

Considerations

Now that we’ve covered various deploy patterns made possible by VPC routing enhancement, let’s review some of the considerations.

• SGs and NACLs are great ways to inspect and protect the traffic between resources residing in different subnets. Wherever possible, implement these features and use inter-subnet inspection only if necessary.
• When a third-party virtual appliance is inserted in the path using VPC routing enhancement, SG referencing can’t be used to allow the source to connect to your destination. You can use IP addresses of your source in the SGs to permit the incoming traffic.
• As mentioned in the “Internet Ingress traffic inspection” section, traffic between the NLB and backends of the instance target type doesn’t follow VPC route table routes. If you want to inspect traffic between the NLB and backends, then use the IP target type.
• When adding routes more specific than VPC CIDR, each route destination must match an existing subnet CIDR.
• GWLBE requires a dedicated subnet.
• GWLBE is a zonal construct. Carefully design you network architecture to avoid any kind of asymmetrical routing.
• The route target can be the ENI attached to an appliance that you built or acquired, a GWLBE to distribute traffic to multiple appliances for performance or high availability reasons, an AWS Network Firewall, or a NAT gateway. The route target can’t be an Transit Gateway and/or AWS Cloud WAN Core Network.
• You can’t inspect traffic coming from VPC peering.
• Deploying a large number of firewall endpoints in a distributed model and across many VPCs could be costly.
• For Network firewall, refer to the Deployment models for AWS Network Firewall with VPC routing enhancements post.

Code samples:

We have created a GitHub repository for code examples that can help accelerate your development of AWS Gateway Load Balancer. The repository has samples for AWS CloudFormation, Python (Boto3), Go, and the CLI. Visit this page to launch the deployment patterns describe in this post using AWS CloudFormation.

Conclusion

GWLB is an AWS service that helps you deploy, scale, and manage third-party virtual network appliances, such as firewalls, intrusion detection and prevention systems, and others. Prior to VPC routing enhancement, to inspect traffic between resources, you had to place those resources in different VPCs. With VPC routing enhancement, you can now insert a network virtual appliance between two subnets in the same VPC. This lets you inspect the traffic between resources residing in different subnets. In this post, we discussed various security deployment patterns made possible by VPC routing enhancement and considerations regarding them.

About the authors