Achieve multipath load balancing in Amazon VPC using Virtual IPs

Many Telco use cases use Equal Cost Multipath (ECMP) for traffic distribution, enhancing high availability (HA), resiliency, and fast failover. AWS Direct Connect and AWS Transit Gateways currently support ECMP (Equal-Cost Multi-Path routing) specifically for connections from on-premises environments to the AWS cloud. However, within an Amazon VPC (Virtual Private Cloud), only a single destination can be specified for a target IP address. As a result, to achieve fast failover for your workloads, you need to implement an active/standby approach. While active/standby approach provides redundancy, it is not horizontally scalable and lacks load balancing capabilities. Moreover, the throughput capacity of a single Amazon Elastic Compute Cloud (Amazon EC2) node can become a bottleneck under high traffic conditions.

This post presents a solution to achieve multipath load balancing using Virtual IPs (VIPs), subnet route tables, and Transit Gateways. The solution achieves load balancing by routing the virtual IP (VIP) address to multiple Elastic Network Interface (ENI) targets from different route tables in the VPC. By leveraging multiple route tables to distribute the VIP traffic, the solution is able to spread the workload across the various target resources. This approach allows for a more balanced distribution of the incoming requests, rather than directing the traffic on a single set of resources. Although this solution doesn’t replicate ECMP functionality, it does offer similar load balancing capabilities, such as HA, faster failover, and scalability to overcome the single Amazon EC2 throughput constraints.

Background

Virtual IP refers to a private IP address outside the CIDR range of the VPCs. A virtual IP (VIP) is a logical IP address that is not directly associated with an ENI. Instead, it serves as a routing abstraction to represent an ENI in the VPC.

Within a VPC, each subnet must reside entirely within one Availability Zone (AZ) and can’t span AZs, whereas route tables provide routing to the AZs within that VPC. Each subnet is associated with a single route table, so you can associate multiple subnets with the single route table. Similarly, you can create multiple route tables and associate selective subnets with these route tables. Each route within a route table delineates a destination and a target.

As a packet exits an ENI, it selects the most specific route matching the destination (based on the longest prefix match) in its subnet route table, using a defined destination as the next hop. The VPC manages routing for VPC CIDRs (Classless Inter-Domain Routing), where the destination is considered “Local” (within the VPC) . However, for destinations based on non-VPC IP addresses, you have the option to specify the target as an ENI within the VPC.

For Transit gateway connectivity, you can achieve numerous paths, equivalent to the number of AZs for a non-VPC VIP. For example, in an AWS Region with three AZs, you can realize three distinct destinations for the same destination VIP. This is possible because Transit Gateway attaches to each AZ with an ENI from a subnet in the underlying AZ. Each Transit Gateway attachment subnet is associated to different subnet route tables. Each one potentially contains different destinations for the ingress VIP, thus offering multiple paths. You can also associate more than one subnet with each subnet route table.

Amazon EC2 Auto Scaling groups can span across multiple availability zones (AZs) within the same AWS Region. This feature enables launching the workload in multiple subnets spanning different AZs, enhancing both availability and resiliency. Furthermore, if necessary, you can configure the Auto Scaling group to use multiple subnets within the same AZ.

Solution

You can use non-VPC IP addresses (private IP addresses outside VPC CIDR) as VIPs for your workloads. This VIP would act as a single ingress IP address and can have multiple workloads (Amazon EC2 virtual machines (VMs) or Kubernetes Pods) behind it. Furthermore, use different VPC route tables to route the ingress VIP to different ENIs on the VMs as destinations. There are two methods to achieve this:

1. For Border Gateway Protocol (BGP)-enabled workloads, you can peer them with a BGP utility that can programmatically provision different next hops (target ENIs) for the same destination VIP across different route tables in the VPC.
2. For non-BGP workloads, you can use health checks or event-based programming using AWS Lambda to programmatically provision different next hops (target ENIs) for the same destination VIP across different route tables in the VPC.

Note that you need to disable source/destination check on the workload network interfaces to allow the non-VPC IP traffic.

In the following sections, I explore four common network deployment and connectivity patterns to achieve multipath load balancing. In these examples, I use three route table groups (Green, Blue, and Yellow) to establish three paths within the VPC. Your workload gains three different paths by associating its subnets with these route table groups. For Transit Gateway connectivity, these route tables are organized based on Transit Gateway attachments across three different AZs.

On-premises to Amazon VPC multipath load balancing

Figure 1 On-premises to Amazon VPC multipath load balancing

In the preceding example, two network functions (NF) are illustrated: NF1 and NF2. NF1 is deployed exclusively in AZ1 using a non-VPC VIP (192.168.1.241). On the other hand, NF2 is deployed across the three AZs using a non-VPC VIP (192.168.1.251). For simplicity, the example showcases three EC2 instances across three subnets, although the setup can include more EC2 instances or more subnets. Subnets are represented by a group, Green, Blue, and Yellow, based on their association with Green, Blue, and Yellow subnet route tables. NF1 has eni1 in the Green group, eni2 in Blue, and eni3 in Yellow. Similarly, NF2 has eni4 in Green, eni5 in Blue, and eni6 in Yellow.

The Transit Gateway has an attachment to this VPC for each AZ through an ENI. These attachment ENIs are from subnets in the VPC, with one per AZ. The subnets associate with the green, blue, and yellow route tables, which have different destinations for the VIPs. In the Transit Gateway route table, static routes for the VIPs (192.168.1.241 and 192.168.1.251) direct traffic to the VPC attachment ID. The VIPs are also configured in the AWS Direct Connect gateway allowed prefixes, to allow the on-premises traffic to the Transit Gateway.

Traffic flow:

On-premises traffic to VIPs (192.168.1.241 and 192.168.1.251) reaches the Transit Gateway through Direct Connect Gateway. Since the IPs are not managed by VPC, Transit Gateway uses the ECMP feature and load balances to each AZ through the Transit Gateway attachment ENIs. Based on the AZ attachment, packets egress the respective route table (Blue, Green, and Yellow) and land on the workload ENI in the routing group. The return path follows the same route in reverse and is symmetrical.

Cross-AWS Region traffic multipath load balancing

Figure 2 Cross-AWS Region traffic multipath load balancing

In this use case another workload NF3 is deployed in VPC2 in another AWS Region (region 2). This acts as the source to communicate with NF1 and NF2 in another region. NF3 is deployed in the three AZs for HA. However, it can be a single AZ deployment. NF1 and NF2 are deployed in VPC1 in region1 with 192.168.1.241 and 192.168.1.251 VIP addresses. In the Transit Gateway route table, static routes for the VIPs (192.168.1.241 and 192.168.1.251) direct traffic to the VPC attachment ID. Transit Gateway peering is used between the Transit Gateways to enable cross-region traffic.

Traffic flow:

The egress traffic to VIPs (192.168.1.241 and 192.168.1.251) from NF3 in region2 arrives to Transit Gateway in region2 through its subnet route tables. The Transit Gateway in region2 sends the traffic to the Transit Gateway peering connection. The Transit Gateway peering forwards the VIP traffic to the TGW-VPC1 attachment. Since these VIPs are not managed by VPC, Transit Gateway uses the ECMP feature and load balances to each AZ through the Transit Gateway attachment ENIs. Depending on the AZ, the packet egresses from the respective VPC route table (Blue, Green, or Yellow) and it lands on the respective ENI/EC2 of the worker/EC2 node. The return path follows the same route in reverse and is symmetrical.

Same AWS Region traffic multipath load balancing

Figure 3 Same AWS Region traffic multipath load balancing

In this scenario, the workload network function NF3 (the source) is deployed in the same AWS Region, but in another VPC: VPC2. This NF acts as the source to communicate with NF1 and NF2 in the same AWS Region. NF1 and NF2 are deployed in VPC1 with 192.168.1.241 and 192.168.1.251 VIP addresses. In the Transit Gateway route table, static routes for the VIPs (192.168.1.241 and 192.168.1.251) direct traffic to the VPC attachment ID.

In this pattern, NF3 must be deployed in the three AZs to use multipathing for cross-AWS Region traffic. As discussed earlier, Transit Gateway attaches to an AZ through an ENI in that AZ’s subnet. Since both VPCs are deployed in the same Region and AZs, Transit Gateway is AZ aware, and it tries to route the traffic through the same-AZ attachment in the connected VPCs. Therefore, to achieve multipath load-balancing, make sure that the source and target NFs are deployed in multiple AZs. This allows packets to be sent to the respective AZ attachments in VPC1, thereby the blue, green, and yellow route tables can provide multipathing for the same-Region traffic.

Traffic flow:

If NF3 traffic originates from AZ1 (Green) in VPC2, then it always arrives in AZ1 in VPC1 (Green route table), and reaches to VIP 192.168.1.241 through eni1 and VIP 192.168.1.241 through eni4. Similarly, when traffic originates from AZ2 (Blue), then it reaches to VIP 192.168.1.241 through eni2 (Blue) and VIP 192.168.1.241 through eni5 (Blue). This means that the Blue, Green, and Yellow route tables can provide multipathing for the same-Region traffic. The return path follows the same route in reverse and is symmetrical.

Within VPC multipath load balancing

Figure 4 Within VPC multipath load balancing

In this scenario, the NF3 is deployed in the same VPC1, so Transit Gateway isn’t used. In this diagram, NF3 is deployed across the three AZs for HA. However, you can also deploy NF3 in a single AZ with multiple subnets (for example three subnets) associated with the green, blue, or yellow route tables. NF1 and NF2 are deployed in VPC1 with 192.168.1.241 and 192.168.1.251 VIP addresses.

Traffic flow:

Since the NF3 workload spreads across the Green, Blue, and Yellow route tables, its egress traffic is load distributed from these tables. The respective egress packets follow the static routes defined for the VIP addresses 192.168.1.241 and 192.168.1.251 in the route tables. This allows them to reach the NF1 and NF2 VIPs in a multipath way. The return path follows the same route in reverse and is symmetrical.

Conclusion

This post presented design patterns to achieve multipath load balancing using VIPs, subnet route tables, and Transit Gateways. You can bring in your traffic and perform load balancing on a single ingress VIP, regardless of the layer 4 protocol. This solution allows you to scale across AZs to avoid single-EC2 throughput and management constraints, and it provides HA for AZ failures.

You can extend these patterns to your existing applications. Leave a comment, as we would love to hear your feedback. Reach out to your AWS account teams and Partner Solution Architects to learn more about 5G and Telecommunications on AWS.