How AWS unified its routing control plane to improve network availability and performance

Why the routing control plane matters

Every time you make an API call to an AWS service, stream video through Amazon CloudFront, or resolve a domain name through Amazon Route 53, your traffic traverses AWS’s border network. Spanning 39 Regions, 123 Availability Zones, and over 750 Points of Presence across six continents at the time of this blog (see the current counts), this network maintains peering relationships with over 5,000 external networks and handles hundreds of terabits of traffic per second. Most of the time, you never think about this infrastructure because it just works.

Behind that experience is the routing control plane: the system responsible for distributing reachability information so that every device in the network knows which destinations are available, where they are, and how to reach them. Think of it as the network’s equivalent of air traffic control. It continuously collects position reports, determines safe routing, and broadcasts instructions so that every aircraft has a consistent, updated view of the airspace. The routing control plane works the same way, continuously collecting reachability information, determining optimal paths, and distributing routing decisions across the network. When it works well, packets reliably reach their destination through the best available path. When something changes in the network, such as a link going down or a new path becoming available, every device needs to arrive at the same updated view of how to reach each destination. Network engineers call this process convergence. Until it completes, parts of the network can briefly disagree on the best path. Packets can be delayed, misdirected, or lost. For workloads like real-time financial trading, healthcare telemedicine, or live media streams, even a few seconds of this inconsistency can mean failed transactions, interrupted services, or degraded experiences.

Scaling the control plane and what we learned

As the AWS network grew, so did the demands on the routing control plane. The number of routes, peering sessions, and interconnection points grew well beyond what a conventional off-the-shelf routing platform was designed to handle. To keep pace, we evolved the network through successive improvements, each bringing better hardware, software, and new architectural capabilities. A natural consequence of this evolution was that different parts of the network operated independent but interconnected control planes, each making its own routing decisions. Individually, each performed reliably. But keeping them in sync across hundreds of locations and thousands of interconnection points introduced real challenges.

The core challenge was making convergence fast enough to be invisible to the applications running on top of it. When a routing change occurs, every device in the network needs to arrive at the same view of the best path. With independent control planes, each system converged at its own pace. During those brief windows of disagreement, one part of the network might forward traffic along a path that another part considered invalid. Imagine two air traffic controllers briefly disagreeing on airspace assignments, each giving different instructions to the same aircraft. The result can be routing loops, as shown in Figure 1, where packets circulate until they expire, or traffic directed to a path that no longer exists. Customers would experience packet loss or elevated latency not because a link was down, but because the network’s routing state was temporarily inconsistent. The impact of these events was difficult to predict because the independent control planes were globally interconnected.

Figure 1: Routing loops and/or suboptimal forwarding

We recognized that adding more coordination between independent systems would only increase complexity. The more systems that needed to agree, the more overhead was required to keep them in sync. Knowing that simplicity scales, we decided to replace the independent control planes with a single unified system that provides one consistent view of the network to every device, converges faster, and contains the scope of any single change.

A three-part solution

Our approach rests on three architectural foundations:

1. A unidirectional control plane that prevents routing loops by making sure information flows in one direction only, with every part of the network learning routes directly from the source.
2. End-to-end tunneling that delivers traffic to the destination fabric selected by the control plane, insulating customer traffic from transient inconsistencies during convergence.
3. A single unified control plane that extended this architecture across the entire network, replacing multiple independent routing systems with one source of truth.

Simplifying route distribution through unidirectional design

In traditional architectures, routing information propagates hop by hop through bidirectional peering sessions, with each node learning from and advertising to its direct neighbors. This creates transitive dependencies: a node’s view of the network depends on what its neighbors share, which depends on their neighbors’ views. When multiple independent control planes operate this way, these dependencies create opportunities for loops and inconsistencies.

We separated this bidirectional exchange into two distinct, unidirectional roles, as shown in Figure 2:

– Collection: Gathers reachability information from local devices and distributes it to the rest of the network.
– Distribution: Receives routing information from remote locations and advertises it to local devices.

Figure 2: Transmit (TX) and Receive (RX) route reflectors in the network

Critically, each part of the network advertises only routes learned from devices directly connected to it and never re-advertises routes received from other locations. Every device learns routes directly from the source, preventing loops that occur when routing information passes through intermediaries with potentially conflicting policies.

Since each control plane node performs only one role, routing policies are simpler and configurations are easier to understand. This separation also reduces the scope of any single change: a misconfiguration on a collection node affects only how routes are gathered outward, with no risk of altering how inbound routes are applied.

Data plane protection through tunneling

Even with consistent routing decisions at steady state, convergence is not instantaneous. When routing changes propagate, intermediate nodes can temporarily hold stale information from a prior network state and make forwarding decisions based on it. A transient divergence anywhere along the path can send a packet on an unintended route.

We complemented the unidirectional control plane with end-to-end tunneling. Tunneling encapsulates a data packet inside another packet: the inner packet carries the original traffic, and the outer packet carries routing instructions that intermediate nodes follow without examining the contents. With tunneling, traffic follows the path selected by the control plane directly from source to destination, as shown in Figure 3. A routing event that might otherwise cause a brief disruption becomes invisible to customers.

Figure 3: End-to-end tunnels between fabrics

Unifying the control plane

The unidirectional design combined with tunneling provided strong routing consistency. The next step was extending this architecture across the entire network, replacing the independent control planes with a single unified system. This was a multi-year effort: we migrated the control plane of a live, globally distributed network to a fundamentally different architecture while maintaining service availability throughout.

We transitioned thousands of devices across multiple tiers of the global infrastructure: the nodes connecting to the internet, AWS Direct Connect, backbone routers, nodes connecting to Availability Zones spanning hundreds of locations, and edge services including Amazon CloudFront, AWS Global Accelerator, AWS Shield, and Amazon Route 53.

The unified control plane also provided several times more compute capacity for routing decisions compared to the previous implementation, directly contributing to improved convergence. A comprehensive validation system analyzed the effects of configuration changes before applying them to the live network, and real-time monitoring helped ensure immediate detection and resolution of any anomalies.

Results and impact

The unified and simplified control plane removed the operational complexity of managing distinct routing architectures. Engineers now work with a single architectural model, simplifying troubleshooting and reducing operational risk.

The architectural change also yielded significant performance gains. Route convergence time for some of the fabrics improved by up to 96%, meaning the network adapts to changes faster and customers experience consistent connectivity. This architecture prevents routing events that could previously place traffic on suboptimal paths. For customers, this translates to fewer retries, more predictable latency, and higher availability for the applications they build on AWS. Whether millions of viewers are tuning into a live event or a financial platform is processing critical transactions, the infrastructure just works.

Conclusion

Network control plane complexity is one of the most significant challenges in operating large-scale infrastructure. We always prioritize availability, performance, and simplicity at scale. The unified control plane embodies this approach. Today, the entire border network operates on this architecture, and it continues to support growth as AWS adds new Regions, Availability Zones, and Points of Presence. This architecture preserves the performance and reliability customers expect while allowing AWS to support emerging workloads with high network demands, such as machine learning and artificial intelligence. Much like unifying air traffic control so every controller gives consistent instructions, our control plane now keeps the entire border network in agreement. Visit the AWS Global Networking page to learn more about how we are building the most resilient and highly available network in the cloud.

About the authors