Integrating MPLS connectivity to the AWS Cloud

Many Amazon Web Services (AWS) customers look to extend their Multiprotocol Label Switching (MPLS) networks into the cloud. MPLS networks offer reliable and performance-optimized routes for data, making them a preferred choice for enterprise connectivity. AWS Direct Connect further enhances this by providing a dedicated network link from an organization’s on-premises networks to AWS.

This post explores the challenges, technical strategies, traffic flow pattern, and best practices for integrating MPLS networks with the AWS Cloud using AWS Direct Connect.

Challenges faced when connecting on-premises networks to the cloud

• Network reliability and performance – Connecting on-premises networks to cloud services over the public internet can sometimes present challenges. Network congestion, variable latency, and packet loss are factors that might affect the performance of cloud-based applications and services. These elements can influence the consistency and efficiency with which data travels across the network, potentially leading to fluctuations in application performance.
• Scalability – As organizations grow, their data needs become more complex and demanding. Traditional wide-area network (WAN) solutions can be slow to scale and may not offer the flexibility to adapt quickly to changing business needs without significant restructuring, leading to decreased performance and increased cost.
• Security and compliance – Maintaining security and compliance is a critical priority for enterprises, particularly when transmitting sensitive data between on-premises environments and the cloud. Therefore, it is important to implement robust security measures to protect against unauthorized access and ensure compliance with relevant regulations.
• Network management complexity – Managing network connectivity between on-premises and cloud environments can be complex, involving multiple networking technologies and configurations. This complexity can lead to operational inefficiencies and increased risk of configuration errors, impacting network performance and security.

Understanding MPLS and AWS Direct Connect

MPLS is a networking technology used in service provider and enterprise networks to speed up and manage traffic flows across WANs. Unlike traditional IP routing where each packet is treated independently and routes are determined per packet, MPLS establishes predetermined paths for traffic, enabling higher performance and improved management of network traffic. By directing data from one network node to the next based on short path labels rather than long network addresses, MPLS decreases routing overhead and speeds up traffic flow. These labels contain information that directs routers on how to handle the packets without needing to inspect the packet’s original headers deeply. This allows for faster packet processing and can support protocols other than IP. The key features for MPLS networks include:

• Label-based forwarding – Data packets are assigned labels, enabling quicker routing decisions.
• Traffic engineering – MPLS allows for the efficient management of network paths and bandwidth, prioritizing critical data flows.
• Quality of service (QoS) – Supports prioritizing different types of traffic, ensuring performance and reliability.

AWS Direct Connect overview

AWS Direct Connect provides a dedicated network link between your on-premises and AWS network. Since this is a physical link, it does not go over the public internet, offering a more reliable and consistent network experience. It is a globally available service and distributed as a logical construct with AWS Direct Connect gateways. With AWS Direct Connect gateways, you are able to establish connectivity to different Regions from your on-premises. AWS Direct Connect key features include:

• Reduced latency – Direct, private connectivity to the AWS Cloud, which helps with latency issues.
• Increased bandwidth – Offers higher data transfer speeds, improving efficiency. Customers can create connections with varying speeds, depending on the AWS Direct Connect location.
• Enhanced security – Private connectivity enhances data security by avoiding public internet exposure. AWS Direct Connect also offers MAC Security to encrypt data from your corporate data center to the AWS Direct Connect location.

Understanding the integration

Integrating MPLS with AWS Direct Connect offers a robust solution for enterprises looking to streamline their connectivity between on-premises infrastructure and the AWS Cloud. This integration requires a detailed understanding of the architecture and traffic flow. The setup involves multiple components from both an on-premises MPLS network and AWS. Here’s an in-depth look at the architecture and technical flow of data.

Network Architecture

Figure 1: Sample network connectivity architecture between AWS and customer on-premises network over MPLS PE (Provider Edge)

Figure 1 shows a physical and logical colocation topology for multiple data center connectivity to AWS. There are many architectural approaches that you can take to extend your MPLS networks into AWS. In this section, we discuss how to use AWS Direct Connect as an option for MPLS connectivity into AWS.

The integration typically involves you setting up an MPLS network or contracts with an MPLS provider that can offer an extension from your on-premises network to an AWS Direct Connect location (typically a colocation facility). Your MPLS network can be extended to a colocation facility by way of last-mile fiber optics or leased lines that physically connect the MPLS provider equipment at the provider edge (PE) to your enterprise’s edge MPLS router located in the colocation facility as the customer edge (CE) device.

You may use your own customer edge (CE) equipment or MPLS provider’s edge (PE) equipment, or you may work with an AWS Direct Connect partner at a colocation facility to extend a circuit from the MPLS managed network PE to the AWS environment. Depending on the last-mile connectivity requirements, one end of this circuit extends through the MPLS provider’s point of presence (POP) to the PE device. The other end of the circuit terminates in a meet-me room or telecom cage located in an AWS Direct Connect colocation.

Within the colocation facility, AWS provides AWS Direct Connect routers. These are dedicated routers that manage the AWS Direct Connect connections. You can then provision an AWS Direct Connect connection through the AWS Management Console or through an AWS partner. This includes choosing an AWS Direct Connect location close to the MPLS network endpoint. A physical cable connection, or cross-connect, is set up within the colocation data center between your MPLS router (CE) and the AWS Direct Connect router.

The customer equipment connects to AWS Direct Connect gateway over this cross connection. Once the physical circuit is up, an exterior Border Gateway Protocol (eBPG) session is set up between CE or PE and AWS Direct Connect gateway using a transit virtual interface to establish IP data communication and routing between AWS, the CE or PE, and your network. The AWS Direct Connect gateway extends this connectivity to AWS Virtual Private Cloud (Amazon VPC) through a transit gateway. A transit gateway provides connectivity in a hub and spoke model to multiple virtual private clouds (VPCs) in the same Region. Because AWS Direct Connect gateway is a global resource, it can be associated to a transit gateway in other Regions. This integration offers highly reliable connections, minimizing packet loss and downtime, while allowing easily scalable network resources based on business needs without significantly reconfiguring the network architecture.

Traffic flow patterns

Data packets traveling between the on-premises infrastructure and AWS are managed efficiently, with MPLS labels facilitating fast and reliable data transfer. This setup ensures that critical applications hosted on AWS can communicate seamlessly with on-premises systems over the MPLS networks. In order to understand what the traffic flow pattern looks like between AWS and on-premises networks using MPLS, we will walk through two patterns as follows:

1. Flow pattern from a VPC to the on-premises networks over MPLS.
2. Traffic flow pattern from the on-premises network to a VPC over MPLS.

Figure 2: Traffic flow patterns between AWS and the customer on-premises network over MPLS PE

From VPC to on-premises

The following steps describe the data-traffic flow walkthrough of the traffic initiated from VPC A in the AWS US East (N. Virginia) Region towards the HQ network. In this scenario, we are only showing the traffic flow from VPC A towards the HQ network, however, the same flow will be applied to the other VPCs present in figure 2.

• 1a: An Amazon Elastic Compute Cloud (Amazon EC2) instance or another resource in the VPC A in us-east-1 sends a packet destined for HQ network as a standard IP packet with an Ethernet header, IP header, and payload. Traffic from VPC A destined for HQ network first reaches the transit gateway through the VPC attachment for VPC A. This happens because a static route entry for the destination CIDR in the VPC A route table exists that routes the traffic to the transit gateway.
• 1b: Once the packet reaches the transit gateway, route-lookup happens in the transit gateway route table associated with the VPC attachment based on its destination, passing it towards the AWS Direct Connect gateway.
• 1c: The standard IP packet is routed through the AWS Direct Connect gateway. The packet traverses the AWS Direct Connect link, reaching the AWS Direct Connect location and then entering the on-premises network through the customer gateway in the colocation over an eBGP session between the AWS Direct Connect gateway and the customer gateway, which as discussed previously could be CE or the MPLS provider’s PE.
• 1d: The CE router receives the IP packet and assigns an MPLS label, converting it into an MPLS packet based on the forwarding equivalence class (FEC) that corresponds to the next-hop or ultimate destination within the HQ network. Subsequently, the packet is relayed to the PE router, positioned at the boundary of the MPLS provider’s network. This interaction typically occurs through the MPLS POP connectivity, bridging the colocation facility and the MPLS PE router. The choice of protocol for this exchange, whether multiprotocol BGP (MP-BGP) or internal BGP, depends on the specific arrangements and requirements of the MPLS provider and the customer.
• 1e: The packet travels through the provider’s MPLS network. Label switch routers (LSRs) route the packet based on the label, with each LSR swapping the label to forward the packet towards its destination, which is HQ MPLS PE router.
• 1f: The HQ MPLS PE router removes the MPLS label, converting the packet back to a standard IP packet, which is then routed to the final destination within the on-premises network.

From on-premises to VPC

The following steps describe the return traffic flow walkthrough for the response traffic from the HQ network to the VPC A in the US East (N. Virginia) Region.

• 2a: A server in HQ network sends the response packet destined for VPC A as a standard IP packet with an Ethernet header, IP header, and payload. Traffic from HQ network destined for VPC A in US East (N. Virginia) first reaches the HQ PE router. The HQ PE router adds the MPLS label on top of the IP packet, converting it into an MPLS packet.
• 2b: The packet travels through the provider’s MPLS network. LSRs route the packet based on the label, with each LSR swapping the label to forward the packet towards its destination, which is MPLS PE router at the boundary of the MPLS provider’s network.
• 2c: The MPLS PE router would forward the MPLS packet to the CE router in the colocation facility. CE receives the MPLS packet and removes the MPLS label, converting it back to a standard IP packet. This interaction typically occurs through the MPLS POP connectivity, bridging the colocation facility and the MPLS PE router. The choice of protocol for this exchange, whether MP-BGP or internal BGP, depends on the specific arrangements and requirements of the MPLS provider and the customer.
• 2d: The CE sends the packet over the AWS Direct Connect link to AWS through the AWS Direct Connect gateway. This happens over an eBGP session between AWS Direct Connect gateway and CE.
• 2e. Once the packet reaches the AWS Direct Connect gateway, it will pass it onto the destined VPC through the associated transit gateway.
• 2f. The packet is handed off the to the EC2 instance in the VPC by the transit gateway attachment.

Considerations

Here are some things to consider when integrating MPLS connectivity to AWS Direct Connect:

• Review the AWS Whitepaper on hybrid connectivity when beginning to explore for your hybrid cloud architecture.
• To access critical workloads over AWS Direct Connect, it is recommended to follow the AWS Direct Connect Resiliency Recommendations. You can also use the AWS Direct Connect Resiliency Toolkit to get started in building resilient hybrid cloud architectures.
• In a multi-MPLS providers scenario, if business requirements are to prefer one MPLS provider over the other, creating active/passive BGP connections over AWS Direct Connect using BGP best path selection algorithms can help achieve that.
• For visibility into the performance of the hybrid connectivity model we have discussed, check out Amazon CloudWatch Network Monitor.
• Review the service quota documentation for the AWS services being used for the most accurate information.
• When using the MPLS PE as the customer gateway in the colocation facility, the MPLS provider would be responsible for enabling the BGP session between AWS and the MPLS provider’s PE and setting up of the 802.1q VLANS to support the VLANs as needed by the customer.
• When using the CE as the customer gateway in the colocation facility, you would be responsible for the BGP connectivity. With this, you are also able to terminate multiple MPLS networks into the same device.
• There is a cap on the number of routes that can be accepted over an AWS Direct Connect private or transit virtual interface. Additionally, many MPLS providers may not offer route summarization at their PE. Therefore, you might need to install a CE or PE router between the PE and AWS Direct Connect to handle the route summarization. It’s advisable to consult the service quota documentation for detailed information on these limitations.
• If utilizing an MPLS Network only for connectivity between on-premises data centers, consider using AWS Direct Connect SiteLink for routing traffic between the data centers over AWS Direct Connect locations without having to traverse any AWS Region. It can also be considered for having a backup connection with the MPLS network for disaster recovery.

Conclusion

In summary, integrating AWS Direct Connect with MPLS provides a streamlined, secure, and efficient way to connect on-premises infrastructure with AWS services, supporting a wide range of enterprise applications and enhancing overall network management and performance capabilities. For more information, see the AWS Direct Connect documentation to get started on integrating your MPLS networks with AWS Direct Connect.

An update was made on September 12, 2024: An earlier version of this post was missing labels for “VPC A” in figure 1 and figure, while being referenced in the text. The post has been updated with these labels added.

About the authors