Networking & Content Delivery

Introducing AWS Direct Connect SiteLink

SiteLink, a new feature of AWS Direct Connect (DX), makes it easy to send data from one Direct Connect location to another, bypassing AWS Regions. If you recall, Direct Connect is a cloud service that links your network to AWS, bypassing the internet to deliver more consistent, lower-latency performance. Prior to SiteLink, it was not possible to route traffic directly between Direct Connect locations. Now, you can create global, reliable, and pay-as-you-go connections between the offices and data centers in your global network by sending data over the fastest path between AWS Direct Connect locations.

When using SiteLink, you first connect your on-premises networks to AWS at any of over 100 Direct Connect locations worldwide. Then, you create Virtual Interfaces (VIFs) on those connections and enable SiteLink. Once all VIFs are attached to the same Direct Connect gateway (DXGW), which is a global and highly available AWS resource, you can start sending data between them. Your data follows the shortest path between AWS Direct Connect locations to its destination, using the fast, secure, and reliable AWS global network. You don’t need to have any resources in any AWS Region to use SiteLink. SiteLink is available at all AWS Direct Connect locations today, except for those used by GovCloud or those in China. If you are using Direct Connect now, either through a direct or hosted connection, you have everything you need to use SiteLink. No new connections are required. Please refer to the Direct Connect SiteLink pricing for our most current pricing information.

The following diagram compares what’s possible before and after the availability of SiteLink (Figure 1).

Figure 1—Before and after SiteLink

You might be wondering how SiteLink relates to AWS Cloud WAN. Cloud WAN can create and manage networks of VPCs across multiple Regions. It does this by defining routing policies, monitor network health, and so on. Cloud WAN networks run inside and between AWS Regions. SiteLink, on the other hand, connects DX locations together, bypassing AWS Regions to improve performance. Depending on your use case, you might choose one or the other, or both.

Getting started with AWS Direct Connect SiteLink

Once you are using Direct Connect, you enable (or disable) the SiteLink feature with a simple configuration change using the AWS Command Line Interface (CLI), API, or AWS Management Console (Console). You can see this in the screenshots that follow (Figure 2). You make this change on an existing or a new Transit or Private virtual interface (VIF) (regular or hosted VIF) attached to a DXGW. The requirements for creating and configuring a SiteLink-enabled VIF are the same as a regular DX Private or Transit VIF. For more information, the create a virtual interface page in our documentation is helpful.

Figure 2—Enabling Direct Connect SiteLink on new or existing VIF’s

You can use any combination of dedicated or hosted direct connect connections with different port speeds to enable SiteLink. As shown in the diagram that follows (Figure 3), Single VIF over a DX connection is used to send and receive IPv4 and/or IPv6 traffic from other on-premises locations and from resources in AWS Regions. Because there is no managed QoS support on DX connections, you must carefully plan port speeds for each DX connection to avoid oversubscription. AWS preserves DSCP markings on traffic forwarded between your routers that are connected using SiteLink enabled VIFs, but the AWS backbone does not enforce any QoS based on these DSCP markings.

Figure 3—Full mesh on-premises connectivity with AWS Region access via Private or Transit VIF

SiteLink enables global full mesh connectivity between all DX locations that have a VIF attached to the same DXGW. Adding a new on-premises location to the global full mesh is easy. In the new on-premises location, if a DX connection and VIF to the same DXGW exist, you enable SiteLink with no configuration changes to your routers. If not, you must create a new DX connection and/or a SiteLink enabled VIF over the connection and associate it with the same DXGW.

If you don’t want to connect specific locations to your private global network built using SiteLink, you can choose not to enable SiteLink on the respective VIFs. As shown in the following diagram (Figure 4), four data centers are attached to the same DXGW using a Transit VIF. But, we have only enabled SiteLink on the VIFs associated with data centers 1, 2, and 3. These three data centers will now establish full-mesh, on-premises, connectivity and AWS Region access, while the fourth data center can only access resources in AWS Regions.

Figure 4—Partial Mesh topology

Routing with Direct Connect SiteLink

Routing configuration for a VIF remains the same irrespective of whether SiteLink is enabled or disabled. When you create a VIF (with or without SiteLink enabled), your router establishes an external Border Gateway Protocol (eBGP) neighbor peering relationship with the AWS Direct Connect logical device (ALD). The ALDs’ form part of the control and data plane of DXGW. We can identify the ALD using the AWS CLI and AWS Management Console, as shown in the following screenshot (Figure 5).

Figure 5—Identifying AWS logical device ID using the Console

With SiteLink, the DXGW learns BGP IPv4/IPv6 Prefixes from your routers over SiteLink enabled VIFs, runs BGP best path algorithm, updates attributes such as NextHop and AS_Path and re-advertises these BGP Prefixes to the rest of your SiteLink-enabled VIFs. If you disable SiteLink on a VIF, the DXGW will not advertise the learned on-premises prefixes over this VIF to the other SiteLink-enabled VIFs. The on-premises prefixes from a SiteLink disabled VIF is only advertised to the DXGW Gateway associations (i.e. AWS Virtual Private Gateways (VGW) or AWS Transit Gateways (TGW) associated with the DXGW).

When a DXGW sends a BGP route advertisement to your router, it prepends its own Autonomous System Number (ASN) to the BGP AS Path attribute for each ALD in the path. Depending on your setup, your router will see the AWS DXGW ASN prepended once or twice in the prefixes received from DXGW. We show this in the following diagram (Figure 6). On the left, it shows your routers must traverse a single ALD to reach each other and hence the BGP route advertisements you receive from the DXGW contains a single ASN prepended to the BGP AS Path attribute. But on the right, since there are two ALD’s in the path between your routers, the BGP route advertisements from the DXGW contains two ASN’s prepended to the BGP AS Path attribute. After receiving these advertisements from the DXGW, your router updates its routing table after executing BGP best path algorithm allowing forwarding of IPv4/IPv6 traffic between SiteLink enabled on-premises locations using the shortest path between AWS Direct Connect locations over the AWS backbone.

Figure 6—BGP route advertisements over SiteLink enabled VIF’s

Customer BGP Autonomous System Number (ASN) consideration

If your router peers with DXGW using the same BGP ASN at all locations, then the BGP advertisements received from DXGW will contain that ASN. Your routers will most likely discard these BGP prefixes after seeing its own ASN in the AS-Path as a BGP loop prevention mechanism. This will break the connectivity between your locations from a routing perspective. You can overcome this situation by either using unique BGP ASN at each site or configuring BGP knobs such as allow-as in or local-as no-prepend replace-as on your customer router’s as shown in the following diagram (Figure 7).

Figure 7 – Autonomous System Number (ASN) configuration scenarios

Note: DXGW propagates transitive BGP attributes such as Atomic Aggregate attribute, Aggregator attribute, and standard BGP communities except 7224:* to all other SiteLink enabled VIFs. This helps you implement custom routing policies.

Influencing Traffic Path when leveraging AWS Direct Connect SiteLink with existing Private Corporate Backbone

You might have an existing private backbone that connects your data centers. If that is the case, as you adopt SiteLink, you can continue using your existing private backbone as primary, and configure AWS Direct Connect SiteLink as a backup or vice versa. Let’s look at the example setup shown in the following diagram (Figure 8). In this scenario, you have two data centers connected using a third-party private corporate backbone. You have also enabled SiteLink on your DX Private VIFs, which has enabled connectivity between your data centers. In this setup, we have configured the SiteLink path as a back-up. If you want to do the same, you can use an egress BGP neighbor policy to attach standard BGP community, and an ingress BGP neighbor policy to change BGP local preference based on the received standard BGP community attached to the BGP Prefix. In our example, we use an egress BGP neighbor policy to mark prefixes advertised to private corporate backbone with blue BGP standard community and the same prefixes advertised to DXGW with green BGP standard community. Please note that actual value of the blue/green BGP standard community can be any supported value except 7224:*, and any values that have been reserved by your Private Corporate Backbone.  We set an ingress BGP neighbor policy to set higher local preference value for prefixes received with blue BGP standard community and a lower local preference value for prefixes with green BGP standard community. This makes all data centers prefer the private corporate backbone.

Figure 8—Using AWS Direct Connect SiteLink as a backup to Corporate Private Backbone

This strategy can help shift traffic between backbones by using the same BGP policy structure and just changing the local preference in the ingress BGP policy. We can achieve the same outcome by advertising more specific prefixes from the primary path and/or by using BGP as-path prepend towards the backup path.

Network Segmentation with Direct Connect SiteLink

You can build network segments inside AWS and attach these to separate Virtual routing and forwarding (VRF) contexts on your customer routers to build your desired network topology and isolate traffic between different trust boundaries. You can achieve network segmentation with SiteLink by creating multiple DXGWs and using each DXGW as a trust boundary. Over a direct connect connection, you will create a private VIF to each DXGW. You extend this network segmentation into the AWS regions by using separate VPC’s that map to a trust boundary. There are several use cases for segmentation.

Environment based Network Segmentation

A common use case is to isolate networks based on environments. Most common environments are production and non-production (development, test, staging… etc.). As an example, in the following diagram (Figure 9), we separate production and non-production traffic between your data centers and extend this segmentation into isolated VPCs in an AWS Region.

Figure 9—Environment based segmentation

To build this setup, you need two separate DXGWs. DXGW1 provides a “Production Segment” connectivity, and DXGW2 provides connectivity for the “Non-Production” Segment as shown in the following diagram (Figure 10).

Figure 10—Environment based segmentation using two Direct Connect gateways

Separating Customer and Partner Networks

Another segmentation strategy is to isolate connectivity to third-party networks from your core network setup. You can isolate connections to partner networks, new acquisitions, or temporary connectivity with customers. In the example that follows (Figure 11) we created a separate isolated network for partners and a separate global private network to interconnect customer data centers. The partners will access your application services through Data Center 3.

Figure 11—Segmentation when connecting to 3rd party networks

To get started, you create a separate DXGW per partner in an AWS account. You share this account number with your partners, allowing them to create a hosted VIF to your account. You accept and associate the partner VIFs to separate DXGWs. You also provision separate private VIF per partner on Direct Connect connection at Customer DC3 and connect it to partner specific DXGW. We show this in the following diagram (Figure 12).

Figure 12—Segmenting connectivity to partner networks with two Direct Connect Gateways

Please note that in this case, the account that accepts the hosted-VIF is the one that is charged. Please refer to the pricing page for most current pricing information of AWS Direct Connect SiteLink.

Using AWS Direct Connect SiteLink alongside inter-Region Transit Gateway peering-based networks

You may have built your global network using AWS Transit Gateway (TGW) inter-Region peering, as shown in the following diagram (Figure 13). Consider simplifying your network by choosing AWS Direct Connect SiteLink. Here, you continue to use inter-Region TGW peering for AWS cross-Region connectivity, and use DXGW along with SiteLink-enabled VIFs for connecting your data centers without transiting AWS Region. In this use case, SiteLink offers reduced latency, end-to-end dynamic routing, improved route quotas, and overall simplification compared to TGW based architecture. However, we recommend a TGW based architecture if you must use compute resources inside AWS Regions for use cases such as transcoding, Network/Port Address Translation, and traffic inspection/policy enforcement for data traffic between your data centers.

Figure 13—Migrating from TGW based architecture to SiteLink for inter-DC connectivity

Additional considerations

Here are some additional considerations when using SiteLink.

  1. SiteLink cannot be enabled on a Public VIF or a Private VIF attached to a Virtual Private Gateway (VGW).
  2. If you have two or more VIFs connected to the same DXGW on the same AWS logical device, and you enable/disable SiteLink on either, you will observe a BGP flap for that specific VIF with no impact on other VIFs. We recommend you make this change during a maintenance window.
  3. SiteLink enabled VIF’s adhere to the same quotas as DX VIF’s. You can find details in the AWS Direct Connect quota entry in our documentation.
  4. The subnet containing Amazon router peer IP and your router peer IP configured on the SiteLink enabled VIF is advertised to all remote SiteLink enabled VIFs by DXGW. We do not count these prefixes towards your service quota.
  5. SiteLink enabled VIF metrics includes traffic to AWS Region and to all remote on-premises locations.
  6. If you are currently using site-to-site VPN or SD-WAN connectivity into AWS, you can continue to do so with no changes to your architecture. SiteLink is an AWS Direct Connect only service and does not have a native VPN termination option.
  7. By default, AWS encrypts all traffic on the AWS global network. If you require additional encryption, MAC security can be used (where available) to add Layer2 encryption to secure traffic between AWS physical devices and your router.
  8. There is no native Multicast support with SiteLink.
  9. If you are using any overlay networks currently, you can also use SiteLink as an underlay for those overlay networks.
  10. Physical setup requirements remain identical to AWS Direct Connect and you are responsible for last mile connectivity between your data center and AWS Direct Connect locations. AWS Direct Connect delivery partners are available to help establish this connectivity.

Conclusion

The Direct Connect SiteLink feature helps you quickly connect your data centers using Direct Connect and the AWS global network. You can use this functionality for many enterprise connectivity use cases, whether it’s a primary network connection, a way to back up your existing private backbone network, or a way to connect new locations as your business grows, even when spanning continents. Given the ease of enabling and disabling SiteLink, you can also use it to create a transient private network for connecting to partners and third-parties for exchanging data when a long-term connection is unnecessary. And, you can use SiteLink, along with multiple DXGWs, for advanced use-cases that require the additional security that comes with network segmentation. Overall, the Direct Connect SiteLink feature makes creating global networks easier by quickly creating low-latency and redundant connections between your data centers worldwide. For more details on how to get started, visit our SiteLink documentation page.

Abdul Rahim

Abdul Rahim is a Senior Cloud Infrastructure Architect – Networking, at AWS in Asia Pacific Japan (APJ) region. He has a Bachelors in Electrical Engineering and MS in Telecommunications. He is passionate about improving customers cloud networking experience.

Sidhartha Chauhan

Sid is a Principal Solutions Architect at AWS based in New York. He joined AWS over 7 years ago and since has been working with enterprises helping them architect and build high-scale modern applications. He holds a Masters Degree in Computer Networking from North Carolina State University.