Connecting VPCs securely and at scale to 3rd party public services in on-premises networks

There are many things to consider when connecting your Amazon Virtual Private Clouds (Amazon VPCs) to third-party applications running in data centers outside AWS. You need connections that are secure and scalable, especially when the third-party vendor uses a multi-tenant architecture. With multi-tenant services, where they have many of their customers sharing the same system, your sensitive data must remain secure and isolated from other users. To do this, most vendors provide their customers with endpoints that have public IP addresses and add the IP addresses of your Amazon VPCs to an allow list. But what about times when you need private connectivity and can’t use the public internet? This post presents several approaches to integration using public IPs and private connectivity. DoorDash, an AWS customer, achieved this using an architecture approach that allowed them to support financial data reconciliation for their local commerce platform, inspiring this post.

Connecting the DoorDash Local Commerce Platform to third-party vendors

DoorDash needed to integrate their local commerce platform with a third-party vendor’s systems while keeping their data secure at all times. DoorDash is a born-in-the-cloud company and operates its local commerce platform on AWS. This platform powers the ecosystem of consumers placing orders, merchants fulfilling orders, and dashers performing deliveries. They built this platform to achieve a high level of scaling and performance to support their expanding business. DoorDash initially used a single payment processor for fulfilling orders. But it set out to increase redundancy by integrating with multiple payment processors for their order fulfillment workflow. DoorDash used this approach to securely connect to third-party payment platforms successfully. They detail its design and implementation on the DoorDash Engineering blog, in a post titled, “How DoorDash Secures Data Transfer Between Cloud and On-Premise Data Centers.”
After considering this post, we found that many AWS customers must integrate with third-party systems sitting outside AWS. Here, we will look at five different scenarios for accomplishing this integration.

Let’s start by imagining that, similar to DoorDash, you have an application that must connect to a vendor’s system that provides logging and monitoring, exchanges transaction data, processes payments, and performs financial reconciliation. When the vendor’s application sits outside AWS and is multi-tenant, you require that:

• The application must be accessible using public IP addresses.
• The vendor must require their customers to use unique public IP addresses to connect, and that the vendor adds the source IPs to an allow list that controls who can connect to the application.

The third-party vendor has their own security mechanisms, including IDS/IPS and Firewalls, to ensure this communication is secure. However, for an additional layer of security, you require virtual or physical private network connections.

Scenario 1: Connecting with third-party vendors using Private NAT and AWS Site-to-Site VPN

In this scenario, your AWS environment supports applications and micro-services running in three Amazon VPCs (VPC A, VPC B, and VPC C). We show this in the following diagram (figure 1).

Figure 1: A multi-VPC environment connected to a third-party vendor’s on-premises data center using AWS Site-to-site VPN and private NAT Gateway.

We connect the VPCs to an AWS Transit Gateway as spokes. The communication between the application and third-party vendor must happen over a secure private network, and the application needs less than 1.25 Gbps of bandwidth (the limit of AWS Site-to-Site VPN connections). Because the third-party vendor’s data center sits outside AWS, we use an AWS Site-To-Site VPN connection to encrypt traffic between the vendor’s data center and the virtual private gateway of VPC A.

To make sure that communication uses public IPs, this architecture uses VPC A as the entry point for the VPN connection to/from the vendor’s data center. VPC A uses two CIDR blocks: one CIDR block of private IPs for communicating with other VPCs, and another CIDR block of public IPs used for communicating with the public IPs of the vendor’s on-premises network. We achieve this through the configuration of a secondary CIDR block for VPC A.

A subset of the public IP block is assigned to the Private Routable Subnet. This subnet is called the Private Routable Subnet because, even though it has no internet connectivity, it uses public IPs, and it can route traffic to the vendor’s network. Likewise, the vendor can route traffic back to the Private Routable Subnet. Using public IPs makes sure that the IP addresses are unique and the third-party system can effectively allow list IP addresses without the potential for conflicts with IP addresses used by other tenants. This is because the public IP space assigned to the VPC as a secondary CIDR is owned by that customer. An AWS Private NAT Gateway is deployed in this subnet to perform address translation as the application running in that and the other VPCs exchange traffic. Using this method, you can access multiple applications from the third-party vendor at the same time, and there are no restrictions on protocols. For example, you could access a vendor’s application using TCP or UDP. Note that the Private NAT Gateway performs source NATing for traffic moving from AWS to the third-party vendor. It doesn’t allow for connections initiated from the vendor’s network.

Scenario 2: Connecting with third-party vendors using Private NAT and AWS Direct Connect

Let’s look at how to connect the same AWS environment and third-party vendor when you need higher bandwidth, and you can’t send data over the public internet. Here, we use AWS Direct Connect to establish dedicated fiber links to the AWS network (it does this at speeds up to 100 Gbps, with MACsec encryption available). Once your vendor connects their data center to AWS using Direct Connect, you can exchange data with them over this private connection. We show this in the following diagram, figure 2.

Figure 2: A multi-VPC environment connected to a third-party vendor’s on-premises data center using AWS Direct Connect and Private NAT.

In this scenario, the vendor must share their private connection with you. The vendor can do this by:

1. Creating a hosted virtual interface (VIF) on the Direct Connect circuit and sharing it with your AWS account. You accept the hosted VIF and attach it to your Direct Connect Gateway. We show this in figure 2.
2. You perform a cross account association of the virtual private gateway in VPC A with the vendor’s Direct Connect Gateway.

This approach follows the same pattern as scenario one, where you put a Private NAT gateway between the service consumers in your VPCs and the applications residing with the vendor, and the same considerations apply. But because it uses Direct Connect, this approach provides better network performance plus a private network connection.

For scenario one and scenario two, you can access the third-party application using its DNS name either by relying on public DNS resolution or having a hybrid DNS setup with Route 53 Forwarding rules configured to route DNS queries to the on-premises DNS server via the Outbound resolver endpoint.

Scenario 3: Connecting with third-party vendors using AWS PrivateLink

In contrast with scenario one and two, let’s say you have a specific third-party application that is accessed over TCP and must be exposed to the applications running in your AWS environment. We can achieve this using AWS PrivateLink. Using PrivateLink, you can privately connect your VPC to services as if they were in your VPC. In this scenario, we are using PrivateLink to make applications running in your third-party vendor’s data center appear as if they are running in VPC A. We show this in the following diagram, figure 3.

Figure 3: A multi-VPC environment connected to a third-party vendor’s on-premises data center using AWS PrivateLink.

PrivateLink requires a Network Load Balancer (NLB) to act as the front end for the applications running in the third-party vendor’s data center. The NLB is set up in a private subnet of VPC A, which is routed to the vendor’s data center using the public IP space. The vendor’s application endpoints are registered as IP based targets with the NLB. Now the NLB acts as a PrivateLink service that is accessed by other VPCs and accounts through a PrivateLink endpoint, and doesn’t require clients to target public IPs. There are two ways to do this:

1. Use the PrivateLink Endpoint in VPC A and allow applications in other VPCs to talk to that endpoint through the Transit Gateway. This approach is discussed in the blog post, Integrating AWS Transit Gateway with AWS PrivateLink and Amazon Route 53 Resolver. Consider this approach when you have many VPCs in your that must talk to the vendor’s application and the amount of data traffic isn’t too large. This reduces the operational complexity of managing multiple endpoints. (Figure 3 shows this approach.) This approach requires that VPC subnets connected to the Transit Gateway to have non-overlapping IP addresses.
2. Create PrivateLink endpoints in each VPC that must communicate with third-party services. Consider this approach when the number of VPCs that must communicate with the vendor’s systems is small, or the amount of data traffic is high. This makes PrivateLink a more attractive option compared to Transit Gateway when optimizing costs. This approach allows clients in VPCs with overlapping IPs to communicate to the service over the PrivateLink endpoint.

Although these are general guidelines, your decision may also depend on your overall network topology, which you’ve setup for other needs besides this integration.

Figure 3 shows the use of Transit Gateway along with PrivateLink. When using this pattern, be sure to consider:

• PrivateLink allows for clients from your VPCs to connect to third-party vendor’s applications, and the connection is from AWS to the vendor’s data center. This arrangement doesn’t allow connections in the reverse direction.
• Since the backend targets of the NLB are outside AWS, they must be addressed by IP, so you must consider client IP preservation. This entry in our documentation goes into more detail on this topic.
• PrivateLink currently supports only TCP traffic.
• Since this requires specifying the vendor’s application targets by IP, if there are any changes in IPs (because of servers restarting or reconfiguration), then you must update the NLB target groups. This adds to the operational complexity and can’t use DNS to learn the new IPs.

As the on-premises IP addresses are outside of the RFC1918 private address range, in this case, these backends must be allow-listed within the Amazon Elastic Load Balancer (Amazon ELB). Contact AWS Support for ELB backend allow-listing for non-RFC1918 IP addresses.

Scenario 4: Connecting with third-party vendors using Network Load Balancer and AWS PrivateLink

Although the previous scenarios involved clients from the AWS environment connecting to third-party applications running outside AWS, there are cases when clients from outside AWS must connect to backend servers inside AWS. This is to handle scenarios where you connect services from AWS to services running in third-party data centers over a dedicated private connection. This must still make sure that the clients are talking to a VPC or a subnet with a CIDR block that doesn’t overlap with the CIDR block in the vendor’s data center.

Figure 4: A multi-VPC environment connected to a third-party vendor’s on-premises data center using Network Load Balancer and AWS PrivateLink.

Although there may not be a requirement for public IP space from the third-party vendor, if there are multiple third-party vendors that must connect to the same endpoint, then having public IP space will eliminate the need to negotiate with third-party vendors to ensure their IPs do not overlap. Public IP space allows for unique IPs from each third-party vendor, addressing that need.

We achieve this architecture using an NLB as the front end of your applications (shown as App1 in VPC A and App2 in VPC C) and expose them over PrivateLink to the Private Routable Subnet of VPC A, as shown in figure 4. Using PrivateLink exposes the application to multiple third-party on-premises clients connecting from different VPCs. Moreover, this approach allows for handling TCP based applications. We show a Transit Gateway in the previous figure to show that it may be present in your environment, but it won’t take part in the traffic flow in this architecture.

Scenario 5: Connecting clients from outside AWS with applications on AWS using Application Load Balancer or Network Load Balancer

A variation on scenario four is to put an Elastic Load Balancer (this could be an NLB or ALB) inside VPC A, and connect it to the third-party vendor’s data center to act as a front-end for application access. We do this using a load balancer in the Private Routable Subnet that acts as a front end for the applications in that VPC and other VPCs connected to the Transit Gateway as shown in the following diagram, figure 5.

Figure 5: A multi-VPC environment integrated with third-party on-premises through AWS Direct Connect and Network Load Balancer.

You can use an NLB to achieve this task if the application traffic is TCP/UDP. Since NLB has a 1:1 mapping between a listener and target group, you may need multiple NLBs if you must allow access to multiple applications on the same port and protocol. However, if the applications can be accessed on different ports, then you can configure separate listener rules within a single NLB.

If you have HTTP based traffic, then you can use a single Application Load Balancer (ALB) to host all applications on a common DNS name (and protocol, port) and use path based routing to direct requests to different target groups corresponding to the backend applications. Refer to this Product comparison page for understanding the differences between NLB and ALB to choose the right one for your needs. You may prefer this approach over scenario four if you have traffic that isn’t TCP based and if you want to use a load balancer to multiplex across multiple backend applications.

Considerations

Although the above scenarios show VPC A connecting to the vendor’s data center outside AWS with a single subnet as the private routable subnet, we recommend using a highly available architecture with subnets in multiple Availability Zones (AZs).

These scenarios show a single third-party vendor connecting to your AWS environment. If there are multiple third-parties that must be integrated in this way, then one problem you encounter is transitive routing between connections that terminate on the virtual private gateway. This kind of routing is suitable for multiple branch sites of a customer connecting to their AWS environment, which is shown in the AWS VPN CloudHub architecture. However, this is undesirable when the connections that terminate on the virtual private gateway are from different vendors. To allow multiple third-parties to connect in this manner, VPC A must be replicated for each third-party and with its own private routable subnet and reuse the public IP secondary CIDR block of 198.51.0.0/28. This is possible because these public IPs are used for connecting to the respective on-premises connections and don’t need to route traffic to each other. This optimizes the public IP space required for this architecture.

Summary

This post covered five different scenarios for connecting your applications hosted on AWS to third-party systems hosted outside AWS, are multi-tenanted, and require considerations like non-overlapping IPs and unique IPs allow-listed to access their environment. You can choose an approach that best suits your workloads and the requirements of your third-party vendor. This post presented a real-life use case of our customer DoorDash, where they used this approach for connecting to their critical third-party systems securely. They met their goals of meeting third-party requirements, network performance, and security. We thank Roger Zeng, Software Engineer, DoorDash and Jay Wallace, Engineering Manager at DoorDash for their collaborative partnership in building the solution for their use case. We encourage our customers to consider these approaches when faced with such constraints.