Migrate your workloads to use VPC endpoints with minimum downtime

Amazon Virtual Private Cloud (Amazon VPC) endpoints are comprised of gateway and interface endpoints that enable users to privately access supported Amazon Web Services (AWS) services and VPC endpoint services powered by AWS PrivateLink. They offer several benefits for organizations looking to enhance their cloud infrastructure’s security, performance, and cost efficiency.

In an earlier post, Reduce cost and increase security with Amazon VPC endpoints, we covered the benefits of using VPC endpoints. In this post, we explore the current challenges faced by organizations using internet egress architectures using Network Address Translation gateway (NAT gateway) and Internet gateway (IGW), as well as ways that VPC endpoints can assist in overcoming these challenges. We also cover steps to migrate to VPC endpoints in your existing architecture with minimum downtime.

Internet egress architectures

One of the common ways that clients located in VPCs access AWS services and other third-party services hosted in AWS is over the internet. This requires clients to either have IPv6 global unicast addresses (GUA) or public IPv4 addresses with NAT to communicate with the services. For IPv4 traffic, an IGW is attached to a VPC and NAT gateways (or self-managed NAT instances running on Amazon Elastic Compute Cloud (Amazon EC2) instances) are provisioned in public subnets of the VPC. For IPv6 traffic, an IGW attached to the VPC is used for bi-directional connectivity, or an egress only internet gateway (EIGW) can be used to enable outbound-only connectivity from the clients within the VPC. Traffic toward the AWS services and third-party services is routed to the NAT gateway, IGW, or an EIGW using VPC route tables configurations.

As you deploy applications in a multi-VPC and multi-account environment, each VPC that needs access to the AWS services and third-party services can be configured in a distributed manner, or it can be configured to be sent to NAT gateways in a centralized egress VPC using an AWS Transit Gateway (or AWS Cloud WAN). In this section, we look at both of these options.

Distributed internet egress architecture

Figure 1: Distributed internet egress architecture

In a distributed internet egress model, each VPC has public subnets with their own NAT gateways and IGW deployed, as shown in the preceding figure, with public IPv4 addresses or IPv6 GUA addresses assigned to public subnet resources. Then, subnet route tables (not shown in the preceding figure) are configured to route internet traffic through the NAT gateways and IGW respectively.

Centralized internet egress architecture

Figure 2: Centralized internet egress architecture

Deploying NAT gateways in every spoke VPC can become cost prohibitive, because you pay an hourly charge and data processing charges per gigabyte for every NAT gateway that you deploy. Centralizing NAT gateways can be a viable option to reduce costs. To centralize, you create a separate egress VPC in the central network services or egress account, deploy NAT gateways in the egress VPC, and route all egress traffic from the spoke VPCs to the NAT gateways residing in the egress VPC by either using a Transit Gateway as shown in the preceding figure, or AWS Cloud WAN, which is not shown in the preceding figure.

Challenges with the internet egress architectures

Using internet egress architectures to access AWS services and other third-party services raises several challenges:

• Security risks: Data transmitted over the internet can be susceptible to potential threats, such as Distributed Denial of Service (DDoS) attacks, man-in-the-middle attacks, and unauthorized access attempts.
• Network performance: Internet-based connections can give inconsistent user experience, for example higher latency, packet loss, and bandwidth constraints. This can lead to suboptimal performance for critical application and services.
• Cost management: Transferring data over the internet can incur significant costs, especially for organizations with high data transfer requirements.
• Scalability: Internet-based connections may not provide the same level of scalability and reliability as private connections within the AWS network, potentially impacting the performance and availability of applications and services.
• Operational overhead: Users must use complex configurations, such as private NAT, to handle overlapping CIDRs, which adds operational complexity.

VPC endpoints as a solution

VPC endpoints are virtual devices. They are horizontally scaled, redundant, and highly available Amazon VPC components that allow communication between resources in an Amazon VPC and services without imposing availability risks or bandwidth constraints on network traffic. There are two types of VPC endpoints:

• Interface endpoints (shown in the following Figure 3)
• Gateway endpoints (shown in the following Figure 4)

Figure 3: Interface endpoints

Interface endpoints are a type of VPC endpoint that allows you to establish private connectivity between your VPC and supported AWS services, which you expose using PrivateLink and other third-party services from PrivateLink partners (as shown in the preceding figure). Traffic between your VPC and the supported services remain within the AWS network, thereby reducing exposure to the public internet.

Figure 4: Gateway endpoints

Gateway endpoints allow you to establish a secure and scalable connection between your VPC and specific AWS services (Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB as shown in the preceding figure). These endpoints act as entry points for traffic destined for the AWS services, allowing you to keep your network traffic within the AWS network, thereby reducing data transfer costs and improving security. Note that both Amazon S3 and DynamoDB also support interface endpoints.

When considering VPC endpoints in AWS, you must understand the service and usage requirements. AWS offers gateway endpoints for Amazon S3 and DynamoDB, which are cost-effective but not accessible from outside of the VPC in which they reside. Although interface endpoints incur costs, they support a wider range of services and can be accessed from outside the VPC in which they reside (for example, from on-premises networks).

When migrating traffic from NAT gateways to VPC endpoints, expect flow disruptions due to the need to establish new traffic paths. You should plan your migration carefully to minimize impact on ongoing operations.

Migrating workloads to VPC endpoints

When migrating workloads from NAT gateway, consider deploying one of the following: a distributed interface endpoint model across multiple VPCs, a centralized interface endpoint model for consolidated access, or distributed gateway endpoints based on your AWS services needs and network architecture.

For the migration steps shown in the rest of this post, we have used access to Amazon S3 as an example. Note that the same steps apply to other supported AWS services and VPC endpoint services powered by PrivateLink (not shown in the following).

Here, we focus on the following three options:

1. NAT gateway to distributed VPC interface endpoints
2. NAT gateway to centralized VPC interface endpoints, and
3. NAT gateway to gateway endpoints

Option 1: NAT gateway to distributed VPC interface endpoints

Figure 5.1: Distributed internet egress architecture with NAT gateways

Figure 5.2: Distributed architecture using interface endpoints

In a distributed VPC interface endpoint architecture, you deploy an interface endpoint in one or more Availability Zones (AZs) in each VPC, where your resources that need access to the service are located (see Figure 5.2). This approach makes sure that each VPC can directly connect to the necessary services privately, without traversing the public internet.

Step 1: Identify the VPCs and create interface endpoints

1. Determine the VPCs that need access to the service. An interface endpoint needs to be created in each of these VPCs.
   • In each of the determined VPCs, create a VPC interface endpoint by following these steps:
     1. Navigate to the VPC dashboard, select the appropriate Region, and Create endpoint.
     2. Under Endpoint setting, add a Name tag and select the Service category as AWS services.
     3. For Services, select your desired service. In this example, we selected com.amazonaws.us-east-1.s3 with the Type as Interface, as shown in the following image.

Figure 5.3: Create an endpoint

• • Under Additional settings, you can enable Private DNS on services that support Private DNS, as shown in the following image.

Figure 5.4: DNS settings

Step 2: Associate subnets and configure security groups

• Under Subnets, attach the endpoint to the appropriate subnets within each VPC. You may select up to one subnet per AZ, as shown in the following image.

Figure 5.5: Subnets selection

• For Security groups, select the security group that you would like to associate with the endpoint. Make sure that the security groups associated with the interface endpoints allow traffic from the respective VPCs or subnets, as shown in the following image.

1. 1. You may refer to AWS docs for ways to configure inbound and outbound rules to control access within security groups.
   2. Select Create Endpoint. 

Figure 5.6: Associate security group

Step 3: Update DNS to gain access

To update the DNS for accessing AWS services through interface endpoints, you can either enable private DNS or manually create an Amazon Route 53 private hosted zone. Make sure the AWS service supports private DNS. If not, then opt for creating a private hosted zone.

Solution 1: Enable private DNS for the interface endpoint

• If you did not enable Private DNS while creating the endpoint as shown in the Step 1, then you may enable Private DNS by selecting the endpoint → navigate to Actions, and select Modify Private DNS name, as shown in the following image.

Figure 5.7: Modify private DNS name

• UnderModify private DNS name settings, select Enable for this endpoint → Save changes, as shown in the following image.

Figure 5.8: Private DNS settings

Solution 2: If private DNS is not supported or if you do not want to use private DNS, then create a private hosted zone and associate the VPC containing the interface endpoint, as shown in the following image.

1. Create aRoute 53 private hosted zone.
2. Under the private hosted zone, selectCreate record.
   1. In the Record name, use the respective subdomain if you have one, or leave it blank if you are creating a record for the root domain.
   2. Select Alias.
   3. For Route traffic to, navigate to Alias to VPC endpoint, and select the endpoint.
   4. Select the Simple routing policy, and set Evaluate target health to Yes.
   5. Select Create records.

Figure 5.9: Create a DNS A record for the VPC endpoint

Step 4: Migrate workloads

1. A VPC endpoint introduces a new network path that becomes the preferred route after a DNS update. As a result, existing TCP connections are interrupted and then reestablished. During the migration of traffic from a NAT gateway to interface endpoints, when you enable private DNS or create a record using a private hosted zone, there may be a few seconds of delay for the DNS changes to propagate (DNS Time-to-Live (TTL) to expire). After the DNS TTL expires, new DNS queries resolve to the private IP addresses of the interface endpoints, causing new TCP flows to be established through the interface endpoints instead of the NAT gateway. We recommend that you do not have critical tasks running when you create an endpoint, or that you test to make sure that your software can automatically reconnect to the endpoint after the connection breaks.
2. Once the DNS switches over, all of the resources within your VPC should be able to communicate with the service through the interface endpoint.

Step 5: Decommission NAT gateways

1. Once traffic for the service has been successfully migrated to the interface endpoints across the relevant VPCs, you can begin decommissioning the NAT gateways that are no longer needed.
2. Make appropriate updates to your route tables to remove references to the decommissioned NAT gateways.

Step 6: As a best practice, you should use CloudWatch metrics available for interface endpoints to monitor and track their performance effectively.

Option 2: NAT gateway to centralized VPC interface endpoints

In a centralized VPC interface endpoint architecture, you can create a more efficient and manageable network by centralizing interface endpoints within a dedicated egress VPC, either in the same AWS account or in a dedicated networking/egress account. Then, the spoke VPCs are connected to the egress VPC through a Transit Gateway, shown in the following figures, or AWS Cloud WAN, not shown in the following figures.

Figure 6.1: Centralized internet egress architecture with NAT gateways

Figure 6.2: Centralized architecture using interface endpoints

Step 1: Create VPC interface endpoints in a centralized VPC

1. You can either make an existing VPC as your centralized egress VPC or create a new egress VPC. You may also have a VPC in a dedicated centralized egress AWS account.
2. To create interface endpoints, follow the same steps mentioned in Option 1 Step 1.

Step 2: Associate subnets and configure security groups

1. Follow Option 1 Step 2 to associate subnets and security groups.

Step 3: Update DNS to gain access

• In comparison to the distributed model, it is necessary to configure a private hosted zone in a centralized model, as a private DNS for interface endpoints is only accessible from within the VPC where it is defined. Once you create a private hosted zone, associate the centralized egress VPC containing the VPC interface endpoint as well as the spoke VPCs to the private hosted zone.
• While creating the private hosted zone, select the Region and associate the respective VPC IDs of the centralized VPC as well as the spoke VPCs, as shown in the following image.

Figure 6.3: Associate VPCs to the hosted zone

• Follow the steps listed under Option 1 Step 3 Solution 2.2 to create an alias record.
• To associate a private hosted zone with VPCs in different accounts, you may use one of the following methods:

Method 1: Using Amazon Route 53 Profiles: Amazon Route 53 Profiles simplify the assignment of VPCs to private hosted zones, especially when the VPCs are in different AWS accounts. This approach eliminates the need for programmatically managing associations between VPCs and private hosted zones across accounts. This provides a centralized and scalable method for configuration and management, thereby increasing operational efficiency.

Method 2: Using programmatic method: Alternatively, you can use a programmatic method, such as the AWS Command Line Interface (AWS CLI) to associate a VPC with a private hosted zone when the VPC is in a different AWS account. Refer to the Associating an Amazon VPC and a private hosted zone that you created with different AWS accounts documentation for more information.

Step 4: Migrate workloads

A VPC endpoint introduces a new network path that becomes the preferred route after a DNS update. As a result, existing TCP connections are interrupted and then reestablished. Once you create an alias record under the private hosted zone, there may be a few seconds of delay for the DNS TTL to expire and change to propagate through the environment. After the DNS TTL expires, new DNS queries from all spoke VPCs resolve to the private IP addresses of the interface endpoints in the centralized egress VPC. This causes new TCP flows to be established through the interface endpoints instead of the NAT gateway. Then, existing TCP flows stall and must be reestablished. We recommend that you do not have critical tasks running when you create an endpoint, or that you test to make sure that your software can automatically reconnect to the endpoint after the connection break.

Step 5 and Step 6 for the centralized model remain the same as the distributed model (Option 1 Step 5 and Step 6).

Option 3: NAT gateway to gateway endpoints

In a distributed architecture with gateway endpoints, you deploy an Amazon S3 and/or DynamoDB gateway endpoint in each VPC where your resources that need access to the service reside (see the following figures). This approach makes sure that the resources within the VPC can directly connect to Amazon S3 or DynamoDB privately without traversing the public internet.

Figure 7.1: Distributed internet egress architecture with NAT gateways

Figure 7.2: Distributed architecture using gateway endpoints

Step 1: Create a VPC gateway endpoints

1. Determine the VPC that needs access to Amazon S3 or DynamoDB. A gateway endpoint needs to be created in this VPC.
2. Create a VPC gateway endpoint by following these steps:
   1. Navigate to the VPC dashboard, select the appropriate Region, and select Create endpoint.
   2. UnderEndpoint setting, add a Name tag and select the Service category as AWS services.
3. For Services, select your desired service. In this example, we selected amazonaws.us-east-1.s3 with the Type as Gateway, as shown in the following figure.

Figure 7.3: Create a gateway endpoint

 Step 2: Associate a route table with the gateway endpoint at creation

1. Under Route tables, select the route table(s) associated with the VPC subnet(s) where your resources are located to add a route entry that points toward the gateway endpoint as the next hop.
2. Select Create endpoint.

Figure 7.4: Associate a route table with the gateway endpoint

Step 3: Migrate workloads

A VPC endpoint introduces a new network path that becomes the preferred route after a DNS update. As a result, existing TCP connections are interrupted and then reestablished. Once you create a gateway endpoint and update the route table with a route entry pointing to the gateway endpoint, the previous connections that used NAT gateway are not reused. We recommend that you do not have critical tasks running when you create or modify an endpoint, or that you test to make sure that your software can automatically reconnect to Amazon S3/DynamoDB after the connection break.

Step 4: Decommission NAT gateways

1. Once all of the traffic for the specific AWS services has been successfully migrated to the interface endpoints across the relevant VPCs, you can begin decommissioning the NAT gateways that are no longer needed.
2. Make appropriate updates to your route tables to remove references to the decommissioned NAT gateways.

Considerations

• Gateway endpoints do not incur additional charges. However, interface endpoints have additional costs associated with them. Refer to the PrivateLink pricing documentation for interface endpoint costs.
• Gateway endpoints do not have a bandwidth limitation associated with them. However, interface endpoints have bandwidth limitations. Refer to the PrivateLink quotas documentation for details.
• If you are currently accessing a third-party application hosted in AWS through NAT gateways and would like to migrate to using interface endpoints, the third-party application needs to support access over PrivateLink. Consult with the third-party provider to make sure they have the necessary architecture in place to support access over PrivateLink.
• Refer to the Choosing TTL values for DNS records section in the Best practices for Amazon Route 53 DNS documentation to understand the effects of DNS TTL and ways to minimize its effects on the migration patterns described in this post.

Conclusion

In this post, we demonstrated the process of migrating access to AWS services and other third-party services from using the internet through NAT gateways and IGWs to using VPC endpoints. We explored the various types of VPC endpoints available, discussed essential design considerations to evaluate before choosing an endpoint type, and outlined steps for migrating workloads to a centralized interface endpoint, distributed interface endpoint architecture, or gateway endpoints.

About the authors