Introducing private DNS support for Amazon S3 with AWS PrivateLink

Compliance requirements often mandate private connectivity when on-premises applications use cloud storage. To satisfy these requirements, customers set up private connections to Amazon S3 using AWS PrivateLink over either AWS Direct Connect, or AWS Site-to-Site VPN. As a result, data is transmitted directly to and from AWS, never traversing the public internet. AWS PrivateLink lets you provision interface endpoints in your Amazon Virtual Private Cloud (Amazon VPC), which assign private IP addresses to S3. PrivateLink automatically provisions globally unique public DNS names for these private IPs, which your applications can use to access S3. While you can create custom DNS entries on-premises to point on-premises clients at these private IP addresses when using S3’s regional names (s3.<Region>.amazonaws.com), this adds operational overhead and is difficult to manage, and as a result, this is not a preferred option.

To help simplify your DNS configuration for private connectivity, Amazon S3 now supports private DNS options for your S3 interface VPC endpoints. With private DNS for S3, your on-premises applications can use AWS PrivateLink to access S3 over an interface VPC endpoint, while requests from your in-VPC applications access S3 using gateway VPC endpoints. Routing requests like this helps you take advantage of the lowest-cost private network path without having to make code or configuration changes to your clients.

In this blog post, we demonstrate how to take advantage of private DNS to access Amazon S3 using AWS PrivateLink. We also discuss configuration options for various scenarios, and how to verify that your clients are connecting to Amazon S3 over gateway VPC endpoints and interface VPC endpoints.

VPC endpoints for Amazon S3

There are two types of VPC Endpoints that you can use to connect to Amazon S3 from your VPC: gateway endpoints and interface endpoints.

Gateway VPC endpoints provide reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for your VPC. Gateway VPC endpoints can be set up with a few clicks in the AWS Management Console, and use your VPC route table to route requests from clients within your VPC to S3’s or DynamoDB’s public IPs, over the AWS network.  Gateway VPC endpoints have no additional charge and support connectivity only from resources local to the respective VPC where the gateway endpoint is created.

Interface VPC endpoints provide private connectivity for over 140 AWS services and third-party SaaS applications using AWS PrivateLink. An interface VPC endpoint creates an elastic network interface (ENI) with a private IP address in your VPC subnets. Interface VPC endpoints support connectivity from on-premises over AWS Direct Connect or AWS Site-to-Site VPN. When you configure an interface VPC endpoint, we create PrivateLink provided public DNS names for your endpoint that are resolvable both within the VPC and from on-premises. Interface VPC endpoints have two charges – an hourly charge for each VPC endpoint provisioned in each Availability Zone, and a per-GB data processing charge. For pricing details, please visit AWS PrivateLink pricing.

The most cost-efficient way to access S3 is to use Gateway VPC endpoints where possible (e.g., from EC2 instances in the Region) and to use Interface VPC endpoints from other locations such as from on premises.

Accessing Amazon S3 interface endpoint with private DNS name

Figure 1 shows a hybrid network setup where you connect from on-premises over an AWS Direct Connect connection or AWS Site-to-Site VPN connection. In this setup, you configure Amazon Route 53 inbound Resolver endpoints and set up conditional forwarding on your on-premise DNS resolver to forward DNS queries to the private IP addresses of inbound resolver endpoints. Then, when you create an interface VPC endpoint for S3, you will have an option to enable a private DNS name.

Figure 1: Setup when connecting from on premises over Direct Connect or Site-to-Site VPN

When you enable private DNS for an S3 interface VPC endpoint, AWS creates a private hosted zone and associates it with the VPC. This hosted zone includes the resource records for interface VPC endpoint with private IPs for each of the following Amazon S3 DNS names:

• Regional Bucket (e.g., s3.<Region>.amazonaws.com)
• Control (e.g., s3-control.<Region>.amazonaws.com)
• Access Point (e.g., s3-accesspoint.<Region>.amazonaws.com)

This enables you to use AWS’s private network connectivity to S3 while making requests to the service’s Regional, control, or access point endpoints. For more information, refer to the AWS PrivateLink for Amazon S3 documentation.

Diagram walkthrough

1. On-premises client initiates a DNS query targeted to a regional S3 bucket.
2. The on-premises DNS server forwards this query to the respective Route 53 resolver inbound endpoint associated to the same VPC that has the S3 Interface VPC endpoint via the Site-to-Site VPN or DX connection.
3. The Route 53 resolver endpoint forwards this query to the Route 53 hosted zone managed by AWS which returns the IP addresses of the S3 Interface VPC endpoints in the DNS response.
4. On-premises client then initiates the connection to the S3 interface VPC endpoint.
5. S3 interface endpoints forwards the client query over AWS PrivateLink towards the S3 bucket specified in client’s query.

New – Enable private DNS only for inbound endpoint

Many customers have applications on premises and in an AWS Region, both running in the same VPC. These customers told us that they wanted an easy way to route traffic from on-premises through interface endpoints, and traffic from within AWS over gateway endpoints. To solve this problem, we introduced the Enable private DNS only for inbound endpoint option. When you check enable DNS name for Amazon S3 interface endpoints, the Enable private DNS only for inbound endpoint option will be checked by default. In this case, the DNS queries for S3 originating from on-premises resolve to the private IPs of S3 interface endpoints, while S3 DNS queries originating within the VPC continue to resolve to public IPs of S3 using gateway VPC endpoint. For this configuration to work, you need to have a gateway endpoint present in your VPC. If you do not, and you try to enable this configuration, you will see an error like this in AWS Management Console (Figure 2):

“To set PrivateDnsOnlyForInboundResolverEndpoint to true, the VPC <vpce_id> must have a gateway endpoint for the service.”

To resolve this, create a gateway endpoint in the VPC. Alternatively, you can disable Enable private DNS only for inbound endpoint and route all your traffic over the interface endpoint.

Figure 2: Error when gateway endpoint does not exist while PrivateDNSonlyForInboundEndpoint enabled

Prerequisites

Before you get started, make sure you have the following prerequisites met:

1. Create a VPC in the same Region as the S3 bucket that you wish to connect over the VPC endpoints. Make sure you have set the attributes enableDnsHostnames and enableDnsSupport to true.
2. Create either an AWS Direct Connect connection with a Private Virtual Interface (VIF) or an AWS Site-to-Site VPN connection to establish connectivity from your corporate data center.
3. Create a gateway VPC endpoint for S3 in the same VPC created in step 1 to use Enable private DNS only inbound endpoint.

Creating an interface VPC endpoint for Amazon S3 and enable private DNS options

1. To create an interface VPC endpoint for Amazon S3, first navigate to the VPC console, select Endpoints, and choose Create endpoint.
2. For Service category, select AWS services. Then, filter the service names by entering S3 in the search box. For Service name, choose the service as “S3” and for Type, ensure that it shows Interface (Figure 3a).

Figure 3a: Create Interface endpoint for S3

3. Select the VPC, desired Availability Zones, and subnet for each one, and then select the appropriate security groups. This should allow traffic from your networks on port 443.
4. Under Additional settings, select Enable DNS name for your interface endpoint. By default, it will select Enable private DNS only for inbound endpoint to make traffic originating inside your VPC flow over the gateway VPC endpoint and traffic originating on premises flow over the interface VPC endpoint.
5. Select Create endpoint. We show this in the following screenshot (Figure 3b):

Figure 3b: Select private DNS options while creating interface VPC endpoint in the VPC console

It takes a few moments to go through various endpoint states while it is created. Once the interface endpoint status says available, you can view its configuration by choosing Details (Figure 4a). The DNS Names field displays the DNS names used to access the service. With private DNS enabled, you can see the default S3 Region DNS name as well.

Figure 4a: Details of the interface VPC endpoint

Choose Subnets to see where the interface endpoint is located, and the ID of the endpoint network interface in each subnet. In the following screenshot (Figure 4b), the private IP address of the endpoint network interface in the VPC are 10.0.4.122 and 10.0.23.155.

Figure 4b: Subnets information for your interface VPC endpoint

Scenarios for private DNS options

Let’s understand the various combinations of DNS options that influence clients connectivity to Amazon S3 from applications hosted in VPC and on-premises using Gateway and Interface VPC endpoint for Amazon S3:

Scenario 1: Without private DNS options

In this configuration, traffic from clients within the VPC where the gateway endpoint has been created can connect to S3 Regional endpoints. Clients outside the VPC (either on-premises or another interconnected VPC) can connect to S3 using the endpoint-specific DNS names or using the options highlighted here in the blog. In this case, you cannot set Enable private DNS only for inbound endpoint when private DNS is set to false.

This option is useful when you want to have the flexibility to manage private DNS names in your own private hosted zone.

Client in VPC using gateway endpoint:

dig s3.<Region>.amazonaws.com +short
54.x.y.z (Public IP address(es) of S3 service endpoint)

Client in VPC or on premises using endpoint-specific DNS name:

dig *.vpce-0cd6fd8a1a7d95f7e-4nyak8fx.s3.<Region>.vpce.amazonaws.com +short
10.0.23.155
10.0.4.122

The preceding output shows that clients inside the VPC resolve to a public IP address(es) of an S3 service endpoint, while the clients in the corporate data center resolve to interface VPC endpoint ENI IP addresses for Amazon S3.

Scenario 2: With private DNS

In this configuration, both in-VPC and on-premises traffic flows over an interface VPC endpoint for S3. This option is beneficial when you want to simplify the architecture to just use one type of endpoint since it simplifies DNS management. However, this is not a cost-efficient solution since now the traffic from resources in VPC to S3 would also incur data transfer charges associated with interface VPC endpoint for S3. VPC As shown in Figure 5, the green and blue color show that the traffic is flowing from the EC2 instances within the VPC and on-premises environment over the interface VPC endpoint for S3.

Figure 5: With private DNS enabled and Enable private DNS only for inbound endpoint disabled

Diagram walkthrough

All the Steps 1 through 5 remain the same as described in Figure 1. However, now with only Private DNS enabled clients inside the VPC as well as the clients in on-premise connect to Amazon S3 via the interface VPC endpoint for S3.

Client inside the VPC:

dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

Client inside on-premises application:

dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

The preceding output shows that both the clients inside the VPC and on premises resolve to interface VPC endpoint ENI IP addresses for Amazon S3.

Scenario 3: With private DNS only for the inbound resolver endpoint

In this configuration, the traffic from applications within the VPC flows over gateway VPC endpoints, while on-premises traffic flows over the interface VPC endpoint for S3. This option provides a cost-effective network design to access S3 from within VPC and on-premises applications. While choosing this configuration, you need to maintain a gateway VPC endpoint in you VPC. This is to keep your traffic always on the AWS private network. This eliminates the possibility where, without a gateway endpoint, you’re in-VPC traffic inadvertently goes over an Internet Gateway or is dropped if there is no Internet Gateway. Hence, if a gateway VPC endpoint does not exist in the VPC where you have applications running, it will prevent you from selecting the Enable private DNS only for inbound endpoint option. If you want to want to update an existing interface endpoint to Enable private DNS only for inbound endpoint, you must confirm that your VPC has a gateway VPC endpoint for S3. For more information about gateway VPC endpoints and managing private DNS names, see Gateway VPC endpoints and Manage DNS names respectively in the AWS PrivateLink Guide.

Figure 6 shows the blue path for traffic from EC2 instances within the VPC flowing via gateway VPC endpoints, whereas the green path shows the traffic flow from on premises to S3 using interface VPC endpoint.

Figure 6: With private DNS enabled and Enable private DNS only for inbound endpoint enabled

Diagram walkthrough

All the Steps 1 through 5 remain the same as Figure 1. However, now with both Private DNS and Enable private DNS only for inbound endpoint enabled, clients inside the VPC connect to the Amazon S3 via Gateway VPC endpoint for S3 while clients on-premise connect to Amazon S3 via the interface VPC endpoint for S3.

Clients in VPC using gateway endpoint:

$ dig s3.<Region>.amazonaws.com +short
54.x.y.z (Public IP address(es) of S3 service endpoint)

Clients in on premises using interface endpoint:

$ dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

You cannot delete a gateway VPC endpoint when both private DNS and Enable private DNS only for inbound endpoint are enabled. If you attempt to do that, it will throw the following error:

“Gateway endpoint cannot be deleted while Interface endpoint for the service has PrivateDnsOnlyForInboundResolverEndpoint set to true.”

In this case, if you want to delete the gateway VPC endpoint, you need to modify your interface VPC endpoint and deselect the option Enable private DNS only for inbound endpoint.

Conclusion

In this blog post, we discussed using private DNS for an Amazon S3 interface VPC endpoint to access Amazon S3 without modifying on-premises applications. We covered using the Enable private DNS only for inbound endpoint option to optimize the network path to S3. These options allow you to advantage of the lowest-cost private network path without having to make code or configuration changes to your clients. To get started with AWS PrivateLink for Amazon S3, visit this page.

For more information on AWS PrivateLink for Amazon S3, you can refer to the following blogs:

• Secure hybrid access to Amazon S3 using AWS PrivateLink
• Choosing Your VPC Endpoint Strategy for Amazon S3
• AWS Partners use AWS PrivateLink to connect privately to Amazon S3