Networking & Content Delivery
Secure hybrid access to Amazon S3 using AWS PrivateLink
AWS PrivateLink for Amazon S3 enables on-premises applications to privately and securely access Amazon S3 over AWS Direct Connect private virtual interface or AWS Site to Site VPN. The Interface VPC Endpoints for Amazon S3 allow security administrators to control which users can access which data in S3 from on premises and cross-Region using their own private IP addresses over a private network.
To privately access Amazon S3 from inside a Amazon Virtual Private Cloud (VPC), you can use Gateway VPC endpoints for Amazon S3. These allow applications running in a VPC to access S3 without an Internet gateway or NAT gateway. When using Gateway VPC Endpoints, VPC endpoint policies are used to restrict access allowing requests to S3 Buckets from only authorized users. In addition, you control which buckets are accessible from a particular VPC. This is the recommended model for accessing S3 from a VPC in the same Region. However, to use a Gateway VPC endpoint from on-premises applications, or to access S3 from a VPC in a different AWS Region, you must set up a fleet of proxy servers with private IP addresses in your VPC. This results in changes to your on-premise applications so that they direct requests to the proxy servers, and then forward them to S3 through your VPC Endpoint. This comes with administrative overhead, adds the cost of running the proxy servers, and increases the operational complexity of your application.
AWS PrivateLink for Amazon S3 solves these challenges and enables multiple use-cases:
- Privately accessing S3 from on premises: This feature lets you to allow on-premises users and applications access to S3 buckets, AWS Accounts, or AWS Organizations. Corresponding S3 bucket policies can restrict access from only specific Interface VPC Endpoints.
- Accessing S3 from other Regions: Allows administrators to use existing private networks for inter-region connectivity (for example, Amazon VPC peering connections or AWS Transit Gateway) while still enforcing VPC, bucket, account, and organizational access policies.
In this blog post, we show how to access Amazon S3 buckets from on-premise networks. For example, AWS Direct Connect or VPN over private connectivity using AWS PrivateLink for S3. We discuss and walk you through how to use various DNS options to enable the connectivity from on-premise applications. We also show how to configure your on-premises DNS resolvers to direct S3 domain names to the interface endpoint IPs by forwarding DNS queries to Amazon Route 53 Resolver Inbound Endpoints.
Pre-requisites
Before you begin, make sure you have an Amazon VPC dedicated to this solution. You will also need an AWS Direct Connect connection configured with a private virtual interface (Private VIF), or have an AWS-Site-to-Site VPN connection up and running to your VPC.
Create an Interface VPC endpoint for Amazon S3
- To create an Interface VPC endpoint for Amazon S3, first navigate to the VPC Console, select Endpoints, and choose Create Endpoint.
- For Service category, ensure that AWS servicesis selected. Then, filter the service names by entering S3 in the search box. For Service Name, choose the service as “S3” and for Type, ensure that it shows Interface.
- Select the VPC, desired AZs, and subnet for each one, and then select the appropriate security groups (this should allow traffic from your networks on port 443) and click on Create endpoint. We show this in the following screenshot (figure 1).
Figure 1: Create VPC Interface Endpoint for S3
It takes a few moments to go through the interface endpoint lifecycle steps while it is created. After the interface endpoint is available, you can view its information by choosing Details. The DNS Names field displays the DNS names used to access the service. You will need the DNS names later, so make a note of them now.
Figure 2: Endpoint in the Available state
Choose Subnets to see where the interface endpoint is located, and the ID of the endpoint network interface in each subnet. In the following screenshot (figure 3), the private IP address of the endpoint network interface in the VPC are 10.0.1.165 and 10.0.2.19.
Figure 3: Interface Endpoint IP Addresses
DNS considerations for accessing S3 buckets using AWS PrivateLink
There are a few DNS options to choose from when accessing Amazon S3 over PrivateLink.
Option 1: Access Amazon S3 without modifying your on-premises DNS resolver
AWS Privatelink creates endpoint-specific DNS hostnames that you can use to communicate with the service. These endpoint-specific DNS hostnames resolve to the private IP address of the endpoint network interface in the VPC.
When using endpoint-specific DNS names to access the interface endpoints for Amazon S3, you don’t have to update your on-premises DNS resolver. You can resolve the endpoint-specific DNS name to the private IP address of the interface endpoint from the public Amazon S3 DNS domain. For example, you can use the endpoint-specific regional DNS hostname, let’s say, vpce-001760f0be22cd4e2-3ew5rhyv.s3.eu-west-2.vpce.amazonaws.com, from your on-premises applications to access Amazon S3 without modifying your DNS resolver configuration.
Or, you can access S3 APIs without altering DNS settings when you cannot change the DNS resolution of your environment in some scenarios. You can do this by using the – -endpoint-url flag to direct API calls to the S3 VPC Endpoint directly. For example:
The VPC Endpoint’s DNS name is publicly resolvable, but resolves to a private IP address. So as long as we have layer 3 access to the VPC, we can connect.
When using TLS/SSL to access Amazon S3 buckets, further consideration is needed. The endpoint URL requires a prefix depending on the action requested for successful TLS/SSL verification. For example, when listing bucket contents, you would use an endpoint address such as https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com. More information on how to access S3 buckets, S3 access points, or S3 control APIs through S3 interface endpoints can be found in the Amazon S3 documentation.
Connections here will use TLS/SSL over the private connection, as shown in the packet capture that follows (figure 4).
Figure 4: Packet capture of private communication
In addition to S3 buckets, you can also use AWS PrivateLink for Amazon S3 to access S3 access points and S3 control APIs. For more information, including restrictions and limitations, please visit the AWS PrivateLink for Amazon S3 section of the Amazon S3 documentation.
Option 2: Access Amazon S3 using Domain Name System Response Policy Zones (DNS RPZ)
The simplest way is to configure your on-premises DNS resolver to resolve the S3 DNS names to the private IP addresses of the Interface VPC endpoint for S3. The IP addresses of Interface VPC endpoints are static and do not change until the interface endpoint is deleted. This provides the flexibility to use the static private IP addresses of the interface endpoint in the DNS zone configuration file of the on-premise DNS resolver. However, the VPC Interface Endpoint for S3 IP addresses can change if you add or remove the subnets associated with the endpoint. If a failure were to occur within one availability zone, AWS may temporarily withdraw an interface endpoint IP from DNS to route clients toward the healthy AZs. On-premises networks, which statically configure IPs in DNS, will not benefit from these mitigations. As a result, we do not recommend this option.
However, if you want to create a DNS zone in your on-premises DNS, you can use Domain Name Service Response Policy Zones (DNS RPZ) to create a CNAME record, provided your on-premises nameserver supports it. This option is beneficial, as the configuration is static even if the interface endpoint IP addresses change. This solution has no cost involved if the DNS resolver supports it.
For example, if you have RPZ set up in BIND, and your RPZ is called s3.eu-west-2.amazonaws.com, you would create two records:
s3.eu-west-2.amazonaws.com CNAME vpce-001760f0be22cd4e2-3ew5rhyv.s3.eu-west-2.vpce.amazonaws.com.
*.s3.eu-west-2.amazonaws.com CNAME vpce-001760f0be22cd4e2-3ew5rhyv.s3.eu-west-2.vpce.amazonaws.com.
This would be visible to any user using:
Implementing RPZ varies and depends on your DNS resolver vendor. For specific configurations, we suggest you refer to the respective vendor documentation.
Option 3: Forwarding DNS requests from on premises using Amazon Route 53 Resolver Inbound Endpoints
This option is useful when the local DNS resolver does not support Response Policy Zones or if you cannot or do not want to change your on-premises applications when accessing Amazon S3 buckets. In this option, you make use of Amazon Route 53 Resolver Inbound Endpoints to forward DNS requests for s3.<region>.amazonaws.com to a private hosted zone to provide private DNS resolution.
Today, Amazon S3 interface endpoints do not support the Private DNS feature of interface endpoints. Therefore, you need to create a custom private hosted zone within Amazon Route 53.
Now we walk you through the steps required to accomplish this. Figure 5, which follows, shows the high-level architecture and the step numbers involved in each.
Figure 5: Forwarding DNS requests via Route 53 Resolver
- Create a private hosted zone for S3 in the region we’re operating in. In this example, we’ve chosen eu-west-2, so we create a private hosted zone with the name s3.eu-west-2.amazonaws.com. Then, create two resource records in Amazon Route 53 of the Alias type – one for the apex of the domain and the second as a wildcard (*) to allow all records within this domain to be resolved to the VPC endpoint.
Figure 6: Apex record creation
Figure 7: Wildcard record creation
When you create the resource record, it presents you with multiple options for your particular VPC endpoint – one endpoint-specific regional DNS hostname, and the other one endpoint-specific zonal DNS hostname for each Availability Zone where you created the VPC Endpoint. You must select the endpoint-specific regional DNS hostname in this case, which for our deployment is ‘vpce-001760f0be22cd4e2-3ew5rhyv.s3.eu-west-2.vpce.amazonaws.com’.
- Associate the private hosted zone with a VPC. We recommend using the same VPC that was used to create the S3 VPC Endpoint, as you should already have connectivity to it using AWS VPN or Direct Connect.
Figure 8: Associate private hosted zone with VPC
- Create an Amazon Route 53 Resolver Inbound Endpoint to receive and forward the requests from your on-premises network. Here, we must use the same VPC where you created an S3 VPC Interface Endpoint, and is associated with the private hosted zone.
Figure 9: Inbound Endpoint example
- Using the IP addresses of the Amazon Route 53 Resolver Inbound Endpoint as displayed above, we now configure a conditional forward on our on-premises DNS resolver. The exact configuration for the conditional forwarder depends on the DNS resolver you are using on your on-premises network. For example, using a BIND server, the configuration of named.conf for the Regional S3 endpoint looks like the following where 10.0.122 and 10.0.2.213 are private IP addresses of Amazon Route 53 Resolver Inbound Endpoints:
Now, we perform a DNS lookup for our S3 bucket from the client on our on-premise network. As you can see in the text that follows, we get the private IP addresses of the Interface VPC Endpoint for S3 in the DNS response.
(Optional) Splitting access between VPC Interface Endpoints and internet
In certain situations, it may be beneficial to send traffic for some S3 buckets by way of the endpoint, and others using standard Internet paths. This is especially beneficial in hybrid environments, where users or applications on the on-premises network need to access specific S3 buckets to perform standard functions, and you wish to utilize AWS Direct Connect or AWS Site-to-Site VPN. Allowing all requests for S3 buckets to use a private connection might result in unnecessary over-utilization or unexpected results.
Perhaps, though, other less critical applications are hosted on S3 in the eu-west-2 region. Currently, the DNS hostname for our other S3 bucket, for example, “adifferentbucket”, will also resolve to a private IP. Requests to this bucket are using our private connection, which may be unnecessary.
To prevent this from happening, we need to make our conditional forwarding rule more specific. To forward only requests for the bucket name “myspecificbucket” to Amazon Route 53 Resolver, we have created the conditional forwarding rule as shown below:
Now, when we perform a DNS lookup for our S3 bucket myspecificbucket.s3.eu-west-2.amazonaws.com from the client on our on-premise network, we can still resolve it privately. However, if we perform a DNS lookup on the bucket name “adifferentbucket”, we are now resolving this S3 bucket publicly:
Securing access to Amazon S3 using VPC Endpoints
In the previous sections, we showed you how to access S3 privately from our VPC and on-premise network. Now, we make sure that the access is secure and limited. There are a couple of different approaches to this task:
Option 1: S3 Bucket Policies
You can limit access to your buckets to only requests coming via the VPC Endpoint using S3 Bucket Policies. To do this, you can use a condition called “aws:SourceVpce”. Please visit the documentation for example endpoint policies for accessing Amazon S3.
Option 2: Interface VPC Endpoint Security Groups
S3 Interface Endpoints support security groups, adding another layer of security to the endpoint. You can restrict traffic to particular source subnets and/or destination ports. In the example below, we allow only HTTPS (TCP port 443) traffic from our on-premise subnet (10.10.10.0/24):
Figure 10: Security Group configuration
Cleanup
Depending on the sections of this blog you have used, you may have created some resources that incur costs. If you are not intending to use this solution, you will want to ensure that you remove:
- Any Amazon S3 buckets created for the purpose of testing
- Any AWS Site-to-Site VPN connections, and/or AWS Direct Connect connections created for testing
- AWS VPC Interface Endpoint for S3
- Amazon Route 53 Private Hosted Zone(s)
- Amazon Route 53 Resolver Inbound Endpoint
Conclusion
In this blog post, we showed how you can create Interface VPC endpoints for Amazon S3 and how on-premises applications can access Amazon S3 buckets over AWS Direct Connect or VPN. We also discussed various options for DNS resolution for queries originating from on-premises applications, and how to securely access Amazon S3. To get started with AWS PrivateLink for Amazon S3, visit this page.
For more information on AWS PrivateLink for Amazon S3, you can refer to the following blogs:
Choosing Your VPC Endpoint Strategy for Amazon S3
AWS Partners use AWS PrivateLink to connect privately to Amazon S3
Josh Dean
Josh is a Senior Partner Solutions Architect at AWS. He works closely with networking partners to build solutions and capabilities to enable and simplify their migrations and operations in the cloud. Prior to AWS, Josh worked all the way up the stack, from data center engineering to application support, generally with a networking focus.
Rohit Aswani
Rohit is a Senior Specialist Solutions Architect focused on Networking at AWS, where he helps customers build and design scalable, highly-available, secure, resilient and cost effective networks. He holds a MS in Telecommunication Systems Management from Northeastern University, specializing in Computer Networking.