Design Considerations Using VPC Endpoints for Amazon S3 with VMware Cloud on AWS

By Kiran Reid, Partner Solutions Architect – VMware Cloud on AWS
By Ziv Klempner, Specialist Solutions Architect – VMware Cloud on AWS
By Jordi Minguez Orozco, Specialist Solutions Architect – VMware Cloud on AWS

With the general availability of AWS PrivateLink for Amazon S3 released earlier this year, customers can take advantage of private connectivity between S3 and on-premises resources using private IPs from your virtual network.

In this post, we’ll show you how to leverage interface VPC endpoints (AWS PrivateLink) for Amazon S3 and its integrations with VMware Cloud on AWS, which brings VMware’s enterprise-class software-defined data center (SDDC) software to the AWS Cloud.

In many organizations, Chief Information Officers (CIOs) and Chief Information Security Officers (CISOs) are driving Zero Trust initiatives to reduce business risk. They need to meet regulatory and compliance requirements, such as avoiding workload traffic from traversing the public internet while maintaining a cost-effective solution.

Previously, VMware Cloud on AWS workloads would access Amazon Simple Storage Service (Amazon S3) by leveraging gateway VPC endpoints in the connected AWS account, or through the public internet. With interface VPC endpoints, you can now access S3 directly and securely in Amazon Virtual Private Cloud (VPC) and the subnet.

This extends the functionality of existing gateway VPC endpoints by enabling you to access S3 using private IP addresses and securing the endpoint by associating a security group with the Elastic Network Interface (ENI) to control access.

Benefits to organizations include simplified hybrid cloud connectivity between VMware Cloud on AWS workloads and Amazon S3, and the ability to keep data flows private and secure.

Comparing Gateway and Interface VPC Endpoints

Let’s first describe endpoints before discussing the different ones available to you. A VPC endpoint is a network component that connects Amazon Elastic Compute Cloud (Amazon EC2) instances in a VPC to AWS services using private connectivity.

With a VPC endpoint, EC2 instances do not need a network access translation (NAT) device, virtual private network (VPN) connections, internet gateway, or AWS Direct Connect to communicate with supported services. They can communicate solely within the AWS network backbone on the global AWS infrastructure.

Figure 1 – VPC endpoints types.

Gateway VPC Endpoint

Gateway VPC endpoints were originally launched only for Amazon S3, with support for Amazon DynamoDB added later. A gateway VPC endpoint allows you to privately connect your VPC to supported AWS services without requiring an internet gateway, NAT device, or VPN connection.

A gateway VPC endpoint adds an entry in your route table that specifies a destination with the prefix list of the service and target with the endpoint id. This allows resources in your VPC to use their private IPv4 addresses to communicate with AWS resources outside of their VPC, and removes the need for public IPv4 addresses. Furthermore, traffic between your VPC and the endpoint does not leave the AWS network.

To secure the gateway VPC endpoint, AWS Identity and Access Management (IAM) policies can define permissions for an action regardless of the method you use to perform the operation.

Using Gateway VPC Endpoints with Amazon S3

This section describes the steps for connecting to Amazon S3 using a VPC gateway endpoint. We’ll cover the high-level configuration, route table, and firewall configuration required.

Figure 2 – Connecting to Amazon S3 using gateway VPC endpoint.

High-Level Configuration

1. A VPC is designated as the connected VPC during the SDDC creation process. ENIs are created in a specified subnet inside the connected VPC.
2. Deploy and configure the S3 bucket to the connected VPC to provide object-storage services.
3. Amazon S3 gateway endpoints provide private access to the storage gateway service and S3 buckets.
4. A VMware Cloud on AWS VPC is based on an account, which corresponds to a group or line of business subscribed to VMware Cloud on AWS services.
5. NSX Edge acts as the Tier-0 router that’s part of the SDDC and handles north-south traffic (traffic leaving or entering the SDDC, or between the management and compute gateways).
6. Configure the VMware Cloud on AWS compute gateway to allow traffic from the virtual machines (VMs) segment to the VPC and subnets where the S3 service is located.
7. ENIs in the connected VPCs provide high bandwidth and low latency access between VMs in the SDDC and native AWS services.
8. Using a route table within the VPC to control the traffic between the gateway endpoint and VMware Cloud on AWS SDDC, the traffic is going through the VPC router.

Next, we describe the overall process to configure access to S3 using a gateway VPC endpoint from the SDDC.

Once the gateway VPC endpoint has been created, a new entry is added to our route table.

Figure 3 – Gateway VPC endpoint is added to the route table.

From the VMware Cloud on AWS console, the compute gateway firewall will have to allow communication to the connected VPC and S3 prefixes.

Figure 4 – Gateway firewall set to allow communication.

By default, Amazon S3 access through the ENI in the connected Amazon VPC is enabled. If you disabled this access to allow S3 access through the internet gateway, you must re-enable it.

Figure 5 – Amazon S3 endpoint is set to ‘Enabled’ (by default).

Using Interface VPC Endpoints with Amazon S3

In 2018, AWS launched a feature called interface VPC endpoints powered by AWS PrivateLink. AWS PrivateLink is based on a technology called Hyperplane; a highly scalable distributed system used for managing connections.

Hyperplane is the same technology that powers AWS services such as NAT gateway and Network Load Balancers. Interface endpoints can be secured using resource policies on the endpoint itself and the resource the endpoint provides access to. Interface endpoints enable the use of security groups to restrict access to the endpoint.

AWS PrivateLink exposes elastic network interfaces within the virtual private cloud. When you create a PrivateLink endpoint for a service, it creates ENIs within the subnets you specify within that VPC.

This interface has a private IP that serves as an entry point for all traffic to any PrivateLink-managed service. Interface VPC endpoints are highly available, can deliver high network performance, and are managed by AWS.

API and HTTPS requests to Amazon S3 from your vSphere workloads running in VMware Cloud on AWS can be automatically directed through interface endpoints leveraging DNS resolution. Unlike gateway VPC endpoints, no changes to the route table are required.

Figure 6 – Connecting to Amazon S3 using AWS PrivateLink.

High-Level Configuration

1. A VPC is designated as the connected VPC during the SDDC creation process. ENIs are created in a specified subnet inside the connected VPC.
2. Deploy and configure the S3 bucket to the connected VPC to provide object-storage services.
3. Configure an AWS security group to allow communication between VMware Cloud on AWS subnets to Amazon VPC subnets where the S3 service will be provisioned.
4. An interface endpoint exposes an ENI with a private IP address within a subnet.
5. A VMware Cloud on AWS VPC is based on an account, which corresponds to a group or line of business subscribed to VMware Cloud on AWS services.
6. NSX Edge acts as the Tier-0 router that’s part of the SDDC and handles north-south traffic (traffic leaving or entering the SDDC, or between the management and compute gateways).
7. Configure VMware Cloud on AWS compute gateway to allow traffic from the VM’s segment to the VPC and subnets where the S3 service is located.
8. ENIs in the connected VPCs provide high bandwidth and low latency access between VMs in the SDDC and native AWS services.
9. Using security groups to control the traffic between the interface endpoint and VMware Cloud on AWS SDDC, the AWS PrivateLink traffic is going through the VPC router.

The security group needs to be configured to permit traffic from the VM workload’s network segment in the SDDC towards the subnet where the interface VPC endpoint’s ENI has been created.

Figure 7 – Security group configuration to be applied on S3 interface endpoint.

In the SDDC gateway firewall, we only need to allow traffic to the connected VPC.

Figure 8 – Gateway firewall set to allow communication to connected VPC.

Since we are not using the gateway VPC endpoint or public S3 access, the “Service Access” from the VMware Cloud on AWS console for S3 can be set to “Disabled.”

Figure 9 – Amazon S3 endpoint can be set to ‘Disabled’ or ‘Enabled.’

It’s important to know that private DNS names are currently not supported for Amazon S3 over interface VPC endpoints (AWS PrivateLink) when creating it. You can manually create an Amazon Route 53 Private Hosted Zone with an alias record and attach it to your VPC. All your traffic would flow automatically via that endpoint.

The following table gives an overview of the key differences for Amazon S3 access:

Security Best Practices

When you create an interface or gateway endpoint, you can attach an endpoint policy to it that controls access to the service to which you are connecting. Endpoint policies must be written in JSON format.

If you’re using a VPC endpoint to Amazon S3, you can also use S3 bucket policies to control access to buckets from specific endpoints, or specific VPCs. For more information, see Amazon S3 bucket policies.

Using VPC Endpoint Policies

A VPC endpoint policy is an IAM resource policy you attach to an endpoint when you create or modify the endpoint. If you don’t attach a policy when you create an endpoint, AWS attaches a default policy for you that allows full access to the service.

If a service does not support endpoint policies, the endpoint allows full access to the service. An endpoint policy does not override or replace IAM user policies or service-specific policies (such as S3 bucket policies). Rather, it’s a separate policy for controlling access from the endpoint to the specified service.

You can’t attach more than one policy to an endpoint, but you can modify the policy at any time. If you do modify a policy, it can take a few minutes for the changes to take effect. For more information about writing policies, review the Overview of IAM Policies in the IAM User Guide.

Your endpoint policy can be like any IAM policy; however, take note of the following:

• Your policy must contain a Principal element. For additional information related gateway VPC endpoints, see the documentation for endpoint policies for gateway VPC endpoints.
• The size of an endpoint policy cannot exceed 20,480 characters (including white space).

Endpoint Policies for Gateway VPC Endpoints

For endpoint polices that are applied to gateway VPC endpoints, if you specify Principal in the format”AWS”:”AWS-account-ID” or “AWS”:”arn:aws:iam::AWS-account-ID:root”, access is granted to the AWS account root user only, and not all IAM users and roles for the account.

If you specify an Amazon Resource Name (ARN) for the Principal element, the ARN is transformed to a unique principal ID when the policy is saved. To learn more, see the documentation for endpoint policies for Amazon S3.

Security Groups

When you create an interface endpoint, you can associate security groups with the endpoint network interface that’s created in your VPC. If you don’t specify a security group, the default security group for your VPC is automatically associated with the endpoint network interface.

You must ensure the rules for the security group allow communication between the endpoint network interface and the resources in your VPC that communicate with the service.

For a gateway endpoint, if your security group’s outbound rules are restricted you must add a rule that allows outbound traffic from your VPC to the service that’s specified in your endpoint. To do this, you can use the service’s AWS prefix list ID as the destination in the outbound rule.

For more information, see the documentation on modifying your security group.

Cost Considerations

There are two different AWS PrivateLink endpoints you can choose to use: interface endpoints and gateway load balancer endpoints.

You can create AWS PrivateLink endpoints to enable private connectivity to a service that’s either owned by AWS or owned by an AWS customer or partner. You will be billed for each hour your VPC endpoint remains provisioned in each AWS Availability Zone, irrespective of the state of its association with the service. See the AWS website for more details.

Such hourly billing for your VPC endpoint will stop when you delete it. Hourly billing will also stop if the endpoint service owner rejects your VPC endpoint’s attachment to their service, and that service is subsequently deleted. Such VPC endpoints cannot be reused and you should delete them.

Data processing charges apply for each Gigabyte processed through the VPC endpoint regardless of the traffic’s source or destination. Each partial VPC endpoint hour consumed is billed as a full hour. Irrespective of the association state of the VPC endpoint, you will incur data transfer charges if you send data to a VPC endpoint.

Conclusion

VMware Cloud on AWS provides more connectivity options available to consume AWS native services such as Amazon S3. In this post, we explained the differences between interface and gateway endpoints for connectivity to VMware Cloud on AWS. We also highlighted considerations around security and costs.

The VPC endpoints in the connected VPC provides low latency, high throughput, and seamless connectivity to VPC workloads and other AWS services. This extends the functionality of existing gateway VPC endpoints by enabling you to access S3 using private IP addresses and secure the endpoint by associating a security group with the ENI, thus reducing risk to your business.

Please contact us at AWS for support in implementing any of these architectures within your environments.