Using VPC hosted endpoints in shared VPCs with AWS Transfer Family

AWS customers can now deploy AWS Transfer Family server endpoints in Amazon Virtual Private Clouds (Amazon VPCs) with shared resources. Amazon Virtual Private Cloud (VPC) sharing, which launched in early 2019, enables AWS customers to share specific resources with other AWS accounts within the same AWS Organization. VPC sharing benefits customers by enabling:

• Role-based separation of authority
• Ownership of certain workflows by certain business units
• Avoiding certain limits that are capped on a per account basis
• Cost optimization through reuse of NAT gateways, VPC interface endpoints, and intra-Availability Zone traffic

You can deploy an AWS Transfer server using publicly accessible endpoints or VPC hosted endpoints. You can use Public endpoints to quickly and easily provide internet access to your AWS Transfer Family servers. Alternatively, you can use VPC hosted endpoints for greater control over how users access your SFTP, FTPS, and FTP servers. In a recent blog post, we discussed using IP filtering, via an allow list, in combination with VPC security groups and VPC hosted endpoints. Thanks to our newly launched feature, you can now deploy a Transfer Family server endpoint within a shared VPC. Before this launch, you could only deploy the Transfer Family server using the default security group. This prevented you from hosting the endpoint within a shared VPC, as shared VPCs have no default security groups.

In this blog, we walk you through using this new feature to deploy a server VPC endpoint in a shared subnet. First, we walk you through the architectural components of the sample deployment scenario. Next, we demonstrate how to configure this architecture in detail. Finally, we review how to test that your architecture is configured correctly.

Architecture

The following diagram shows the key components to deploy a secure server endpoint in a VPC with shared subnets. In this example, access to this server is filtered using an allow list when clients connect over the internet. This architecture comprises two accounts, the owner account and participant account. The owner account acts as the shared services account by sharing two subnets with the participant account. We deploy a server endpoint in each subnet shared by the owner account using Elastic IPs owned by the participant account. A security group is used to control access to the server. A VPC internet gateway allows access to the endpoint via the internet.

In the AWS Organizations service, you must enable sharing from the management account for your organization. In this architecture, the AWS Organizations owner account is the shared services account. When you create your server from a participant account, you select a VPC created by an owner account, which has subnets shared with the participant account using AWS Resource Access Manager. As the participant, you allocate two Elastic IP addresses that you associate with your server endpoint. Then select an existing security group, or create a new security group at the time of endpoint creation (or create a security group separately in the VPC console) from the participant account. You can then control which clients can access your VPC hosted endpoint by creating an allow list of individual IP addresses or ranges of IP addresses.

In the next section, we walk you through each of these steps in detail.

Setup

To get started, connect to the management account for your AWS Organizations, select Settings, and enable sharing for your organization. Then choose Save settings.

You must also have an owner account, which deploys your VPC and shares its resources, and a participant account, which deploys your server. Both the owner and participant account must belong to the same organizational unit, otherwise resource sharing will not work. Once enabled, log in to the account you intend to be the owner account of your shared VPC. From the owner account, deploy a VPC that contains two subnets (from different Availability Zones), and an internet gateway. With these resources created, create a resource share. To create a resource share, go to AWS Resource Access Manager in the AWS Management Console. Select Create resource share.

For Name, enter a name for your resource share.

In the Resources – optional section, select Subnets from the drop-down. Check the selection boxes next to the subnets you created in your VPC.

In the Principals – optional section, enter the AWS account number of the account from which you are creating your server.

Once this is configured, select Create resource share to complete this step.

Next, from another console session, log in to the participant account with which you shared your subnets in the previous step. In order to create an internet facing endpoint that is VPC hosted, you must first allocate two Elastic IP addresses. Earlier you created a VPC in the owner account. In the same Region, go to the VPC service in the AWS Management Console, and select Elastic IPs. Select Allocate Elastic IP address, and select Allocate.

Next, go to the AWS Transfer Family console and choose Create Server. Then, under Choose protocols, we check SFTP (which we are using in this example), and then select Next.

Then, for Identity provider type, select the radio button for Service managed, and select Next.

Then, under Endpoint Configuration, select VPC hosted for a VPC hosted endpoint. In this exercise, you are creating an Internet Facing server, so select that option. From the VPC drop-down menu, select the VPC with the ID you created and shared from your owner account.

Once you have selected your VPC, notice that the two Availability Zones available are the ones your AWS CloudFormation template selected to create your subnets in the owner account. Select each of those, and then select your Subnet ID in the left drop-down menu and one of your Elastic IPs in the right drop-down menu for each IPv4 Address.

Now is time to take advantage of our newly released feature that allows you to create a security group, or select an existing security group when deploying a server. Under Security Groups, select Create a security group, which launches the Security Groups console page in a new tab.

Note: Our new feature also allows you to select an existing security group, meaning you could create a security group in advance, and select it at the time of server creation.

On the Create security group page, under Basic details, supply a Security group name and a Description. From the VPC drop-down, select the owner account VPC, and then select the subnets. Additionally, under Inbound rules, now would be a good time to allow an IP from which you will later test your connection. Assuming you plan to test from your current IP, select Add rule, then under Type, from the drop-down select SSH (SFTP uses the SSH protocol to transmit data), and for Source, select My IP.

Next, select Create security group to complete this step.

Return to the tab or window for the console session where you were creating your server, and select the arrow to refresh the list under Security Groups. From the drop-down, you should now be able to select your newly created security group, and select Next to proceed to the next step.

On the Configure additional details page, select Next to accept the defaults for this example. On the Review and create step, you can review all of your configuration details, then select Create server to complete the creation of your server.

It takes a few minutes to create your SFTP server. When the creation is complete, and the server status shows as Online, select the new server under Server ID to get more information.

Test

You can test access to your server either via your terminal on Linux or macOS systems, or using a third-party tool such as Cyberduck, WinSCP, or Filezilla. Before attempting to connect to the server, you must first return to the AWS Transfer Family console for your server to create a user account.

Note: Our example uses a service-managed identity provider for the server, which authenticates users using SSH keys. AWS Transfer Family also supports custom authentication methods, which allows you to do password authentication, in addition to authentication via third-party providers. With custom authentication, you can deploy a server that supports both SFTP and FTPS.

Once you’ve created a user account, you’re able to attempt connecting to your server using the private key that corresponds with the public key used during user creation. Using the hostname of your SFTP server, try to connect using your preferred SFTP client. Since you allowed your IP during Security Group creation, you should be able to connect and authenticate to your server.

Automating this setup

It is important to note, that while this tutorial focused on using the AWS Management Console, we also support deployment via API. Additionally, customers may want to leverage AWS CloudFormation templates to associate previously created security groups during server creation. For more information on how to create a security group with AWS CloudFormation, follow this link.

AddressAllocationIds:
- String
SecurityGroupIds:
- String
SubnetIds:
- String
VpcId: String

Cleaning up

To clean up the resources you created as part of this post, first delete your server. Once the server has been deleted, from the participant account, delete your elastic IPs. From the owner account, delete your resource shares. Additionally, from the owner account, delete the VPC you created earlier to host your shared resources.

Conclusion

In this blog, we extended on the concepts introduced in our previous blog on VPC hosted endpoints. We showed you how to deploy an AWS Transfer Family server endpoint in a shared VPC. First, we ensured that we enabled sharing for our AWS Organization. Then, from the owner account, we created a VPC, created two subnets within the owner VPC, and shared those subnets with the participant account using AWS Resource Access Manager. Next, from the participant account we assigned two new Elastic IPs, and used those to create an AWS Transfer Family server VPC hosted endpoint in our shared subnets from the principal account. We walked through using our new feature to create a security group at the time of server creation, and how to filter access to your server via an allow list.

Using the new features described in this post, you can now deploy server endpoints in shared VPCs. This allows you to use AWS Transfer Family in an AWS Organization, potentially reducing the overall complexity of your networking and deployments in cross account scenarios. Additionally, being able to host AWS Transfer Family server endpoints in resource sharing situations enables customers to potentially run fewer servers overall. Not only does this reduce cost, but it can also simplify their overall strategies around AWS Transfer Family.

Thanks for reading this blog post, please leave a comment if you have any questions. Also, remember to join us on November 10, 2020, for the AWS Storage Day virtual event, to learn what is new across the AWS Storage portfolio. To learn more about AWS Transfer Family, check out the following links:

• AWS Transfer Family product page
• AWS Transfer Family documentation
• AWS Transfer Family now supports shared services VPC environments
• Creating a server in a virtual private cloud