Update your AWS Transfer Family server endpoint type from VPC_ENDPOINT to VPC

In January 2020, AWS Transfer Family launched the VPC endpoint type, which gave customers the ability to host their Transfer Family server endpoints within their VPC using a single API call. If you were using Transfer Family with AWS PrivateLink before this launch, you are probably using the VPC_ENDPOINT endpoint type. After May 19, 2021, you will not be able to create a server using the VPC_ENDPOINT endpoint type in your AWS account if your account has not already done so before May 19. If you have already created servers with the VPC_ENDPOINT endpoint type in your AWS account on or before May 19, this change will not affect you. After May 19, new customers must use the VPC endpoint type.

The VPC endpoint type is functionally equivalent to the VPC_ENDPOINT endpoint type, but is more feature rich, and does not charge customers for use of AWS PrivateLink. As of today, additional features include the ability to attach Elastic IP addresses directly to endpoints to make them internet facing and use Security Groups for source IP filtering and host the endpoint in a shared VPC environment. The VPC endpoint type also supports FTPS/FTP protocols. There is no plan to add these features for servers with the VPC_ENDPOINT endpoint type. Additionally, new Region launches will no longer include VPC_ENDPOINT as an option.

In this blog, I discuss the transition and why you should migrate to the VPC endpoint type. To help you migrate, I also show you how to identify AWS Transfer Family servers using the VPC_ENDPOINT endpoint type. Finally, I cover a few viable migration patterns based on common scenarios, which include using Transfer Family for sharing data over public and private networks.

History of AWS Transfer Family’s AWS PrivateLink integration

In March 2019, AWS Transfer Family announced that AWS customers could configure use of Transfer Family within their Amazon Virtual Private Cloud (VPC), powered by AWS PrivateLink. This enabled AWS customers to configure a new Transfer Family server with a VPC_ENDPOINT endpoint type, which enabled SFTP connectivity without using public IP addresses or traversing the internet. To create a Transfer Family server, customers would make two calls. First, they would need to create a VPC endpoint. They would then pass the endpoint’s identifier as an argument to the Transfer Family CreateServer API call.

This brings us to Transfer Family’s next steps in maturing its VPC endpoint offering, and actions required by AWS customers moving forward.

What’s new

In 2020, with the support of the new VPC endpoint type, AWS Transfer Family discontinued the creation of Transfer Family servers with the VPC_ENDPOINT endpoint type from the AWS Management Console. Now, after May 19, 2021, Transfer Family will be discontinuing the creation of new Transfer Family servers with the VPC_ENDPOINT endpoint type from the AWS CLI and SDKs.

To take advantage of the pricing, additional features, and new releases, if you already own servers using the VPC_ENDPOINT endpoint type, we recommend you start using VPC as soon as possible. Transfer Family will not discontinue support for servers using the VPC_ENDPOINT endpoint type for customers who choose to continue to use it.

Identifying servers with the VPC_ENDPOINT endpoint type

In this section, I show you how to identify which servers are using the old VPC_ENDPOINT endpoint type, and then discuss ways you can change to using the new VPC endpoint type.

When you log into the AWS Management Console, and navigate to AWS Transfer Family, you can identify servers with the VPC_ENDPOINT endpoint type in the list of servers.

For customers using multiple AWS Regions, this can be a tedious and error prone process. For you, our recommendation is looping through each Region using the EC2 describe_regions and Transfer Family list_servers APIs, to pull server IDs with endpoint type set to VPC_ENDPOINT and corresponding Region. If you have many AWS Accounts, you could loop through your AWS Accounts with an IAM role with read-only auditor access if you authenticate via session profiles to your identity provider. The following is a simple example looping through your Regions:

import boto3

profile = input("Enter the name of the AWS account you'll be working in: ")
session = boto3.Session(profile_name=profile)

ec2 = session.client("ec2")

regions = ec2.describe_regions()

for region in regions['Regions']:
    region_name = region['RegionName']
    if region_name=='ap-northeast-3': #https://github.com/boto/boto3/issues/1943
        continue
    transfer = session.client("transfer", region_name=region_name)
    servers = transfer.list_servers()
    for server in servers['Servers']:
       if server['EndpointType']=='VPC_ENDPOINT':
           print(server['ServerId'], region_name)

How to change your endpoint type

As a customer there are a few ways to modify your existing AWS Transfer Family servers from VPC_ENDPOINT to VPC endpoint type. Note that you may have properties set to different values than the samples, and you will want to tailor the examples based on your individual needs.

Some standard scenarios for updating your Transfer Family server from VPC_ENDPOINT to VPC:

1. In-place update using AWS Management Console
2. In-place update using CloudFormation
3. In-place update using update_server API

Separate, more involved, scenarios I walk through in this blog include:

1. In-place update to retain Elastic IPs and firewall rules for internet facing Transfer Family servers
2. Blue/green deployment when sharing Transfer Family servers over AWS PrivateLink

Sharing data using AWS Transfer Family over internet

In this section and the next, I walk through two common customer scenarios that involve integrations between Transfer Family and other services in your AWS environment. Customers often want to securely share data using their Transfer Family server over a public internet connection. On this client side, customers required a static IP address associated with the Transfer Family server, so they could create firewall rules outbound from their private networks. On the Transfer Family server side, customers wanted to allow list only public IP addresses from those private networks.

Previously, with Transfer Family servers with the VPC_ENDPOINT endpoint type, using static IP addresses required the creation of a Network Load Balancer (NLB) to associate a static Elastic IP address per subnet to the Transfer Family server. Allow listing traffic required customers to set stateless VPC network access control list (network ACL) rules at the subnet level where the NLB was provisioned, which can be overly complicated for customers. Due to the NLB serving as a proxy, the client’s source IP is not preserved for logging and IP-based access controls.

With Transfer Family’s recent enhancements with the VPC endpoint type, customers no longer need to take on the burden of managing these additional networking resources, and can preserve the client’s source IP for logging into CloudWatch Logs and IP-based access controls with Security Groups.

The following architecture diagram represents a complicated customer setup using the no-longer recommended VPC_ENDPOINT endpoint type.

Now, Transfer Family servers using the VPC endpoint type can create an internet facing VPC hosted Transfer Family server. The Elastic IP addresses are attached directly to the VPC endpoint Elastic Network Interface (ENI). Traffic can now be allowlisted by means of stateful AWS Security Group firewalls or AWS Network Firewall service applied at the Transfer Family VPC endpoint level.

Using the VPC endpoint type, customers can simplify their setup by removing the NLB, target group, network ACLs, and a subnet unique to the NLB.

Customers with this configuration can migrate from the VPC_ENDPOINT endpoint type to the VPC endpoint type in a few short steps. Keep in mind, this change applies to customers using Elastic IP addresses, as opposed to public IP addresses assigned by AWS, which do not stay in your AWS account once disassociated with a resource. Please note, with NLBs, you can assign as many subnets as are available within your Region, whereas with VPC endpoints you can assign up to three; therefore, you can use a maximum of three Elastic IP addresses to migrate.

The steps are as follows. We recommend testing this in a pre-production environment before production.

1. (Recommended) Create new subnets with a public route table associated.
2. Create a security group and allow ingress source IPs based on existing network ACLs.
3. Stop your Transfer Family server.
4. Delete your NLB to disassociate the Elastic IP addresses.
5. Update your Transfer Family server to the VPC endpoint type and internet facing access, associate the new subnets and same Elastic IP addresses, and if desired set up a custom hostname.
6. Ensure that the new security group is assigned to your new VPC endpoint.
7. Start your Transfer Family server.

Your clients can now route through their firewall to the same Elastic IP addresses, and your server will allow connections from the same client IP addresses. Your Transfer Family server will show an Endpoint configuration specifying the subnets and Elastic IP addresses.

For cleanup, you will want to delete your NLB Target Group. Then, cleaning up the unused Elastic IP addresses is a two-part process. First, ensure that the client removes the Elastic IP address from their firewall if it will no longer be in use for that purpose. Second, you pay for unused Elastic IP addresses, so if you had more on your NLB than on your Transfer Family server, you should consider either re-associating them, or releasing them.

Please note if you change your DNS record from your NLB to your Transfer Family server, ensure your outage window accounts for the time-to-live (TTL) for the record to update fully.

Sharing data using AWS Transfer Family over private networks

In special cases, you may want to deploy two Transfer Family servers and redirect traffic from an old server to a new server, then later deprecate the old server. This may be useful for customers in regulated industries who provide their users’ access to Transfer Family over private connections using VPC endpoint services. These customers may want to reduce reboot downtime and have a change control plan in place to roll back the change. This approach assumes usage of a custom authorizer and a custom host key on your Transfer Family server; however, you can adjust it to work with any server configuration that uses a VPC endpoint.

For example, some customers have Transfer Family configured and privately shared in the following way:

Customers can deploy a new Transfer Family server of VPC endpoint type, get the new VPC endpoint ENI IP addresses, and create a Target Group. You can then attach the target group to the NLB to begin getting traffic from customers of the VPC endpoint service. This way there is a controlled means to roll back changes if necessary.

To prevent your Transfer Family server users from being notified to verify the authenticity of your SFTP-enabled server upon login, import the same host key you imported from the older Transfer Family server. Doing this also prevents your users from getting a warning about an unknown host, which is a warning to indicate a potential man-in-the-middle attack. See the documentation on changing the host key for your SFTP-enabled server for more details. If you did not import the host key for your older Transfer Family server, you cannot export the private host key from the Transfer Family server to migrate to the new Transfer Family server.

The steps to make this change are to:

1. Describe the existing Transfer Family server and VPC endpoint.
2. Create a new server with the VPC endpoint type, your existing host key, and attributes from the existing server and endpoint.
3. Once you have provisioned the server, describe the server to get the new VPC endpoint, get its ENIs, and then get its IP addresses.
4. Create the Target Group using the new IP addresses.
5. Adjust the NLB to route traffic to the new Target Group.

Here is a sample script for creating a new Transfer Family server. Note that you would need to edit this, adding in other parameters (service IAM role, identity provider details, etc.) based on your Transfer Family server configuration. This outputs the new IP addresses that you should use for your new target group. In addition, assuming you are keeping the same host key, you must add in the host key variable and create_server argument.

import boto3
import time

profile = input("Enter the name of the AWS account you'll be working in: ")
region_name = input("Enter the AWS Region you're working in: ")
server_id = input("Enter the AWS Transfer Family Server Id: ")

session = boto3.Session(profile_name=profile)

ec2 = session.client("ec2", region_name=region_name)
transfer = session.client("transfer", region_name=region_name)

group_ids=[]

transfer_description = transfer.describe_server(ServerId=server_id)
if transfer_description['Server']['EndpointType']=='VPC_ENDPOINT':
    # capture server attributes, you might need to add or remove attributes based on your server configuration
    transfer_vpc_endpoint = transfer_description['Server']['EndpointDetails']['VpcEndpointId']
    domain = transfer_description['Server']['Domain']
    identity_provider_type = transfer_description['Server']['IdentityProviderType']
    protocols = transfer_description['Server']['Protocols']
    security_policy_name = transfer_description['Server']['SecurityPolicyName']
    tags = transfer_description['Server']['Tags']
    # capture endpoint attributes
    transfer_vpc_endpoint_descriptions = ec2.describe_vpc_endpoints(VpcEndpointIds=[transfer_vpc_endpoint])
    for transfer_vpc_endpoint_description in transfer_vpc_endpoint_descriptions['VpcEndpoints']:
        subnet_ids = transfer_vpc_endpoint_description['SubnetIds']
        group_id_list = transfer_vpc_endpoint_description['Groups']
        vpc_id = transfer_vpc_endpoint_description['VpcId']
        for group_id in group_id_list:
             group_ids.append(group_id['GroupId'])
    # create new server passing server and endpoint attributes as arguments
    # to prevent disruption, you should add your host key as a variable and pass it in
    transfer_create = transfer.create_server(EndpointType='VPC',Domain=domain,IdentityProviderType=identity_provider_type,Protocols=protocols,Tags=tags,EndpointDetails={'SecurityGroupIds':group_ids,'SubnetIds':subnet_ids,'VpcId':vpc_id})
    new_server_id = transfer_create['ServerId']
    # wait for endpoint to provision
    time.sleep(180)
    # describe the server and get the endpoint id
    new_transfer_description = transfer.describe_server(ServerId=new_server_id)
    new_vpc_endpoint_id = new_transfer_description['Server']['EndpointDetails']['VpcEndpointId']
    # get the transfer family server's endpoint id, to get the ENIs, to print the private IPs you will use for your new target group
    vpc_endpoint_descriptions = ec2.describe_vpc_endpoints()
    for vpc_endpoint_description in vpc_endpoint_descriptions['VpcEndpoints']:
      vpc_endpoint_id = vpc_endpoint_description['VpcEndpointId']
      if new_vpc_endpoint_id == vpc_endpoint_id:
        for eni_id in vpc_endpoint_description['NetworkInterfaceIds']:
          eni_descriptions = ec2.describe_network_interfaces(NetworkInterfaceIds=[eni_id])
          for eni_description in eni_descriptions['NetworkInterfaces']:
            print(eni_description.get('PrivateIpAddress'))

In this sample, the old resources are intentionally not deleted for the sake of having a rollback plan. After the change, the old target group, VPC endpoint, and Transfer Family server with VPC_ENDPOINT endpoint type should be decommissioned so you do not incur cost of unused resources.

Conclusion

In this blog, I discussed the evolution of the AWS Transfer Family’s AWS PrivateLink offering. We discussed the advantages of the VPC endpoint type compared to the VPC_ENDPOINT endpoint type, and talked about how AWS is phasing out the latter after May 19, 2021. You can change the endpoint type for your server using the Transfer Family console, AWS CLI, API, SDKs, or AWS CloudFormation. We walked through how to identify a Transfer Family server with the VPC_ENDPOINT endpoint type, and examples of options to change your server.

You can now take advantage of the VPC endpoint type. With the VPC endpoint type, your architecture is more simplified, feature rich, cost effective, and you can benefit from new AWS Transfer Family feature releases.

Thanks for reading this blog post! If you have questions on changing your AWS Transfer Family server endpoint type, don’t hesitate to leave a comment in the comments section.