Lift and Shift migration of SFTP servers to AWS

Organizations across a variety of industries and geographical locations exchange data as part of their critical business workflows. Quite often, the operational burden of managing file transfer infrastructure and high capital expenses drains IT budgets. More importantly, it keeps those budgets from resourcing value added projects to differentiate businesses. To address these challenges, and unlock the data for deriving key insights, AWS Transfer for SFTP (AWS SFTP) was launched. AWS SFTP enables customers to replace their existing file transfer infrastructure with a fully managed service backed by storage in the cloud. One of the primary objectives of this service is to ensure that migrating this workflow should have no disruption to end users (SFTP client users). As such, we designed features to help “lift and shift” so customers can migrate to the service without requiring any changes to their applications or account configuration.

From the SFTP client’s point of view, a server’s “identity” consists of three elements: its DNS name, its IP address, and its SSH host key. In this post, I talk about how you can bring along as much of your existing server’s identity as you need during this migration.

DNS portability

The simplest lift and shift scenario is when you simply create an AWS SFTP server, and a DNS CNAME alias pointing at your new server’s endpoint. You can create the CNAME record with your current ISP or via Amazon Route 53 right from the AWS SFTP console:

Before publishing the new DNS record, be sure to create an account for each of your users on the new server. How you accomplish this depends on whether you’re using the service to manage end-user identities or an identity provider of your own (custom) for authentication and access.

After the credentials have been imported from your on-premises server, you can set up a folder to mirror the existing home directories. You can either do this in the Amazon S3 bucket itself, or use AWS SFTP’s “HomeDirectoryMappings” feature to create a logical directory structure for each SFTP user. Either way, when users connect with their SFTP clients, they see the same familiar environment.

IP address portability

If an SFTP user is behind a firewall with strict whitelisting, DNS portability may not be sufficient; the actual IP address associated with the server must be preserved. This is possible in AWS SFTP to the Bring Your Own IP Address (BYOIP) feature of Amazon EC2 Networking. This lets you bring public IPv4 addresses from your on-premises network to your AWS account. You continue to own the address, but AWS advertises it on the internet.

There are several steps to binding your existing IP address to your new server:

1. Import the IP addresses into your VPC, as described in the BYOIP documentation.For high availability, AWS best practice is normally to use multiple IP addresses, one in each Availability Zone. This may or may not be appropriate given the constraints of your existing user base.

2. If necessary, create a new VPC, with the required number of subnets. Additionally, since your server is going to be accessible from the internet (“internet facing”), the VPC must have an Internet Gateway attached to it. Additionally, the routing table for the subnets must forward through that Internet Gateway:

3. Create an internet-facing server whose endpoint is hosted in your VPC. If you’ve already created the server, you can stop it first and then change its endpoint type.

4. On server creation, your VPC’s default Security Group is automatically assigned to the endpoint. You can modify the security group associated with your endpoint to add source IP addresses of clients that can access it. If you are unable to connect to your server, check whether the rules specified in the security group or the NACLs around your endpoint’s subnet are permitting client access.

Note: AWS SFTP listens for SFTP connections on the standard SSH port (TCP/22). If your existing on-premises server listens on a different port, and you must preserve that behavior, you can do so by following the instructions here. You must ultimately change the listener port using a Network Load Balancer as a bridge between the public internet and your server’s VPC endpoint.

Host Key Portability

This brings us to the third and final piece of your old SFTP server’s “identity,” namely its SSH host key.

SFTP clients typically remember a server’s public key when they first connect to it. If the key ever changes, it may indicate an insecure connection, and the client issues a terrifying warning to the effect of:

$ sftp -i wayne-transfer-key1 wayne@my-sftp-server.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uRrhorK4Le+WPBS1ihytaHDHerBB+ZQi2Uq4kkuOrlU.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Offending RSA key in ~/.ssh/known_hosts:145
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Permission denied (publickey,password,keyboard-interactive).
Couldn't read packet: Connection reset by peer
$

AWS SFTP allows customers to create or update servers with a host key of their choice. (If you don’t provide a host key, AWS SFTP creates one when it creates the server.) The update host key operation is not available in the AWS SFTP console, but it can be done via the API or CLI:

$ aws transfer update-server --server-id s-9b87ba0ffcee4d158 \
        --host-key file://old-server-host-key
{
    "ServerId": "s-9b87ba0ffcee4d158"
}

Note: The private portion of any RSA key pair must be carefully protected. This is especially true of host keys. Should a host key ever be compromised, rotation would be disruptive to all users. For this reason, it’s a good practice to delete any temporary copies of the key file as soon as it is uploaded and verified:

$ aws transfer describe-server --server-id s-9b87ba0ffcee4d158 \
        --query Server.HostKeyFingerprint
"SHA256:FU8AqSf/Cu/wxT810IZq5DA74uYV4CGBKuGpsSiz4NY="
$ ssh-keygen -l -f old-server-host-key
2048 SHA256:FU8AqSf/Cu/wxT810IZq5DA74uYV4CGBKuGpsSiz4NY
$ rm old-server-host-key

You may want to keep a copy of the private key in a safe place. Once uploaded, the private key itself cannot be retrieved from the AWS SFTP service.

With the old private key installed, the lift and shift is complete. Existing users continue to connect to your server just as they always have, unaware that the server has migrated to the cloud:

$ sftp -i wayne-transfer-key1 wayne@my-sftp-server.example.com
Connected to my-sftp-server.example.com.
sftp>

Summary

In this post I showed you how to carry over much of your existing server’s identity during a migration of SFTP servers to AWS. This includes your own hostname, server endpoint IP address, and host key. Being able to bring all these with you allows seamless migration of your SFTP server(s), without any disruption to your SFTP users. They continue to connect and access the files just as they always have, unaware that you have successfully moved your SFTP server to AWS. With the data now available to you in Amazon S3, you have a number of options in AWS to unlock this data for processing, analytics, AI/ML, and more! Get started today, by visiting the AWS Transfer for SFTP Console. Comment below with any questions, and thanks for reading!

This post is an updated version of a blog post originally published on May 23, 2019.