Gov-Cloud-Import Tool: How to Transfer Information Between Identity Boundaries

Note: S3 Buckets created are PUBLICLY EXPOSED. Please stay within your company’s security posture.

Amazon Web Services (AWS) has multiple Identity partitions: AWS, AWS GovCloud (US), and AWS China. You can see these represented in their ARNs (arn:aws, arn:aws-us-gov, arn:aws-cn). For security, services like Amazon Simple Storage Service (Amazon S3) do not have access to credentials beyond their boundaries, which can make it difficult to transfer information from inside one Identity boundary to another.

There is a new tool though, that reduces time and complexity for customers migrating their workloads into AWS GovCloud (US) – gov-cloud-import.

With gov-cloud-import, you have a web-based UI that allows you to import an AMI or Snapshot from AWS to AWS GovCloud (US) (up to 100GiB). Within the same tool, you can input an S3 bucket and an AWS GovCloud (US) Destination Bucket and perform one-way synchronizations (up to 1TiB).

Install the Script

To get started, you must first run the installation shell script. The installation script deploys two CloudFormation Templates, one in AWS and one in AWS GovCloud (US).

In AWS GovCloud (US), we deploy an S3 Bucket (for importing images only) and IAM resources to make the necessary API Calls for importing. And in AWS, we deploy Step Functions, EC2, Lambda, SSM, S3, and SNS. Step Functions will control the workflow as well as show progress. EC2 Worker is used for the actual transfer of images or synchronization of buckets. AWS Lambda is used for calling all other functions in the workflow. SSM parameter store securely keeps sensitive keys and other parameters necessary to the overall application. S3 houses the user interface to gov-cloud-import. SNS is used for notification at the end of an import.

Prior to install, remove the vmimport role from AWS GovCloud (US) if it exists. The script should be able to install with any BASH shell (tested on MacOS and Amzn Linux) and AWS CLI. You’ll need API Keys with admin privileges for AWS and AWS GovCloud (US). Installation takes a few minutes and will share the progress during that time. When finished, it will give you a URL to access the Web UI. See the sample install below. Please note the errors. These are from checking if the gov-cloud-import CloudFormation stack exists. If it doesn’t exist, the AWS CLI directs errors to stdout.

git clone https://github.com/awslabs/aws-gov-cloud-import.git
cd aws-gov-cloud-import
chmod +x gov-cloud-import-install.sh
./gov-cloud-import-install.sh

Using gov-cloud-import-image

Browse to the URL output by the installation script. You need keys that allow for lambda-invoke, lambda-list, and e2-describeRegions. Browse to the API Key page, enter keys, and click validate. Then click notifications if you would like to add your email or phone (sms) alerts.

For importing images, input an AMI or Snapshot ID. The AMI or Snapshot must be owned by the account and contain a single volume. While the input field checks the format of the string, it does not check permissions. Select the OS and Source/Destination Regions. Verify your input with a dialog box and import. You’ll receive a State Machine Execution ARN to the import job. Click to open in a new tab and watch the progress.

Using gov-cloud-import-s3

For importing S3 buckets, you must give gov-cloud-import permissions to the source and destination buckets. Download the sample policy and modify to fit your needs. Then click the link for permissions and install your policy inline to the us-goc-import-ec2role. Click on the second link for permissions and install your second inline policy for user gov-cloud-import-user.

Once permissions are set, proceed to the Import S3 page. From the first drop down menu, select your source. From the second drop down menu, select your destination. Verify your input and submit.

After submitting, a link to the S3 Sync logs will appear, which keeps a text log of time, source, and destination for each file synchronized.

Using gov-cloud-import via AWS SDK

If you want to build gov-cloud-import into your application, here are sample calls in Javascript that can be used with AWS SDK to start an import. Be sure to find the correct FunctionName as cloudformation suffixes randomize characters. These can be sent to us-west-2 or us-east-2 depending on which AWS GovCloud (US) Region (and adjacent AWS Region) you have installed and will use to import images.

For Importing Images:

function initImportImage(lambda){
    return new Promise((resolve, reject) => {
        let image = 'ami-1234abcd' /*AMI or Snapshot ID*/
        let os = 'Windows'/*Windows or Linux*/
        let region = 'us-east-1'/*Source AWS Region */
        //Params for Lambda invoke
        let params = {
            FunctionName : initStepFunction,
            InvocationType : 'RequestResponse',
            LogType : 'Tail',
            Payload : JSON.stringify({"image": image, "region": region, "os": os})
        };
        // Call the Lambda function
        lambda.invoke(params, function(err, data) {
            if (err) {
                reject(err);
            } else {
                resolve(data);
            }
        });
    });
}

For Importing from S3:

function initS3Import(lambda){
    return new Promise((resolve, reject) => {
        let comBucket = 'my-aws-bucket' /*Source AWS Bucket*/
        let govBucket = 'my-govcloud-bucket' /*Destination AWS GovCloud (US) Bucket*/
        //Params for Lambda invoke
        let params = {
            FunctionName : initS3Sync,
            InvocationType : 'RequestResponse',
            LogType : 'Tail',
            Payload : JSON.stringify({"source": comBucket, "dest": govBucket})
        };
        // Call the Lambda function
        lambda.invoke(params, function(err, data) {
            if (err) {
                reject(err);
            } else {
                resolve(data);
            }
        });
    });
}

For HTTPS Success/Failed Callbacks:

function initSNSSubscribe(lambda){
    return new Promise((resolve, reject) => {
        let snsProtocol = 'https',
        let snsTopic = '<gov-cloud-import-image or gov-cloud-import-s3>';
        let snsRegion = '<us-west-2 or us-east-2>';;
        let snsEndpoint = 'https://my.application.com/some/ping/back';
        //Params for Lambda invoke
        let params = {
            FunctionName : snsSubscribe,
            InvocationType : 'RequestResponse',
            LogType : 'Tail',
            Payload : JSON.stringify({
                "protocol": snsProtocol,
                "topic": snsTopic,
                "endpoint": snsEndpoint,
                "region": snsRegion
            })
        };
        // Call the Lambda function
        lambda.invoke(params, function(err, data) {
            if (err) {
                reject(err)
            } else {
                resolve(data);
            }
        });
    });
}

Success notification:

{
  "sourceRegion": "us-west-2",
  "source": "ami-0123abdc",
  "destRegion": "us-gov-west-1",
  "dest": "ami-wxyz9876" 
}

Failure notification:

{
  "sourceRegion": "us-west-2",
  "source": "ami-0123abdc",
  "destRegion": "us-gov-west-1",
  "dest": "failed"
}

Troubleshooting

First, look at the errors and logs directly in the State Machine. Lambda functions should give a high-level error in the exception panel near the top. To look further into the logs, click on the last Lambda Function executed in the State Machine. Under Step Details (near the top right), there should be a link to the Lambda Function, as well as one to the logs for the function. These logs contain more detail to explain any error that might have occurred.

For the EC2 Worker, if there are issues copying to AWS GovCloud (US), you may terminate and start a new instance. Application logs are kept in CloudWatch.

Scheduling S3 Synchronization

With the use of scheduled CloudWatch event rules, you can trigger the initS3Sync Lambda Function based on any schedule you need. Take note, be sure your schedule interval is greater than the time it takes to synchronize the bucket. While it can run in parallel, it will decrease performance or may produce failures. Remember to have your Bucket permission set first. See an example input below.

Caveats

AMIs may only have one (root) volume. The application ignores other volumes.
AMI or Snapshot must be < 100GiB.
S3 Buckets must be < 1TiB.
Windows imports lose .pem password decryption (Linux works). Set the admin credentials prior to import.

By using gov-cloud-import for the importation of AMIs, Snapshots, and S3 Buckets, you can bring your workloads and applications to AWS GovCloud (US).

Learn more about AWS GovCloud (US) and contact us with questions.

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox.

AWS Public Sector Blog