Automating AWS Migration and Reducing Issues with CloudEndure Scripts

By Wally Lu, Sr. Consultant, Infra Architect at AWS

Many Amazon Web Services (AWS) customers use tools like CloudEndure, a company acquired by AWS earlier this year, to migrate their on-premises servers to AWS.

CloudEndure has some unique features you won’t find elsewhere, including the ability to migrate servers from any source platform to any destination platform. The machine conversion process does all the hypervisor and OS configuration changes for you.

CloudEndure is a block-level replication, supporting almost any application. It uses continuous replication, which means any change on the source machine will be replicated to the staging area in less than a second. This minimizes downtime during cutover.

In this post, I will show you how to automate AWS migration with CloudEndure, and provide you with an automation script. With this script and YAML configuration file, you can manage the AWS migration as code, relaunch your machines in just a few minutes, and avoid a lot of potential mistakes during the cutover process. Managing the migration as code will also be easier for auditing and reporting purposes.

Prerequisites

CloudEndure is a great tool to help you migrate workloads, but there are some challenges if you use the console to migrate a large number of servers, such as:

• You need to update blueprints for your servers one by one, which takes a long time to do a peer review.
• You need to rebuild the environment with the same blueprint for all the servers.
• You can’t force the user to migrate to the test environment before migrating to the cutover environment.
• You may want to stop the cutover task if replication lag is over 180 minutes, but there’s no such option in the CloudEndure console.
• You may want to remove the migrated servers from CloudEndure but do not want to remove any other servers by mistake.

Before getting started, ensure you have the following:

• Experience with the CloudEndure service, installing the agent, and using it from the console. Learn more about how to use CloudEndure.
• CloudEndure account created and licenses purchased.
• CloudEndure project created and AWS credential configured.
• CloudEndure agent installed on the source machines.
• Download the Python scripts and configuration files from the Github repository:
  • config-cutover.yml – The config file for the production environment.
  • config-test.yml – Config file for the test environment.
  • config-project.yml – Config file for project replication settings.
  • CloudEndure.py – Master Python script to drive the orchestration process.
  • CheckMachine.py – Python script to check the status of the machine before migration.
  • LaunchMachine.py – Python script to launch the machine in a test or production environment.
  • Machine.py – Python script to call three other functions: check status, update blueprint, and launch target machines.
  • StatusCheck.py – Python script to check the migration status of the target machine.
  • UpdateBlueprint.py – Python script to update the machine blueprint, such as security group, subnet ID, tags, etc.
  • UpdateProject.py – The Python script to update settings for the replication server, such as encryption key, security group ID, and subnet ID.
  • Cleanup.py – Python script to remove machines from CloudEndure after cutover.

Migration Scripts Features

The CloudEndure script provides the following core features:

Update CloudEndure Replication Server Settings

CloudEndure uses replication servers in your account to replicate the data from the source machine to the staging area in that account. In the CloudEndure console, you can select subnet and security group ID for the replication server, and you can encrypt the volume with a KMS key.

Figure 1 – CloudEndure project replication settings.

This script can help you update replication server settings as well. You just need to update the replication section in the config-project.yml file. Figure 2 shows a YAML configuration file, and there are three keys: encryptionkey, subnetID, securitygroupIDs. Please replace the value with the correct one in your account.

Figure 2 – Config file for project replication settings.

Migrate Servers to the Test Environment

There are two types of server launches: test and cutover. Test mode is for the customer to launch machines in the test environment and make sure everything looks good before launching servers in cutover mode, which will be in the production environment.

In Figure 3, you can see that launch type parameter accepts two values: test and cutover.

Figure 3 – Migration mode.

Migration Protection

The script will return an error if one of the following conditions is met:

• Launch type is cutover but the server has not been migrated to the test environment.
• Launch type is cutover but the server has been migrated to the production environment already.

In Figure 4, the script will check “lastTestLaunchDateTime” and “lastCutoverDateTime” attributes. The “lastTestLaunchDateTime” value will be none if the machine has not been tested, and the “lastCutoverDateTime” will be none if the machine has not been migrated to production.

This forces users to migrate to the test environment before cutover.

Figure 4 – Migration protection.

Dry Run: Syntax Check for the YAML File

The dry run feature gives you the ability to validate all your configurations in the YAML file and can save you a lot of time during cutover because syntax errors could take a while to troubleshoot. Using the –dryrun parameter in the script will force the CloudEndure to update blueprints for all of the machines in the configuration file and revert back, which ensures that CloudEndure will accept blueprint updates during cutover.

Clean Up Servers After Migration

After cutover, you may want to remove machines from CloudEndure. One way to do this is to login to the console and select the right machine to remove. However, using the script is much easier, because you can use the same YAML configuration file for cutover.

In Figure 5, the script calls delete method to remove machines from CloudEndure project.

Figure 5 – Migration clean up.

The scripts also check for the following errors, and will fail and return a proper error message if the user tries to:

• Clean up servers before cutover completes.
• Migrate a server with more than 180 minutes lag.
• Use an incorrect machine name or project name.
• Launch a machine that hasn’t completed the initial replication.

The Deployment Process

Let’s use this script to launch instances in AWS via CloudEndure. First, we need to prepare a configuration YAML file because the script will read all the server information from the YAML file and update the CloudEndure machine blueprint accordingly. Next, we’ll launch these machines in the test environment and make sure everything works as expected. Then, we’ll launch these machines in the production environment, and finally we will clean up the migrated machines from CloudEndure.

Please note that before you do anything in this section, make sure the CloudEndure agent is installed on the source machine and the machine has finished replication. You can check the replication status in the CloudEndure console.

Step 1: Prepare Configuration Files and Get the API Token

In this step, we’ll prepare three YAML files. First is Config-project.yml, which is used to update project-level replication settings. The second is Config-test.yml, and you’ll need to add your servers to this file and update the subnet and security group IDs for the test environment.

The third file is Config-cutover.yml, which is used for the production environment. You will need to change the security groups and subnet IDs to the one in your production VPC, and this file is similar to the Config-test.yml. The only difference should be the subnetIDs and securitygroupIDs attributes.

So log in to the CloudEndure console, click Setup & Info and choose Other Settings. Next, scroll down to the bottom of the page and you’ll see the API token. Do not use the installation token, which is for agent installation only. If the API token does not exist, click the Generate New Token button at the right bottom.

This is how the API token looks like:

Next, remove the hyphens and get the new token, which you will use in the script later. In this case, the token looks like this—6405A45D0773DC4C.

Now, let’s update the configuration files for your target machine:

• Config-cutover.yml: For the cutover environment, you need to update machinecount, machineName, subnetIDs, securitygroupIDs, instanceType, iamRole (optional, leave as “None” if not required) and tags.
• Config-test.yml: For the cutover environment, you need to update machinecount, machineName, subnetIDs, securitygroupIDs, instanceType, iamRole (optional, leave as “None” if not required) and tags.
• Config-projects.yml: For the replication settings, you need to update encryptionkey, subnetIDs, and securitygroupIDs attributes for the replication server.

Note that ‘machinecount’ is the number of machines in this config file. If you migrate five machines, change the number from two to five. Also, ‘count’ is the number of tags for the target machine. The default value is three. If you add 10 tags, you need to change count from three to 10.

Step 2: Launch Machine in the Test Environment

In this step, we will use the configuration file prepared in the Step 1 to launch a machine in the test environment. First, we’ll update the CloudEndure project replication server settings as needed. We’ll then validate the YAML syntax with dry run, and finally launch the machines in the test environment.

It’s optional, but you can use default CloudEndure replication server settings. If you want to change them, however, run the following command to update project settings. Please replace <APIToken> with the token prepared in Step 1 and replace <project name> with your CloudEndure project name.

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-project.yml –updateproject yes

Next, run the following command to validate YAML file syntax:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-test.yml –launchtype test –dryrun

You will get a message like this for each of your servers:

Figure 6 – Dry run result.

Now, run the following command to launch the machines in the CloudEndure test environment:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-test.yml –launchtype test

Figure 7 – Migration test job.

After the third command, you will see the output as shown in Figure 8. Click the Job Progress button, and you should see the jobs for all the machines in this configuration file. Each job will take 10-15 minutes to complete. The other way to check the job status is to run the following command:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-test.yml –launchtype test –statuscheck Yes

You should see “XYZ has been migrated to the TEST environment” after the job complete.

Figure 8 – Migration testing status.

After testing, let’s continue to migrate the servers to the production environment.

Step 3: Launch Machine in the Production Environment

After testing, we can use a similar method to migrate servers to the production environment. In the example below, the only thing you need to change is the –launchtype and –configfile name, because launch type is cutover now and the configuration file should be cutover configuration file as well. You will get a similar output as Step 2, so I will not add more screenshots in this step.

First, run the following command to validate YAML file syntax:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-cutover.yml –launchtype test –dryrun

Next, after validation, run the following command to launch machines in the AWS production environment:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-cutover.yml –launchtype cutover

Again the launch process will take 10-15 minutes to complete. You can run the following command to check the status of migration or check the job progress on the console:

Python CloudEndure.py –userapitoken <APIToken> –projectname <project name> –configfile config-cutover.yml –launchtype cutover –statuscheck Yes

You should see “XYZ has been migrated to the PROD environment” after the job is complete.

Figure 9 – Migration cutover status.

At this stage, you should have all the servers running in the AWS production environment, and we can continue with server-level and application-level testing. However, this is not the last step of this cutover because we need to remove unnecessary machines from CloudEndure to avoid confusion in the future.

Step 4: Clean Up Machines After Migration

After migration, run the following command to remove machines from the CloudEndure console. The command below will clean up machines from the CloudEndure console only and will not touch machines in AWS.

Python CloudEndure.py -userapitoken <APIToken> -projectname <project name> –configfile config-cutover.yml -cleanup Yes

You should see “XYZ has been removed from CloudEndure” after the job is complete.

Figure 10 – Migration clean up result.

Summary

In this post, I demonstrated the process to use a python script to automate CloudEndure migration.

First, I listed some of the challenges if we use the CloudEndure console to do the migration. Next, I explained the key features of this automation script. Then, I demonstrated the deployment process, from preparing the YAML configuration to launching instances in the test environment and launching instances in the production environment. Finally, I demonstrated how to clean up the machines in CloudEndure after cutover.

With this automation approach, you’ll be able to migrate much faster and improve the success rate of the entire migration. If you want to learn more about CloudEndure API or develop your own features, check out the CloudEndure API documentation.