Manage EC2 Launch Templates at scale during rehost migrations with AWS MGN

Many customers rehost their environments to AWS using AWS Application Migration Service (MGN), and have to prepare future-state configuration for every server they are migrating within each migration wave. When a migration wave includes multiple servers, customers have to manually configure multiple EC2 Launch Templates in the AWS console. For each server migrated using AWS MGN, you need to validate and configure parameters in two sections: first in General launch settings and then in EC2 Launch Template section. Then, you have to mark the latest version of EC2 Launch Template as default for the changes to take effect during the new instance launch.

Even for a Proof-of-Concept (PoC) project with just a few servers in scope managing these configurations might provide a challenge. For any migration project with a scope from a few servers to a dozen of servers per migration wave this becomes a real issue. Note that large migrations covering 100 and more servers would require a scalable migration orchestration solution such as Cloud Migration Factory on AWS (AWS CMF). AWS CMF doesn’t require managing target servers’ configurations in the AWS Console. For more information on best practices for large-scale migrations, please see AWS large-migration strategy and best practices – AWS Prescriptive Guidance.

This blog post will focus on solving the challenge of managing multiple future-state configurations for any migration project managed directly through AWS console, regardless of its scale. We will provide three different options to automate the management of both General launch settings and EC2 Launch Template configurations:

Option 1: use the new feature of AWS MGN service, launch template (as per What’s New post and AWS MGN Release Notes), to set default configurations at an account/region level, a common template that will be used for all new source servers added for the migration
Option 2: manage target servers’ configurations by a script, with the ability to copy templates between servers or clone a few parameters from one template to several others (i.e., change the target Subnet ID, Security Group ID, or instance type)
Option 3: import all target servers’ configurations from a CSV file with a script, using a data prepared during the discovery and assessment phase.

Each of these options could be used independently or together, depending on the migration scenario. Below we will explain how to use these options in different use-cases.

Solution Overview

The General launch settings and EC2 Launch Template determine the configuration of the launched server, including its instance type, volume types, security groups, target launch subnet, etc. The new launch template feature of AWS MGN simplifies the problem described previously using an account/region level default settings for both General launch and EC2 Launch Template settings and helps prevent possible manual misconfigurations.

In addition to the default launch template that sets the same values for every migrated server, we will provide two scripts to automate individual per-server configurations:

a management script to further fine-tune these parameters for a selected subset of servers or individual server (in the section “Option 2 – Manage General Settings and EC2 Launch Templates with a script”)
an import script to set all target servers’ configurations at once from a single CSV file (see the section “Option 3 – Import General Settings and EC2 Launch Template parameters from a CSV file”).

Option 1 – AWS MGN Default Launch Template configuration

At re:Invent 2022, AWS announced the availability of the launch template feature that allows to configure default parameters for target servers at an AWS account/region level. In the examples below we are replicating two servers that are part of the same application. Although the servers have different CPU/RAM parameters and might require different instance types, other parameters such as target Subnet ID, Security Group IDs, EBS volume types, licensing type, and right-sizing recommendations are typically stay the same across multiple servers and need to be set to default values before the migration starts. Each new source server added to the AWS MGN console will inherit the settings from the account/region-level default template first, and then can be adjusted individually on a per-server basis.

To access the new default “Launch template” section, go to the AWS MGN console, then select “Settings” on the left side navigation menu (press “Get started” button first if AWS MGN was not previously configured in the account/region). For more details on how to initialize and configure AWS MGN service, follow one of these blog posts:

Under “Settings” menu on the left, go to the second “Launch template” tab (see Figure 1), and click Edit.

Figure 1. AWS MGN default settings – Launch template tab

Here, we can see several sections that we can configure default parameters for:

“General launch settings” (see Figure 2),
“Default EC2 Launch Template” (see Figure 3)
“MAP program tagging” (see Figure 4)

Figure 2. AWS MGN default settings – General launch settings section

Figure 3. AWS MGN default settings – Default EC2 Launch Templates section

Figure 4. AWS MGN default settings – Map program tagging section

Note, in order for AWS MGN to use the Default instance type, which we will specify on this page in the “EC2 Launch Templates” section, we must also uncheck “Activate instance type right-sizing” (as shown in the Figure 2) in the “General Launch Settings” section.

Once we save the template, any new server added to the AWS MGN service will use the template default values we just set. AWS MGN service will not change these parameters for servers that are already added to the list of source servers before we made this update. This provides a certain baseline and allow us to group similar workloads in a migration wave that utilize the same parameters and allow an iterative approach such as:

define migration wave of servers with similar target configurations;
create default launch template settings for that migration wave;
migrate (add source servers to the AWS MGN console and go through the migration lifecycle)
move to the next group of servers and repeat.

However, if we need to change any of the settings for individual servers, we can do that by navigating to the “Source servers” menu of the AWS MGN console, choosing the server’s name we need to modify settings for, and opening the “Launch settings” tab as shown in the Figure 5 below. Any changes made here will only affect this particular source server and its future-state configurations that will be used during the launch of test or cutover instances.

Figure 5. AWS MGN individual source server Launch settings configuration

Option 2 – Manage General Settings and EC2 Launch Templates with a script

While the using default launch template can help you set a standard template for all new servers, there is typically a need to customize individual servers’ launch templates even further. These could include scenarios like:

Change target subnet or security group(s) after the test launch is complete, to use different settings for cutover launch
Change subnet or security group(s) for a subset of servers within the same migration wave. This could include use cases where you need application servers to be migrated to a public subnet while database servers to be migrated to a private subnet, or just to use different private subnets per different components of the same application. Another use. case would be a need to assign a specific instance type to database servers, etc.

In such cases you have two options:

Use already added to the AWS MGN source server, and configure this individual source server’s launch settings inside of AWS MGN console (both “General settings” and “EC2 Launch Template”)
Under EC2 console, create a new EC2 Launch Template from scratch, and use it as a “template” for future migrated servers, so that certain parameters can then be copied to new source server’s EC2 Launch Template using the CLI script below, after these new servers are added to the AWS MGN console

This CLI python script (available here) allows you to specify either of these two options as the “source” for the parameters to be copied from, and you can then apply it to a group of replicating servers in AWS MGN. Below we will illustrate this with two examples:

Example 1 – Copying from existing Source Server’s template

In this example we have 3 application servers and 3 database servers replicating in AWS MGN. We have defined the EC2 Launch Template for app server s-A1 and database server s-DB1. Server s-App1 uses subnet “public-subnet-1” and security group “public-SG-1”. Similarly, server s-DB1 uses subnet “private-subnet-1” and security group “DB-SG-1”.

Now, we will copy the EC2 Launch Template from server s-App1 to the other 2 app servers, s-App2 and s-App3. And for the 2 DB servers, s-DB2 and s-DB3, we will copy the EC2 Launch Template from server s-DB1. This can be achieved by running the following CLI command:

python target-templates-update --target s-App2,s-App3 --source-server s-App1
python target-templates-update --target s-DB2,s-DB3 --source-server s-DB1

Note: We have used s-App1, s-DB1 to differentiate between different servers in the example above. The actual server id in AWS MGN is in s-0123456789abcdfa format.

Here is a sample run in our account where we updated the template for 2 servers based on a third one:

Figure 6. An example of “target-templates-update” script cloning configuration from 1 source server to 2 target servers

Example 2 – Copying from existing EC2 Launch Template

In this example we have configured an EC2 Launch Template, lt-a1b2c3097ecb147e2, that defines the settings for application servers. We can run the following command to update target server settings for all 3 application servers based on this source template:

python target-templates-update --target s-App1,s-App2,s-App3  --template-id lt-a1b2c3097ecb147e2

Similar to the example above, we might need to update only “Subnet-Id” and “InstanceType” from the launch template, lt-a1b2c3097ecb147e2, and apply these changes to the servers s-App1, s-App2, and s-App3. We can achieve this by running the following command:

python target-templates-update --target s-App1,s-App2,s-App3  --template-id lt-a1b2c3097ecb147e2 --parameters SubnetId,InstanceType

The other parameters that we can specify in the command above are AssociatePublicIpAddress, DeleteOnTermination, Groups, Tenancy and IamInstanceProfile. These are all passed “comma separated” to the --parameter argument. If the --parameter argument is not specified, all the above parameters are copied.

Specifying --copy-launch-settings would also copy the general launch settings when source server is specified. You could also pass a json file of general launch settings using --launch-settings-file launch_configuration.json where “launch_configuration.json” is the input file.

Option 3 – Import General Settings and EC2 Launch Template parameters from a CSV file

Any migration (medium or large sized) goes through 3 phases of the migration journey – Assess, Mobilize, Migrate & Modernize. To learn more about the phases of a migration journey, please read the prescriptive guidance document here. In the assess phase, you build your business case for migration, and analyze portfolio and server inventory in the scope of migration. Customers often use existing configuration management database (CMDB) or discovery and assessment tools to build an inventory of the IT landscape that is in scope of the migration. These outputs from CMDB or Discovery & Assessment tools are often delivered in a CSV file.

The following solution focuses on using this inventory data from CSV file for rehosting your servers through AWS MGN. You would need to convert the output from the CSV file into a standard flat file (sample_template.csv) which will be then used to automate General launch settings and EC2 Launch Template for each of the source machines. The sample_template.csv file looks as shown below:

Figure 7. An example of a flat CSV file in a specific import format

There are some rules to follow while updating this flat csv file. For example, to add multiple network interfaces on target server and to select appropriate subnets and security groups for each, you need to specify network device index and corresponding subnet or security group in the CSV file as shown below:

In the “Subnet” column: 0:subnet-xxxxxxxx,1:subnet-yyyyyyyy

In the “Security Group” column: 0:sg-xxxxxxxx,1:sg-yyyyyyy

Note: ‘0’ and ‘1’ above indicates the network device index for the network interfaces.

For guidance on how to update the csv file and run the script, please read the “Readme” section of the code here.

Once you have updated the sample_template.csv file, you are ready to run the script, “target-templates-import.py”, which will ingest the CSV file and update the AWS MGN General launch settings and EC2 Launch Template parameters for each of the source machines, as shown in figure below.

Figure 8. An example of “target-templates-import” script updating configuration for 3 target servers

Conclusion

In this blog post we reviewed three options to solve very common challenge that customers are facing during rehost migrations with AWS MGN – managing target instance configurations at scale.

The three solutions we provided enable a scalable approach to:

set default values per migration wave using AWS MGN default launch template feature,
manage and fine-tune future-state server configurations during the test and/or cutover phase,
mass-set all servers’ target configuration withing the same migration wave from a CSV file, based on the assessment tool output.

With the combination of the above approaches customers are able to significantly reduce manual efforts and potential misconfiguration issues during rehost migrations with AWS MGN and, as a result, accelerate rehost migration journeys to AWS.

About the authors:

AWS Cloud Operations Blog