Create patched clones of your workloads in a multi-account set up using AWS Systems Manager Automation

Zainab Allawi, Associate Solutions Architect, WWPS Solutions Architecture

For businesses of different sizes, patch testing has mostly been manual, and patching process errors or incompatibilities could risk disrupting production workloads.

AWS Systems Manager Automation simplifies the common maintenance and deployment tasks of Amazon Elastic Compute Cloud (Amazon EC2) instances and other AWS resources. It also lets you centrally manage your AWS resources through multi-account actions.

This post shows how you can utilize AWS Systems Manager Automation documents to centrally create a clone environment of your running workload, install patches, and test for any disruption that patch installation may cause.

Overview of solution

In this solution, an automation document resides in the parent account, and lets you run actions against AWS Resource Groups in the child account. Resource groups make it easy to run operations on a large number of AWS resources simultaneously.

In this post, the EC2 instances have a tag called “Workload” that identifies the workload to which the instance belongs. This key is used to create a resource group to group the managed instances logically in the child account. Customize the tag name to match existing tags in your EC2 instances, or attach a new tag if they don’t have any tags applied to them.

The automation document will create an Amazon Machine Image (AMI) of each EC2 instance in the resource group, check the EC2 instances type and tags, launch identical EC2 instances according to the original EC2 instances type and tags, and patch the clone EC2 instances based on the default patch baseline for the operating system. Alternatively, you can create a custom patch baseline and associate it to your EC2 instances using a patch group.

Two AWS CloudFormation templates automatically create all of the resources necessary to run this solution. You must use the CloudFormation template parent-account.yaml in the parent account, and the CloudFormation template child-account.yaml in the child account. Later in this post, I will show you how to deploy the templates and run the automation document.

The following figure shows a high level view of the solution architecture.

Walkthrough

In this post, I show you how to use the CloudFormation templates provided to replicate the solution in your own accounts, as well as run the automation in the parent account to create a test environment in the child account.

I will walk you through the following steps:

1. Create a stack in the parent account using the parent account CloudFormation template.
2. Create a stack in the child account using the child account CloudFormation template.
3. Run automation to create a clone environment in child account.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• An AWS account for management (parent account) and a target AWS account (child account). If you are already using Systems Manager Explorer or AWS Organizations, then you can use the Delegated Administrator account or the AWS Organizations management account as a parent account. For more information, check Configuring a Delegated Administrator.
• AWS Systems Manager agent (SSM Agent) installed on your EC2 instances. By default, SSM Agent is preinstalled on instances created from the Amazon Machine Images (AMI) listed here. If you do not have SSM Agent configured on your instances, check Working with SSM Agent.
• EC2 instances must have an IAM instance profile for Systems Manager attached. This instance profile has the permission policy AmazonSSMManagedInstanceCore attached to it. Note the name of instance profile, as this will be important in this walkthrough. As an example, I will be using the name “SSMInstanceProfile”.
• Permissions in your AWS accounts to create the resources.

Step-by-step deployment

Choose an account to be a parent account, and a management region. From that account and region, you can run automation tasks on other AWS accounts (child accounts) in the same region.

Note that the role names in the provided CloudFormation templates are hardcoded due to the cross-account nature of this solution. Consequently, stack updates that require replacement of these resources will be disallowed by CloudFormation.

Complete the following steps:

Step 1: Create a stack in the parent account using the parent account CloudFormation template

In this step, you will deploy a stack in the parent account from a CloudFormation template. The stack will create the following resources:

• • AWS Identity & Access Management (IAM) service role for AWS Lambda.
  • Lambda function.
  • IAM service role for the parent account Automation, AWS-SystemsManager-AutomationAdministrationRole-Parent.
  • Automation document to create and patch the clone EC2 instances.

In the parent account:

1. On the AWS CloudFormation console, choose Create stack.
2. From the dropdown list, choose With new resources (standard).
3. On the Create stack page, select Upload a template file, select Choose file, choose the saved template file titled “parent-account.yaml”, and then choose Next.
4. In the Specify stack details page:

• • For Stack name, enter a name for the stack.
  • For ChildAccountId, enter the AWS Account ID of the child account.
  • For KeyPairName, enter the name of your existing EC2 key pair.
  • For KmsKeyArn, enter the Amazon Resource Name (ARN) of the CMK (Customer Master Key) that you wish to use to encrypt the log group data for the lambda function. This parameter is optional, and you can leave the field blank if you don’t need to encrypt the log group data.
  •  For SSMInstanceProfile, enter the name of the Systems Manager instance profile attached to your EC2 instances in the child account.

5. Choose Next.
6. Step through the remaining pages.
7. On the final page, select the acknowledgement that IAM resources can be created.

8. Choose Create Stack.

The following figure shows the resources created in the parent account and how they interact with the resources in the child account.

Step 2: Create a stack in the child account using the child account CloudFormation template

In this step, you will deploy a stack in the child account from a CloudFormation template. The stack will create the following resources:

• • IAM service role for child account automation, AWS-SystemsManager-AutomationExecutionRole-Child.
  • IAM role called lambda-cross-account-access, which allows a Lambda function in the parent account to get information of the EC2 instances in the child account.
  • Resource group to logically group your managed EC2 instances.

In the child account:

1. On the AWS CloudFormation console, choose Create stack.
2. From the dropdown list, choose With new resources (standard).
3. On the Create stack page, select Upload a template file, select Choose file, choose the saved template file titled “child-account.yaml”, and then choose Next.
4. In the Specify stack details page:

• • For Stack name, enter a name for the stack.
  • For ParentAccountId, enter the AWS Account ID of the parent account.
  • For ResourceGroupTagKey, enter the key of the tag that you have attached to your instances. Assume your instances have the tag Workload: Sharepoint. In this field, enter “Workload”.
  • For ResourceGroupTagValue, enter the value of the tag that you have attached to your instances. For example, “Sharepoint”.
  • For SSMInstanceProfile, enter the name of the Systems Manager instance profile attached to your EC2 instances

5. Choose Next.
6. Step through the remaining pages.
7. On the final page, select the acknowledgement that IAM resources can be created as shown in the previous figure.
8. Choose Create Stack.
9. After the stack reaches the state CREATE_COMPLETE, select the Outputs tab and copy the value of LambdaRoleArn. We will use this ARN later on for executing the automation.

Note that if you have set up the required permissions for AWS CloudFormation StackSets, then you can use the CloudFormation stack to create a StackSet in the parent account. This will automatically create the stack in several child accounts, or in an Organization Unit (OU). For more information, check prerequisites for stack set operations in the AWS CloudFormation documentation.

Step 3: Run automation to create clone environment in child account

In the parent account:

1. On the AWS Systems Manager console, select Automation from the left-side menu.
2. Select Execute automation.
3. In the Choose document page, choose the tab Owned by me and choose the document that was created by the CloudFormation stack, named Automation-CreateTestEnvironment.
4. Select Next.
5. Select Multi-account and Region.
6. In the Target accounts and Regions section:

• • For Accounts and organizational units (OUs), enter the child account ID.
  • For AWS Regions, Choose the AWS Region in which your workload resides.
  • For Automation Execution Role Name, enter AWS-SystemsManager-AutomationExecutionRole-Child.

You can optionally control location (account-Region pair) concurrency and error threshold under Multi-Region and account rate control.

7. In the Targets section:

• • For Parameter, select InstanceId from the drop down menu.
  • For Targets, select Resource Group from the drop down menu.
  • For Resource group, enter MyTagBasedResourceGroup. This resource group is created by the CloudFormation template in the child account.

1.   In the Input parameters section:
   1. 1. For AutomationAssumeRole, select AWS-SystemsManager-AutomationAdministrationRole-Parent.
      2. For Workload, enter the name of your workload. This will be used for naming the AMI and EC2 instances.
      3. For RoleARN, paste the value of the LambdaRoleArn that you copied earlier from the outputs of the child account CloudFormation stack.

9. Select Execute.

You should now be able to see an automation in the parent account that targets the child account, as shown in the following image.

After the automation is complete, navigate to the EC2 console in the child account and choose Instances in the left navigation pane. As shown in the following image, you can see the patched clones of the EC2 instances, named TEST_[Your workload name]_[original instance id].

You can now begin testing your workload for any disruptions.

Clean up

To avoid incurring costs, you may want to either stop or terminate the resources created by the automation document after you conclude your tests.

In your child account:

1. Navigate to the Amazon EC2 console.
2. In the left navigation pane, find AMIs under Images:

• • Select the AMIs generated by the automation.
  • Select Actions, Deregister.

3. In the left navigation pane, find Snapshots under Elastic Block Store:

• • Select the snapshots generated by the AMIs.
  • Select Actions, Delete.

4. In left navigation pane, find Instances under Instances:

• • Select the EC2 instances generated by the automation.
  • Select Actions, Instance State, Terminate.

To clean up all of the resources in the solution, navigate to the CloudFormation console in both the parent and child accounts and delete the stacks.

Conclusion

In this post, I have provided CloudFormation templates and demonstrated how you can leverage AWS Systems Manager automation to centrally create a patched replica of your existing workloads for your testing purposes.

You can customize these automation documents to match your business needs. For example you can edit them with additional steps to enable the clone environment to sleep for a few days while you are testing and then terminate the clone environment after you are done with the testing (delete the image and change the state of EC2 instance).

Author: