Scheduling centralized multi-account and multi-Region patching with AWS Systems Manager Automation

Update 01/2023: AWS Systems Manager announces Patch Policies, enabling cross account and cross Region patching. Patch Policies provide a user experience in a single console to easily define and enforce patch compliance across accounts and Regions with a few clicks. For more information, see Centrally deploy patching operations across your AWS Organization using Systems Manager Quick Setup.

In an earlier blog post, I showed how you can use AWS Systems Manager Automation to patch managed instances across multiple AWS accounts and Regions. I showed how to perform this operation manually (or through the AWS CLI using start-automation-execution). In this blog post, I show you how to schedule a multi-account and multi-Region patching operation using Amazon EventBridge (formerly Amazon CloudWatch Events), AWS Lambda, and AWS Systems Manager Automation.

I walk you through the steps to set up your management account and target accounts for scheduled multi-account Automation patching using AWS CloudFormation stacks.

Management models

The architecture and tutorial use a centralized management approach to perform patching on target accounts in an organization from a single location.

The benefit of using AWS Systems Manager is that you can be flexible in your management model while maintaining a central source of reporting by using Systems Manager Resource Data Syncs. You can use Resource Data Syncs to send inventory, patching, and the compliance data. This data is collected from your managed instances to a single Amazon Simple Storage Service (Amazon S3) bucket. Resource Data Sync then automatically updates the centralized data when new inventory data is collected. With all inventory data stored in a target S3 bucket, you can use services like Amazon Athena and Amazon QuickSight to query and analyze the aggregated data.

Architecture overview

The tutorial relies on a management or centralized account from which you schedule the multi-account and multi-Region Automation runbook using EventBridge and Lambda.

Figure 1: Architecture for scheduling a multi-account/multi-Region patching workflow

Here is how the process works:

1. In the management account, the EventBridge rule is triggered based on the cron or rate-based expression specified.
2. The EventBridge rule then invokes a Lambda function, which, in turn, initiates a multi-account and multi-Region Automation workflow.
3. The Systems Manager administration role assumes the execution role in each target account and Region.
4. The execution role initiates a Run Command task for AWS-RunPatchBaseline. This command scans for, or install missing updates on target managed instances based on membership in the provided AWS Resource Group.

Prerequisites

When you use tag-based targeting in Automation workflows or Run Command tasks, you can target only a single tag key-value pair. Because AWS Resource Groups support grouping criteria based on multiple tags, it is the target method.

The Automation runbook used in this post targets Systems Manager managed instances by using resource groups. To scan for missing patches or install updates on your managed instances, the resources must be in the targeted resource group.

For more information, check build a query and create a group in the AWS Resource Groups documentation. The resource group must have the exact same name in all target accounts.

Multi-account and multi-Region management

This post assumes that you have set up a management account that monitors and aggregates the results from other accounts in your company. It uses AWS CloudFormation StackSets to propagate and run stacks from a central account to the target accounts. Make sure that your management and target accounts are set up with the permissions required for these stacks to run. For more information, check prerequisites for stack set operations in the AWS CloudFormation documentation.  If you don’t have this setup or want to avoid using StackSets, you can use individual CloudFormation stacks in each target account.

Tutorial

These are the steps you need to perform:

1. Deploy an AWS CloudFormation stack to set up the management account and create the following resources:
   • EventBridge rule
   • AWS Identity & Access Management (IAM) service role for Lambda
   • Lambda function
   • IAM service role for the management account Automation, AWS-SystemsManager-AutomationAdministrationRole
   • Automation document to invoke the Command document AWS-RunPatchBaseline
2. Deploy an AWS CloudFormation stack to set up the target accounts and create the IAM service role for target account Automation, AWS-SystemsManager-AutomationExecutionRole.

Step 1: Deploy resources into the management account

First, choose one account as your management account and one AWS Region as your management Region. From this management location (management account and Region), you can schedule Automation tasks targeting other AWS accounts and Regions. If you are already using Systems Manager Explorer or AWS Organizations, then you can use the AWS Organizations management account or the account specified as the delegated administrator for Systems Manager Explorer. For more information, check Configuring a Delegated Administrator.

Open the following GitHub page and download the Scheduled-Patch-Automation.yaml file.

https://github.com/aws-samples/aws-systems-manager-schedule-central-patch-example/blob/main/Templates/Scheduled-Patch-Automation.yaml

1. In the management account and Region, navigate to the AWS CloudFormation console.
2. From the left navigation pane, choose Stacks, and then choose Create stack.
3. From the dropdown list, choose With new resources (standard).
4. On the Create stack page, select Upload a template file, select Choose file, choose the Scheduled-Patch-Automation.yaml file, and then choose Next.
5. In the Specify stack details page:
   • For EventBridgeRuleSchedule, enter a cron or rate-based expression for the schedule of the EventBridge rule. For example, “cron(15 22 ? * TUE *)” schedules the rule to initiate patching on Tuesdays at 22:15 UTC. For more information, check schedule expressions for rules in the Amazon EventBridge documentation.
   • For ExecutionRoleName, optionally modify the Automation execution role to be assumed in target accounts during multi-account and multi-Region Automation patching operations.
   • For ExistingAutomationAdministrationRole, optionally enter the IAM role ARN of an existing Automation administration role that has permissions to invoke multi-account and multi-Region Automation workflows. If you do not provide an ARN, the AWS CloudFormation template creates the Automation administration role.
   • For MaximumConcurrency, specify the maximum number of targets allowed to run this task in parallel. You can specify a number, such as 10, or a percentage, such as 10%. The default value is 10%.
   • For MaximumErrors, specify the maximum number of errors that are allowed before the system stops running the task on additional targets. You can specify a number, such as 10, or a percentage, such as 10%. The default value is 10%.
   • For ResourceGroupName, specify the name of the resource group that includes the resources you want to target. The resource group name is case-sensitive.
   • For RunPatchBaselineInstallOverrideList, optionally enter an HTTPS URL or an Amazon S3 path-style URL to the list of patches to be installed. This patch installation list overrides the patches specified by the default patch baseline.
   • For RunPatchBaselineOperation, choose Scan to scan for missing updates only. Choose Install to scan and install missing updates based on the rules of the patch baseline.
   • For RunPatchBaselineRebootOption, choose the reboot behavior for the patching operation. Valid options are RebootIfNeeded and NoReboot. The default option, RebootIfNeeded, enforces a reboot if updates are installed. Important: Updates are not installed if the Scan operation is selected. For more information, check AWS-RunPatchBaseline parameters.
   • For TargetAccounts, enter the list of target accounts as a comma-separated list. You can specify AWS account IDs or AWS Organizations OU IDs (for example, 012345678901,987654321098).
   • For TargetLocationMaxConcurrency, optionally modify the maximum number of AWS accounts and AWS Regions allowed to run the Automation concurrently. The default value is 1.
   • For TargetLocationMaxErrors, optionally modify the maximum number of errors allowed before the system stops queuing additional Automation executions at the account-Region pair level. The default value is 1.
   • For TargetRegionIds, enter the list of target Regions as a comma-separated list (for example, us-east-1,us-east-2).

Figure 2: Specify stack details page of the AWS CloudFormation console

6. On the Configure stack options page, add any required tags, and then choose Next.
7. Review all the information, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, and then choose Create Stack to submit your stack configuration.

After the page is refreshed, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section.

Step 2: Deploy resources into the target account

If you have already created the Automation execution role, AWS-SystemsManager-AutomationExecutionRole, in your target accounts, you can skip this step. However, if you have not created this role and established a trust relationship with the management account, use the following steps to configure the target account.

Open the following GitHub page and download the AWS-SystemsManager-AutomationExecutionRole.yaml file.

https://github.com/aws-samples/aws-systems-manager-schedule-central-patch-example/blob/main/Templates/AWS-SystemsManager-AutomationExecutionRole.yaml

This template creates the AWS-SystemsManager-AutomationExecutionRole in your target accounts.

1. In the management account, go to the AWS CloudFormation console and from the left navigation pane, choose StackSets, and then choose Create StackSet.
2. On the Choose a template page, select Upload a template file, select Choose file, choose the AWS-SystemsManager-AutomationExecutionRole.yaml file, and then choose Next.
3. In the Specify StackSet details page, for ManagementAccountNumber, enter the account number for the management account, and then choose Next.

Figure 3: Specify StackSet details page of the AWS CloudFormation console

4. On the Configure StackSet options page, add any required tags. The content of the Permissions section varies, depending on whether you have AWS Organizations enabled:
   • If you do not have AWS Organizations enabled, choose Self service permissions, and then choose Next.
   • If you have AWS Organizations enabled, you can choose Service managed permissions to apply the StackSets to an organization or OU. For more manual control, you can also choose Self service permissions. In this case, choose Self service permissions to add a list of accounts as targets. Choose Next.
5. In the Set deployment options page, enter as a comma-separated list the target account numbers and Regions where you want to run this automated process. Or you can also upload a .csv file with the numbers. Choose Next.

Figure 4: Set deployment options page

6. On the next page, review the options, select I acknowledge that AWS CloudFormation might create IAM resources with custom names. Then choose Submit to submit your StackSet configuration.

After the page is refreshed, you will be able to see your StackSet. The status will change to SUCCEEDED after it’s been created.

Checking Automation executions

After your automations have had a chance to run, you should start seeing the results of the executions in Systems Manager.

Navigate to Systems Manager console and choose Automation. On this page, you find a list of the Automation documents that ran.

You can see your executions in the results using Document name and Executed by. Lambda function executions populate the Executed by column with the ARN of the Lambda execution role.

Figure 5: Automation executions in the AWS Systems Manager console

Cleanup

To delete the resources created in step 1 of this tutorial, go to the AWS CloudFormation console in the management account. Choose the stack you created, and then choose Delete.

To delete the resources created in step 2 of this tutorial, go to the AWS CloudFormation console in the management account. From the left navigation pane, choose StackSets, and then choose the StackSet you created. From Actions, choose Delete StackSet.

Conclusion

In this blog post, I showed you how to schedule patching of your Systems Manager managed instances across multiple AWS accounts and Regions by using EventBridge, Lambda, and Systems Manager Automation.

I started by showing you how to create the required resources into your management account using AWS CloudFormation. After you configured the management account, I showed you how to use AWS CloudFormation StackSets to deploy the Automation execution role into target accounts.

Following the first invocation of the multi-account and multi-Region patching operation, you can view the results and determine which managed instances are missing required updates. To view patching compliance across your environment, you can use AWS Systems Manager Explorer. It provides a customizable operations dashboard that includes the patching compliance state of your managed instances. For more information, check AWS Systems Manager Explorer and setting up Systems Manager Explorer to display data from multiple accounts and Regions in the AWS Systems Manager documentation.

You can aggregate patching, compliance, and inventory data into a single location by creating a Resource Data Sync, which syncs this data with a centralized S3 bucket of your choosing. For more information, check configuring Resource Data Sync for Inventory.

After creating the Resource Data Sync, you can then configure Amazon Athena and Amazon QuickSight to start visualizing patching and inventory-related data. For more information, check querying inventory data from multiple Regions and accounts.

For CloudFormation sample templates to create the resources referenced in this blog and the required resources for visualization, see the GitHub sample Operational Management: Inventory, Patching, and Compliance.

About the author