Implement a centralized patching solution across multiple AWS Regions

August 3, 2021: We’ve updated the link to download the CloudFormation template in “Step 2: Deploy the CloudFormation template.”

In this post, I show you how to implement a centralized patching solution across Amazon Web Services (AWS) Regions by using AWS Systems Manager in your AWS account. This helps you to initiate, track, and manage your patching events across AWS Regions from one centralized place.

Enterprises with large, multi-Region hybrid environments must determine whether they want to centralize patching by using Systems Manager to map all their instances under one Region, or decentralize patching to each Region where instances are deployed. Both approaches have trade-offs in terms of cost and operation overhead. For centralized patching under one Region, you must enable the Systems Manager advanced-instances tier if your running instances count exceeds the registration maximum for on-premises servers or VMs per AWS account per Region. (At the time of this blog post, the maximum count is set to 1,000). This tier is priced at a higher pay-as-you-go rate, but provides additional features on top of the standard-instances tier solution, such as the ability to connect to your hybrid machines by using Systems Manager Session Manager, Microsoft application patching, or other solutions. Using a decentralized patching approach, if you aren’t interested in advanced-tier features and have more instances than the AWS Region registration maximum that is allowed at the standard-tier level, you can distribute your instances across Regions and run it under the standard-tier section which is priced at a lower rate with respect to the advanced tier.

Solution overview

Figure 1 shows the architecture of the centralized patching solution across multiple Regions.

Figure 1: Solution architecture

The automated solution I provide in this post is focused on scheduling and patching managed instances across AWS Regions. Systems Manager Maintenance Windows initiates a series of steps for automated patching for the instances, regardless of which Regions the instances are in.

Here are the key building blocks for this solution:

AWS Systems Manager Maintenance Windows is a feature you can use to define a schedule for when to perform potentially disruptive actions on your instances, such as patching an operating system, updating drivers, or installing software. Maintenance Windows also makes it possible for you to schedule actions on other AWS resource types, such as Amazon Simple Storage Service (Amazon S3) buckets, Amazon Simple Queue Service (Amazon SQS) queues, AWS Key Management Service (AWS KMS) keys, and others that are out of scope for this blog post.

AWS Lambda automatically runs your code without requiring you to provision or manage infrastructure. It can automatically scale your application by running code in response to each event. Also, you only pay for the compute time you consume, so you’re never paying for over-provisioned infrastructure.

AWS Systems Manager Automation simplifies common maintenance and deployment tasks for Amazon Elastic Compute Cloud (Amazon EC2) instances and other AWS resources, without the need for human action.

An AWS Systems Manager document (SSM document) defines the actions that Systems Manager performs on your managed instances.

Solution details

Figure 2 shows the centralized patching solution for a multi-Region hybrid workflow in detail.

Figure 2: Detailed workflow diagram: Centralized patching solution for multi-Region and hybrid instances

You implement the solution as follows:

1. In a central management Region, configure a maintenance window with a custom Lambda function as a target, with a JSON payload input that defines your target Regions, custom SSM document information, and target resource groups.
2. Configure the Lambda function that will first filter out the target Regions where there are no instances mapped to resource groups, and then initiate the Systems Manager Automation API for the remaining Regions that have instances mapped to the resource groups.
3. Configure a Systems Manager Automation API to initiate Run Command in all target Regions according to the custom AWS document.
4. Configure the AWS custom automation document to call the AWS-RunPatchBaseline document against all instances for patching according to the resource group defined in the input payload JSON.

Solution deployment

To deploy the solution, you perform these steps:

1. Verify prerequisites in your AWS account
2. Deploy an AWS CloudFormation template
3. Create a test patching event

Step 1: Verify prerequisites in your AWS account

The sample solution provided by this blog requires that you set up Systems Manager in your account and resource groups in the target Regions. Before you get started, make sure you’ve completed all of the following steps:

• Prepare Systems Manager for your account by following the steps in Setting up AWS Systems Manager.
• Add your hybrid instances to Systems Manager by following the steps in Setting up AWS Systems Manager for hybrid environments.
• Use AWS Resource Groups to create a resource group in each Region that you want to use as a target for patching, as shown in Figure 3. Each resource group must have the same name.
  

  Figure 3: Sample resource group creation

• Create or configure a Systems Manager-compatible instance. You need at least one EC2 instance that is configured for use with Systems Manager in order to complete this walkthrough. This means that SSM Agent is installed on the instance, and an AWS Identity and Access Management (IAM) instance profile for Systems Manager is attached to the instance. You must also add the tag target to the instance with the value DummyTarget, which will be used by Systems Manager Maintenance Windows to run the Lambda function against this target.

Step 2: Deploy the CloudFormation template

In this next step, you deploy a CloudFormation template to implement the centralized patching solution across Regions in your account. Make sure you deploy the template within the AWS account and Region from which you want to centralize patching coordination.

To deploy the CloudFormation stack

1. Choose the following Launch Stack button to launch a CloudFormation stack in your account.

   

Note: The stack will launch in the N. Virginia (us-east-1) Region. It takes approximately 15 minutes for the CloudFormation stack to complete. To deploy this solution into other AWS Regions, download the solution’s CloudFormation template and deploy it to the selected Region.

• In the AWS CloudFormation console, choose the Select Template form, and then choose Next.
• On the Specify Details page, provide the following input parameters. You can modify the default values to customize the solution for your environment.
  

  Input parameter	Description
  Duration	The duration for the maintenance window automation job, in hours. The default is 5.
  OwnerInformation	The owner information for the maintenance window. The default is Patch Management Team.
  Schedule	The schedule for the owner of the maintenance window, in the form of either a cron or rate expression). The default is cron(0 4 ? * SUN *).
  TimeZone	The time zone for the maintenance window automation job. The default is S/EasternU.

  

  Figure 4: An example of the values entered for the template parameters

• After you’ve entered values for all of the input parameters, choose Next.
• On the Options page, keep the defaults, and then choose Next.
• On the Review page, under Capabilities, select the check box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names, and then choose Create stack.
  

  Figure 5: CloudFormation capabilities acknowledgement

After the Status field for the CloudFormation stack changes to CREATE_COMPLETE, as shown in Figure 6, the solution is implemented and is ready for testing.

Figure 6: Completed deployment of the AWS CloudFormation stack

Step 3: Create a test patching event

After the CloudFormation stack has completed deployment, an AWS maintenance window is created. To test the centralized patching solution, you can use this maintenance window to initiate patching across Regions.

(Optional) To create a test patching event, edit the Lambda task as follows. Under the Tasks tab, add the following JSON data as the payload, and update the following parameters with your own data: resource group, AutomationAssumeRole ARN, MaxConcurrency, MaxErrors, Operation (Scan/ Install), and Regions, as needed for the target environment.

{
  "WindowId": "{{WINDOW_ID}}",
  "TaskExecutionId": "{{TASK_EXECUTION_ID}}",
  "Document": {
    "Name": "CustomAutomationDocument",
    "Version": "1",
    "Parameters": {
      "AutomationAssumeRole": [
        "arn:aws:iam::111222333444:role/AWS-SystemsManager-AutomationAdministrationRole"
      ],
      "Operation": [
        "Scan"
      ]
    }
  },
  "TargetParameterName": "InstanceIds",
  "Targets": [
    {
      "Key": "ResourceGroup",
      "Values": [
        "DevGroup"
      ]
    }
  ],
  "MaxConcurrency": "10",
  "MaxErrors": "1",
  "Regions": ["us-east-2","us-east-1"]
}

Wait for the next execution time for the maintenance window. On the History tab, you should see status Success to indicate that patching is complete, as shown in Figure 7.

Figure 7: The History tab for the maintenance window, showing successful patching

To see more details related to the completed automations, look on the Automation Executions tab, shown in Figure 8.

Figure 8: The Executions tab showing details

Congratulations! You’ve successfully deployed and tested a centralized patching solution for an AWS multi-Region hybrid environment. In order to fully implement this solution, you’ll need to add the resource groups in all your target Regions and update the payload JSON in Systems Manager Maintenance Windows.

Summary

You’ve learned how to use Systems Manager to centralize patching across multiple AWS Regions and to include on-premises instances in your patching solution. All of the code for this solution is available as part of an CloudFormation template. Feel free to play around with the code; we hope it helps you learn more about automated security remediation. You can adjust the code to better fit your unique environment, or extend the code with additional steps. For example, you could extend it across accounts and also create a custom Systems Manager document to run across Regions.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about using this solution, contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.