AWS Cloud Operations Blog

Centrally deploy patching operations across your AWS Organization using Systems Manager Quick Setup

Organizations managing cloud infrastructure in Amazon Web Services need effective mechanisms to manage compliance and security for their resources and applications. Previously, customers were able to scan instances daily for missing patches across all instances in their organization through the Host Management Quick Setup Configuration. Additionally, customers could implement patching using default patch baselines in patch groups.

Today, we are excited to announce the release of Quick Setup Patch Policies, powered by Patch Manager, which enables you to easily set up patch management across an AWS Organization. Patch policies enable customers to scan and schedule patch installation for multiple patch baselines across AWS accounts and across AWS Regions.

For the patch baselines, you can apply AWS default or your own custom patch baselines to multiple operating systems. You can also target Amazon Elastic Compute Cloud (EC2) instances and hybrid managed nodes across the entire AWS Organization or to specific Organizational Units (OUs) and Regions, as well as select all managed nodes or filter based on specific resource tags. You can create and manage multiple patch policies at once, enabling you to control patching operations for different sets of instances.

With patch policies and Quick Setup, you can now scan and apply patches to managed nodes across your environment with more control. Prior to this release, customers may have needed to log into multiple accounts to view patch compliance and apply patches. Now, customers can apply a patch policy across an entire organization for multiple operating systems, across multiple accounts and Regions, and review resource compliance for the target managed nodes.

In this post, we show you how to create a patch policy using the Quick Setup Patch Manager configuration type, and show you how you can view the compliance of your managed nodes against these patch policies.

Quick Setup overview

Use Quick Setup, a capability of Systems Manager, to quickly configure frequently used AWS services and features with recommended best practices. Quick Setup simplifies setting up services by automating common or recommended tasks. You can use Quick Setup in an individual AWS account or across multiple AWS accounts and AWS Regions by integrating with AWS Organizations.

Using Quick Setup across multiple accounts helps to ensure that your organization maintains consistent configurations. Additionally, Quick Setup periodically checks for configuration drift and attempts to remediate it. Configuration drift occurs whenever a user makes any change to a service or feature that conflicts with the selections made through Quick Setup.

To create a consistent configuration, Quick Setup uses AWS CloudFormation StackSets to deploy Quick Setup configurations across your organization.

Here is how the process works for the Patch Manager configuration:

  1. You create the patch policy using Quick Setup and the parameters selected are sent to CloudFormation.
  2. CloudFormation creates a stack set with the defined parameters and defined target accounts and Regions.
  3. CloudFormation creates stack instances in each target account and Region.
  4. The stack instances create a Systems Manager State Manager association for the defined patch scan and an association for patch installation, if selected. These associations are applied using the schedules provided when you create the patch policy.
Figure 1: Architecture for creating patch policies using the Patch Manager configuration within Systems Manager Quick Setup.

Figure 1: Architecture for creating patch policies using the Patch Manager configuration within Systems Manager Quick Setup.

In addition to the resources referenced above, there are other resources created by Quick Setup. Within the Organization management account, the following resources are created:

  • Amazon Simple Storage Service (S3) bucket to store the patch baselines specified as a JSON file.
  • AWS Lambda function to evaluate custom patch baselines specified within Quick Setup for changes. If changes are made to the custom patch baselines, Quick Setup propagates those changes across the target accounts and Regions.
  • Systems Manager Automation runbook to invoke the Lambda function.
  • Systems Manager State Manager association to initiate the Automation runbook every hour.
  • AWS Identity and Access Management (IAM) roles for Lambda and Automation.

In the target accounts and Regions, the following resources are created:

  • Automation runbook and State Manager association to create and attach the Quick Setup IAM role to EC2 and hybrid managed nodes
  • State Manager association to enable Systems Manager Explorer
  • State Manager association to remediate Quick Setup related tags on managed nodes

Prerequisites

Amazon Elastic Compute Cloud (EC2) instances, AWS Internet of Things (IoT) Greengrass core devices, on-premises servers, edge devices, and VMs must be Systems Manager managed nodes to be patched. This means your nodes must meet certain prerequisites and be configured with the AWS Systems Manager Agent (SSM Agent). For more information, see Setting up AWS Systems Manager.

To use custom patch baselines within a patch policy, the custom patch baseline must exist in the same account and Region prior to using Quick Setup. For more information, see Working with custom patch baselines (console).

Walkthrough

In this walkthrough, we take you through creating a patch policy using Systems Manager Quick Setup and explore the various configuration options for scanning, patching, and targeting managed instances.

Create a Quick Setup Patch Manager Configuration

  1. Open the AWS Systems Manager console.
  2. In the navigation pane, choose Quick Setup.
  3. In the Library tab, choose Create for Patch Manager.
  4. For Configuration name, enter a descriptive name, such as patch-policy-blog.
  5. For Scanning and installation, perform the following steps
    1. For Patch operation, choose Scan and install.
    2. For Scanning schedule, choose Use recommended defaults to scan managed nodes daily at 01:00 AM UTC.
    3. For Installation schedule, choose Use recommended defaults to install patches once a week at 02:00 AM UTC on Sunday. Optionally, choose Custom install schedule to provide a custom CRON expression, such as, cron(30 23 ? * TUE#3 *). For more information, see Reference: Cron and rate expressions for Systems Manager.
    4. For Reboot if needed, optionally enable this option to reboot the nodes after patch installation. Rebooting after installation is recommended but can cause availability issues. Leave the option disabled to defer reboots to a later point in time.
Figure 2: Under the Scanning and installation settings, you can choose to scan only or scan and install missing updates based on recommended scheduling or custom schedules based on CRON expressions.

Figure 2: Under the Scanning and installation settings, you can choose to scan only or scan and install missing updates based on recommended scheduling or custom schedules based on CRON expressions.

  1.  For Patch baseline, choose the default value Use recommended defaults or you can choose Custom patch baseline to select custom patch baselines that you have previously created in the same account and Region from where you are deploying Quick Setup. The baselines selected will be used for patch operations initiated using patch policies in the target accounts and Regions. For demonstration purposes, we have created custom patch baselines for Amazon Linux 2, Ubuntu server, and Windows server.
Figure 3: Under the Patch baseline settings, you can choose custom patch baselines created in the same account and Region. These baselines are used during patching operations initiated by the patch policy in the target accounts and Regions.

Figure 3: Under the Patch baseline settings, you can choose custom patch baselines created in the same account and Region. These baselines are used during patching operations initiated by the patch policy in the target accounts and Regions.

  1. For Patching log storage, optionally select whether you would like to store patch operation logs in an Amazon Simple Storage Service (S3) bucket.
  2. For Targets, select whether you want to target the Entire organization, a Custom selection based on organizational units (OUs) and Regions, or nodes under the Current Account.
    1. (Optional) If selecting Custom, use the Target OUs list and checkbox to select OUs of choice. Additionally, select which Target Regions to target by using the check boxes.
    2. (Optional) If selecting Current account, select whether you want to target the Current region or Choose Regions using the picker.
    3. (Optional) If selecting Custom or Current account, select whether you want to target All managed nodes or Specify node tag and target nodes using tags. For Current account, you can select Manual to manually select instances from the instance picker.
Figure 4: Under the Targets setting, you can choose to target your entire AWS organization, custom OUs and Regions, or the current account and Regions. For Custom and Current account, you can choose to target all managed nodes or specify a node tag.

Figure 4: Under the Targets setting, you can choose to target your entire AWS organization, custom OUs and Regions, or the current account and Regions. For Custom and Current account, you can choose to target all managed nodes or specify a node tag.

  1. For Rate control, perform the following steps:
    1. For Concurrency, enter a number or percentage of nodes to run the patch policy on at the same time.
    2. For Error threshold, enter the number or percentage of nodes that can experience an error before the patch policy fails.
  1. For Instance profile options, optionally select to have Quick Setup add the required AWS Identity and Access Management (IAM) policies to existing instance profiles attached to your instances.

Note: By default, Quick Setup creates IAM policies and instance profiles with the permissions needed for the configuration you choose. The instance profiles created by Quick Setup are then attached only to instances that do not have an instance profile attached. If you enable this option, Quick Setup will also add the AmazonSSMManagedInstanceCore policy and a custom in-line IAM policy, granting s3:GetObject to the S3 bucket created by Quick Setup, to instances with instance profiles already attached. This operation is performed once every 30 days.

Figure 5: Under the Instance profile options, you can choose to optionally enable adding required IAM policies to existing instance profiles attached to your instances.

Figure 5: Under the Instance profile options, you can choose to optionally enable adding required IAM policies to existing instance profiles attached to your instances.

  1. Choose Create.

On the subsequent page, you can see that Quick Setup initiates the deployment of the patch policy across the target accounts and Regions. Here you can monitor the deployment status, association status, and resource compliance.

Following deployment, the patch policy initiates a patch scan or scan and install during the specified scheduled periods. You can see the overall patch compliance state of your managed nodes in the Resource compliance widget.

Figure 6: Within the configuration details of the patch policy, you can monitor the deployment status, association status, and resource compliance.

Figure 6: Within the configuration details of the patch policy, you can monitor the deployment status, association status, and resource compliance.

Within a target account, you can check the compliance of your managed nodes by navigating to the Patch Manager console. In the Compliance reporting tab, you can filter and search for nodes based on their compliance status, count of noncompliant updates, tag key-values as well as other node details.

Figure 7: In the Patch Manager console, you can use the Compliance reporting tab to see details for compliance status for your managed nodes.

Figure 7: In the Patch Manager console, you can use the Compliance reporting tab to see details for compliance status for your managed nodes.

Note: The Patch configuration type value in the Node patching details section of the Compliance reporting tab will be Patch policy for patch policies created in the Quick Setup Patch Manager Configuration Type and Patch group for previous Patch Manager operations.

You can also check the compliance of a managed node by navigating to the Fleet Manager console, selecting a managed node, and choosing View details. On the managed node page, choose the Patch tab to see the patch summary details for the selected managed node.

Figure 8: The Patch Summary details for a selected managed node in the Fleet Manager console.

Figure 8: The Patch Summary details for a selected managed node in the Fleet Manager console.

Cleanup

To delete the patch policy created in this post, navigate to the Quick Setup console, select the Patch Manager configuration type created, choose Actions, choose Delete configuration, choose Remove all OUs and Regions. After all OUs and Regions have been removed, choose Delete.

Conclusion

In this post, we showed you how you can quickly set up patch scans or scans and installs across an AWS Organization using patch policies in Quick Setup. By using patch policies, you can centrally define patch scanning and installation schedules and centrally define the patch baseline criteria for the types of updates to install. Additionally, you can create multiple patch policies to ensure the appropriate resources are being patched during well-defined windows. We also showed you how to retrieve a high-level overview of patch compliance across your environment.

You can aggregate detailed patching, compliance, and inventory data into a single location by creating a Resource Data Sync. This syncs this data with a centralized S3 bucket of your choosing. For more information, check configuring Resource Data Sync for Inventory.

After creating the Resource Data Sync, you can configure Amazon Athena and Amazon QuickSight to start visualizing patching and inventory-related data. For more information, check querying inventory data from multiple Regions and accounts.

About the authors:

Anthony Verleysen

Anthony Verleysen is a Senior Product Manager – Technical within the AWS Systems Manager team. He is the the product manager for Patch Manager and Distributor. Outside of work, Anthony is an avid soccer and tennis player.

Erik Weber

Erik Weber is a World-wide Specialist Solutions Architect for AWS Cloud Operations services. He specializes in AWS Systems Manager, AWS Config, AWS CloudTrail, and AWS Audit Manager. Outside of work, Erik has a passion for hiking, cooking, and biking.