How to patch Windows EC2 instances in private subnets Using AWS Systems Manager

Patching Windows instances in private subnets could be challenging since those Amazon EC2 instances have no internet connectivity. In this blog post we explain how to use AWS Systems Manager and Windows Server Update Services (WSUS) to keep those instances updated. We’ll create a new VPC with the proper endpoints, security groups, and network access control lists (ACLs), so the instances in the private subnet will not have connectivity to the internet, but at the same time they will stay up-to-date with software updates.

Setup

Creating VPC and endpoints

For this exercise we are going to create a new VPC using the Amazon Virtual Private Cloud (Amazon VPC) Launch VPC Wizard. However, before we create the VPC we need to allocate an Elastic IP (EIP) address for the NAT gateway that will be provisioned as part of the VPC creation.

Open the AWS Management Console, and navigate to the Amazon VPC console. On the Create VPC page, choose Elastic IPs on the left navigation pane and allocate a new EIP address.

Next, create the VPC by choosing the VPC with public and private subnets option. Use the default settings to create the VPC, and choose the Elastic IP allocation ID for the EIP that was previously reserved for the NAT gateway.

For more information, see the VPC creation procedure in the documentation. To create the VPC using AWS CloudFormation, see this VPC template.

After the VPC is created we need to add VPC endpoints for System Manager to the private subnet. VPC endpoints enable you to privately connect your VPC to supported AWS services and VPC endpoint services powered by PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Instances in your VPC do not require public IP addresses to communicate with resources in the service. Traffic between your VPC and the other service does not leave the Amazon network. See the documentation for setting up VPC endpoints for Systems Manager. In the next steps we will create the following endpoints:

• ssm – the endpoint for the Systems Manager APIs
• ec2messages – the endpoint for the Run Command messaging service
• ec2 – the endpoint used to enumerate attached Amazon EBS volumes
• ssmmessages – the endpoint for the Session Manager messaging service
• s3 – the end point for Amazon S3, used for logs and documents, and to update the Systems Manager agent

Use this endpoint for the AWS Systems Manager service: com.amazonaws.region.ssm.

Use this endpoint to make calls from SSM Agent to the Systems Manager service: com.amazonaws.region.ec2messages.

If you’re using Systems Manager to create VSS-enabled snapshots, you need to ensure that you have an endpoint to the EC2 service. If you haven’t defined the EC2 endpoint, a call to enumerate attached EBS volumes fails, which causes the Systems Manager command to fail.

This is the endpoint to the EC2 service: com.amazonaws.region.ec2.

The next endpoint is required only if you are connecting to your instances through a secure data channel using Session Manager.

This is the endpoint to the EC2 service: com.amazonaws.region.ssmmessages.

Finally, create the Amazon S3 gateway endpoint on the VPC. Systems Manager uses this endpoint to upload Amazon S3 output logs, and to update SSM Agent.

Security – Creating security groups and network ACLs

Create a security group for the public subnet and another security group for the private subnet. The inbound rules for the public subnet should allow all traffic from the private subnet and TCP port 3389 for RDP, from a known IP address.

The outbound rules for the public subnet should allow traffic to all destinations.

The inbound rules for the private subnet should allow for TCP port 3389 traffic from and to the public subnet.

This allows the use of a “jump box/bastion host” in the public subnet to use remote desktop to connect to the Windows instances in the private subnet. This is only required if you want to use Remote Desktop to connect to your instances in the private subnet. Customers using AWS Session Manager can skip this step. For more information see AWS Session Manager. The Windows instance running WSUS in the public subnet could be used for this purpose. The outbound rules for the private subnet should allow all traffic to the public subnet.

For more information on connecting remotely to a Windows instance, see the documentation for connecting to a Windows instance and for authorizing inbound traffic for your Windows instances.

The SSM Agent running on the private subnet needs to connect to Amazon S3 to install the Windows PowerShell module that allows the Agent to scan and patch the instances. For the SSM Agent to be able to connect to Amazon S3, we need to add the S3 prefixes to the private subnet network ACL. The S3 prefixes are Region-specific. To get the S3 prefixes, for example for the us-west-2 AWS Region, we can run the following command from the AWS CLI:

aws ec2 describe-prefix-lists --region us-west-2

Among other things, the output from the CLI command will return the following information for S3:

           "PrefixListName": "com.amazonaws.us-west-2.s3",
            "Cidrs": [
                "54.231.160.0/19",
                "52.218.128.0/17",
                "52.92.32.0/22"
            ],

For more information on S3 prefixes, see the AWS CLI Reference documentation for describe-prefix-lists and the Amazon VPC documentation for gateway VPC endpoints.

Configure HTTPS (port 443) traffic to each of the S3 prefixes on the outbound rules of the private subnet. The outbound rules for the private subnet should look something like this:

Install the WSUS server in the public subnet

Install a Windows Server instance in the public subnet and assign to it the public subnet security group. Install and configure a WSUS role to periodically download patch data from Windows Update. Make sure you have at least 150 GB of hard drive space for the WSUS updates folder. For more information on installing and configuring WSUS, see the Microsoft documentation.

Install Windows instances in private subnets

Create Windows instances on the private subnets and assign the private subnet security group and the default VPC security group to the instances. It’s very important to assign the default VPC security group to the instances, so that the SSM Agent can communicate with the Systems Manager service. Configure the Windows Update Agent to use the WSUS server on the public subnet. This can be done by editing the registry on the instances or by using a group policy. You can also use State Manager and the new PowerShell DSC to configure the servers. See PowerShell DSC and DSC Registry Resource for more information. Also see the Microsoft documentation on configuring group policy settings for automatic updates and configuring automatic updates using registry settings.

Here is a registry configuration example:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate]

"WUServer"="http://10.0.0.126:8530"

"WUStatusServer"="http://10.0.0.126:8530"

"AcceptTrustedPublishersCerts"=dword:00000001

"TargetGroups"="Servers"

"TargetGroupEnabled"=dword:00000000

"UpdateServiceUrlAlternate"=""


[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU]

"AUOptions"=dword:00000004

"NoAutoUpdate"=dword:00000000

"UseWUServer"=dword:00000001

"ScheduleInstallDay"=dword:00000000

Keep in mind that the latest WSUS uses HTTP port 8530 or HTTPS port 8531 instead of ports 80 and 443. Also, use the WSUS administration console to make sure that WSUS finished its synchronization with Windows Update before scanning or patching your instances.

Scanning and patching Windows instances in a private subnet

Verifying the SSM Agent is communicating with Systems Manager

The Windows instances in the private subnet should now be visible in Systems Manager. To verify this, open the Systems Manager console, and then navigate to the Managed instances page. The Window instances should be listed and their Ping status should be Online.

Run Command

To scan or patch the Windows instances we are going to use the AWS-RunPatchBaseline document under AWS Systems Manager Run Command. The AWS-RunPatchBaseline runs the default patch baseline for Windows. The AWS-DefaultPatchBaseline is the default Windows patch baseline. It includes all critical updates, and security updates that have a critical or important severity. WSUS is configured by default to auto approve the same patch categories and severities. If the default AWS patch baseline is not adequate for your environment, a new patch baseline can be created using AWS Systems Manager Patch Manager. For more information, see Default and Custom Patch Baselines. Make the new patch baseline the default baseline for Windows and configure WSUS auto approval to meet your needs, or use the Patch Group tag to target the new patch baseline to the appropriate instances.

To perform a scan, in the AWS Systems Manager console, on the Run Command page, choose the AWS-RunPatchBaseline document and select the Scan option under Operation. Specify your target, chose if you want to enable writing to S3 or not, and then run the command. This command will scan for the patches included in the AWS-DefaultPatchBaseline and report compliance. After the command successfully runs, check for the results of the scan under Compliance. You should see something similar to this:

To manually patch the Windows instances, run the AWS-RunPatchBaseline document again, but this time choose Install as the operation. This instructs your Windows instances to apply the patches that are missing from the configured patch baseline. The Windows instances go to the WSUS server to download the necessary patches, install them, and then reboot.

State Manager

To schedule patching to occur on a regular basis you can use the AWS Systems Manager Maintenance Windows features, or AWS Systems Manager State Manager. If you want complete control over when your instances are rebooted after being patched we recommend you use Maintenance Windows. If you’re less sensitive to reboots you can use State Manager and drive patching as a form of desired state configuration. In addition, we just launched a new, easy on-boarding experience in the console for patching (look for the Configure patching button in the Patch Manager section of the AWS Systems Manager console). Configure patching makes it easy for customers to set up automated patching.

State Manager can be used for hands-off patching. An association defines the state you want to apply to a set of targets. It includes three components: a document that defines the state, the targets, and a schedule.

An association can be created to, for example, run the AWS-RunPatchBaseline in scan or install mode on a daily basis.

In the AWS Systems Manager console, navigate to the Systems Manager page, and in the Create association section, search for and then select the AWS-RunPatchBaseline document:

When you choose the Scan option, AWS-RunPatchBaseline determines the patch compliance state of the instance and reports this information back to Patch Manager. Scan does not prompt updates to be installed or instances to be rebooted. Instead, the operation identifies where updates are missing that are approved and applicable to the instance.

When you choose the Install option, AWS-RunPatchBaseline attempts to install the approved and applicable updates that are missing from the instance.

Furthermore, you can specify a CRON style schedule to run your association.

Systems Manager is not just for patching

Since the SSM Agent is installed on Windows instances, you aren’t limited to just patching. You can take full advantage of Systems Manager and collect tag-based inventory from instances, use Run Command to execute other documents, and so on. See the following blog posts for more information on the many facets of AWS Systems Manager:

• Using AWS Systems Manager Inventory with tags
• Using S3 sync capability with Systems Manager
• Centralized multi-account and multi-Region patching with AWS Systems Manager Automation

We used a private subnet on AWS for this exercise, but the same concept could be used to patch Windows on your private cloud. Systems Manager can manage servers and virtual machines in your hybrid environment. To learn more about AWS Systems Manager Activations and setting up AWS Systems Manager in a hybrid environment, see the following topics in the documentation:

• AWS Systems Manager Activation
• Setting Up AWS Systems Manager in Hybrid Environments

About the Authors

Carlos Santiago is a Sr. Technical Account Manager with more than 23 years of experience in Windows systems management. He is passionate towards helping customers move from legacy management systems to the cloud.

Imtiaz (Taz) Sayed is a Principal Technical Account Manager and an engineer at heart. He loves working with customers and enabling them with solutions that accomplish more with less.