Avoid zero-day vulnerabilities with same-day security patching using AWS Systems Manager

This post was co-authored by Jordan Koch at Veradigm.

Applying operating systems patches is one of the easiest ways to secure a system from ever-changing cybersecurity threats. However, for many organizations it is one of the most difficult and time-consuming tasks. Many organizations deploy operating system patches through their various environments, first applying to Development, then to other non-production like Test/QA environments, and finally to their Production environment. That process causes a situation where a known security patch is available but not applied to their most critical environment for weeks or possibly months after the patch is released. This article will demonstrate how Veradigm has shifted their processes to apply operating systems security patches on the same day they are released.

Veradigm is an integrated data systems and services company that combines data-driven clinical insights with actionable tools to help healthcare stakeholders improve the quality, efficiency, and value of healthcare delivery. The Veradigm Network is dedicated to simplifying the complicated healthcare ecosystem with next-generation healthcare solutions for biopharma, health plans, healthcare providers, health technology partners, and most importantly, the patients they serve.

Same-Day Patching Process

A traditional practice is to install operating systems patches, including critical security patches, through Development and Test environments before installing into Production environment to identify any potential issues. If this practice takes a few days to weeks, it imposes potential risk from zero-day vulnerability attacks. The rationale for adopting the same-day patching is to avoid such zero-day vulnerabilities in the customer facing environments. The same-day patching practice allows customers to adopt a regular patching schedule for critical security patches, and helps avoid unknown consequences of potential compromise.

The challenge customers face to adopt the same-day patching process is to shift their operations practices with Operations as Code, with event-driven alerts and pipelines to notify and take decision to deploy into customer facing environment. Many of our customers adopt the Kernal Live patching feature of Amazon Linux 2, which allows you to apply security vulnerability and critical patches to a running Linux kernel without reboots or disruptions to running applications. The same-day patching process follows the principle of making frequent, small, reversible changes, to apply small sets of patches as they are released, compare to waiting for days or weeks and apply the patches in a larger batch. For critical business applications, AWS recommends a rolling deployment model to avoid customer disruption.

Solution

Shifting from the traditional thinking on how to mitigate security vulnerabilities, the same-day patching process is to apply patches through all the different environments, including Production, on the same-day to avoid security vulnerabilities, rather test your patches in Development and Test environments while waiting for days or weeks to apply into Production.

The Veradigm SysOps team worked with their development teams to review the trade-off with adopting same-day patching process. Veradigm found that the chances of the instances being rebooted daily is rather low compare to the potential of security compromise in the production environment that would cause much larger impact. Veradigm implemented automated monitoring of key business processes running on the instances, to automate alerting, and pipeline to deploy patches into different stages of their environments. Using AWS Systems Manager, and AWS Event Bridge Veradigm implemented the same-day patching process for all critical security patches released, and could avoid many potential zero-day vulnerabilities over the last few years.

Technical Steps

Step 1: Create Patch Baseline

Veradigm manages their patching process for every one of their EC2 instances using AWS Systems Manager Patch Manager. To start, create a same-day Patch Baselines for each of the operating systems types in the environment. The default AWS Patch Baselines are configured to only apply security patches with a default approval window of “Wait 7 days before approving”. To achieve same-day patching, you configure a custom patch baseline for Critical and Important Security patches to be approved after 0 days.

Step 2: Configure Maintenance Windows

Configure Maintenance Windows, Maintenance Window Targets, and Maintenance Window Tasks.

The Maintenance Window is created in order to execute at a specific time, that will be least disruptive to the business in case the patching operation needs a reboot.

EC2 instances for each application is separated into groups for testing and rolling deployments, and multiple Maintenance Windows are created in a staggered schedule for the respective Maintenance Windows.

Based on Amazon EC2 instance tagging, set the maintenance window targets to a specific tag or set of tags. When multiple environments exist in the same account, the Maintenance Window target is set to the key/value pair, for example (“Environment: Dev” OR “Environment: Prod”) in order to have the patching Maintenance Window target only specific environments in a given timeframe.

The maintenance window task is created to use the AWS default Command document “AWS-RunPatchBaseline” with the document parameters set (as follows) to “Operation: Install” and “Reboot Option: RebootIfNeeded”.

Step 3: Monitor patch compliance with AWS Systems Manager Explorer.

Utilizing the AWS Systems Manager Explorer can generate an overview of which systems are not receiving regular patches and are not compliant. Moreover, this view can highlight which systems may not be tagged properly, or were unintentionally excluded from the patching operation.

Summary

After adopting the same-day patching process, Veradigm has successfully applied thousands of security patches over the last few years, with minimal instances of impact. Veradigm believes that this process, and shifting from a “test then apply” to “apply then test” for security patches, has greatly increased their security posture, and helped avoid many potential zero-day attacks while simplifying their operations procedure.

About the authors