AWS Cloud Operations Blog

Automate vulnerability management and remediation in AWS using Amazon Inspector and AWS Systems Manager – Part 2

Update 11/2022 – In September 2022, Amazon Inspector added support for Windows operating systems for continual software vulnerability scanning. This post supports remediating vulnerabilities only on Linux operating systems supported by the Systems Manager agent.

This post is the second part of the Automate vulnerability management and remediation series using Amazon Inspector and AWS Systems Manager.

This series provides methods for remediating Amazon Inspector findings on-demand using AWS Systems Manager Automation runbooks. In Part 1 of this series, you learned how to remediate Inspector findings for a specific vulnerability affecting multiple EC2 instances using AWS Security Hub custom actions. You will have the flexibility of selecting Inspector findings you want to remediate in Security Hub using the custom action. For example, you should use the Security Hub custom action method to remediate Inspector findings for multiple EC2 instances in your AWS organization affected by ‘CVE-2017-8816’.

This second part will demonstrate how to remediate Amazon Inspector findings for Amazon EC2 instances at scale within an AWS Organization. You can patch all vulnerabilities identified by Inspector for your EC2 instance fleet matching a specified tag. You can also filter Inspector findings based on a severity filter if you do not want to remediate vulnerabilities of all severities. For example, you should use this method to remediate all ‘Critical’ vulnerabilities for instances marked with tag key ‘Environment’ and tag value ‘Prod.’

Solution overview

Remediate using AWS Systems Manager Automation console

You can use automationRunPatchBaselineRunbook, an AWS Systems Manager custom Automation runbook provided with this post. You initiate the Automation runbook directly from the Systems Manager Automation console.

The runbook requires parameters for the Organizational Unit (OU) ID or the account ID. Next, you’ll specify resource tags for targeting and an optional severity filter. Choose to reboot the Amazon EC2 instance after installing updates so that the processes can benefit from the updates immediately. You can defer the reboot by selecting the no reboot option to avoid disrupting the workload outside a maintenance window.

The Automation runbook performs a query to identify all Amazon EC2 instances in the OU or the account with the mentioned tag. Then, it conducts another query to Amazon Inspector to identify the findings for the Amazon EC2 instances with the specified tag and severity (if provided). Next, the runbook performs an install patch operation for the affected packages listed in these findings if an update is available. The information on the affected packages is part of the Amazon Inspector finding, as shown in the following figure. Finally, the automation will reboot or defer reboots on the Amazon EC2 instance based on the selected reboot option.

Figure 1: Finding details in the Amazon Inspector Console

Figure 1: Finding details in the Amazon Inspector Console.

This Automation runbook lets you remediate Amazon Inspector findings with the chosen severity for all Amazon EC2 instances with the specified tag. You can trigger this Automation runbook from a Systems Manager delegated administrator account to be executed on Amazon EC2 instances in member accounts.

Architecture overview

The automationRunPatchBaselineRunbook automation is initiated by directly invoking it from the Systems Manager Automation console as described in the solution overview section. Let’s review the process as illustrated in the following figure:

Figure 2: Automation process in multiple accounts

Figure 2: Automation process in multiple accounts.

  1. When initiating the automationRunPatchBaselineRunbook directly from the Systems Manager Automation console, the OU ID or account ID, the severity, and the tag key/value pair are provided as user inputs.
  2. The automation creates a multi-account and multi-Region StartAutomationExecution API call to initiate the patch workflow in the target accounts and regions specified in Step 1. An individual workflow is started for each Amazon EC2 instance with the specified tag key-value pair in each target account and Region.
  3. The automation runbook then queries Amazon Inspector for findings associated with the target Amazon EC2 instances. Then, it creates a list of all of the affected packages per target Amazon EC2 instance. Next, the automation creates a patch install override list from the affected packages gathered. The delegated administrator account stores this list in an Amazon Simple Storage Service (Amazon S3) bucket. The AWS CloudFormation template creates the S3 bucket during the walkthrough section in part-1 of this series.
  4. Next, the automaton runs the AWS-RunPatchBaseline document on the Amazon EC2 instances using the install override list created in Step 3. The reboot option is passed based on the provided user input. If an existing patching operation is running on the target Amazon EC2 instance, then the Automation runbook does not proceed. If an update is available, the AWS-RunPatchBaseline install operation updates the affected packages identified in the patch install override list to the latest version available.
  5.  Then, the Automation runbook queries AWS Systems Manager for the State Manager association applied to the Amazon EC2 instance to gather software inventory using the AWS-GatherSoftwareInventory document.
  6. The association for AWS-GatherSoftwareInventory initiates to apply on the Amazon EC2 instance to refresh inventory immediately.
  7. The Amazon EC2 instance sends refreshed inventory information to AWS Systems Manager.

Amazon Inspector sets the finding status of the remediated findings to CLOSED after installing the patches successfully.

Prerequisites

  1. AWS Systems Manager must manage the Amazon EC2 instances.
  2. You must follow steps 1, 2, and 3 as described in the walkthrough section in Part 1 of this series.

You must deploy the automation runbook in the Systems Manager delegated administrator account. Part 1 of this series walked through the runbook deployment in Security Hub delegated administrator account. Suppose your Systems Manager delegated administrator is the same as Security Hub delegated administrator. In that case, you can move on to the next section.

­Suppose your Systems Manager delegated administrator is NOT the same as the Security Hub delegated administrator. In that case, you must follow Step 1 of Part 1  to deploy in the Systems Manager delegated administrator account. Furthermore, you must follow Step 3 of Part 1 again to add the permissions needed for the new S3 bucket.

You’re ready to initiate the automationRunPatchBaselineRunbook directly from the Systems Manager Automation console.

Walkthrough

Remediate vulnerabilities using the AWS Systems Manager Automation console

Let’s walk through how to invoke the automationRunPatchBaselineRunbook Automation directly in an OU or account to resolve Amazon Inspector findings based on tag and Amazon Inspector finding severity.

  1. In the Systems Manager delegated administrator account, navigate to the AWS Systems Manager console, select Automation from the left navigation pane, and choose to Execute automation.
  2. On the Choose document page, select Owned by me and pick the automationRunPatchBaselineRunbook-******. Then and choose the Next button as illustrated in the following figure.

The runbook name contains a prefix from the AWS CloudFormation stack’s name.

Figure 3: Systems Manager Automation choose document page

Figure 3: Systems Manager Automation choose document page.

  1. On the Execute automation document page, choose Multi-account and Region.

Figure 4: Execution options

Figure 4: Execution options.

  1. In the Target accounts and Regions section,
    1. Use the Accounts and organizational (OUs) field to specify the different AWS accounts or OUs where you want to run the automation, as illustrated in the following figure. Separate multiple accounts or OUs with a comma.The automation won’t recursively enumerate through nested OUs structures. You need to target the OU containing the desired account or execute the automation for each child’s OU ID.
    2. Use the AWS Regions list to choose one or more Regions where you want to run the automation.
    3. Enter resolveInspectorExecutionRole under Automation Execution Role Name.
    4. Use the Multi-Region and account rate control options to restrict the automation to a limited number of accounts running in a limited number of regions at a time. These options don’t limit the number of AWS resources that can run the automation.

Figure 5: Target accounts and regions
Figure 5: Target accounts and regions.

  1. In the Target accounts and Regions section,
    1. Use InstanceIds under Parameters.
    2. Use Tags under Targets.
    3. Enter the tag key/value pair assigned to the Amazon EC2 instances you want to remediate and select Add.

Note that you can only use one tag key/value pair.

Figure 6: Automation targets

Figure 6: Automation targets.

  1. For the Input parameters section, perform the following steps
    1. For SeverityFilter, select the severity of the Amazon Inspector findings that you want to remediate. Note that you can only choose one severity per execution. Specify NONE to remediate all Amazon Inspector findings associated with the Amazon EC2 instance.
    2. For IncludeKernelUpdates, enter ‘true’ if you want to install kernel updates to remediate kernel vulnerabilities, or enter ‘false’ to skip kernel updates.
    3. Leave the InspectorFindingArn parameter empty.
    4. For RebootOption, choose the appropriate reboot behavior.
    5. The Automation runbook pre-populates the parameters for AutomationAssumeRole, Operation, S3BucketName, Source, and S3BucketKey, don’t change these values.
  1. Confirm the parameters, as shown in the following, and choose Execute.

Figure 7: Systems Manager Automation document input parameter section

Figure 7: Systems Manager Automation document input parameter section

  1. Use the options in the Rate control section to restrict the number of AWS resources that can run the Automation within each account-Region pair.
  2. Choose an option in the Error threshold section.
  3. You can monitor the Automation execution status on the next page, as illustrated in the following figure.

Figure 8: Systems Manager Automation execution status detail page

Figure 8: Systems Manager Automation execution status detail page.

  1. Once patching the affected packages completes the Amazon Inspector findings update to Closed status.

Notes

  1. This method will only remediate Amazon Inspector Package Vulnerability findings associated with Amazon EC2 instances. It doesn’t remediate Amazon Inspector Network reachability findings or Amazon Inspector Package Vulnerability findings associated with container images in Amazon ECR.
  2. The Automation runbook installs the latest version of the affected packages as available in the package manager repositories.
  3. You can only use one tag key/value pair with this method.

Conclusion

This series demonstrates how the new Amazon Inspector can help you detect vulnerabilities in Amazon EC2 instances. We walked through two on-demand automation methods to remediate Amazon Inspector findings for package vulnerabilities on Amazon EC2 instances.

In Part 1, we walked through how to use the Security Hub custom actions Rem-Inspector-RBT and Rem-Inspector-NoRBT to remediate selected Amazon Inspector findings for a specific CVE across Amazon EC2 instances in different AWS accounts and regions.

In Part 2, we discussed remediating Amazon Inspector findings at scale by initiating the Automation runbook, automationRunPatchBaselineRunbook, directly from the Automation console.

Suppose you want to remediate Inspector findings for selected EC2 instances affected by a specific vulnerability. In that case, you should use the Security Hub custom action method described in part 1. You should use the Systems Manager Automation runbook direct invocation method described in this blog to patch all software vulnerabilities identified by Inspector for one or more EC2 instances matching a specified tag.

The automation runbook can help you reduce the operational complexity of manual updates, reduce errors, and increase the remediation speed.

Authors:

Erik Weber

Erik Weber is a Worldwide Specialist Solutions Architect for AWS Cloud Operations services. He specializes in AWS Systems Manager, AWS Config, AWS CloudTrail, and AWS Audit Manager. Erik has a passion for hiking, cooking, and biking outside of work.

https://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/05/26/Priyank-Ghedia.jpg

Priyank Ghedia

Priyank is a Solutions Architect focused on Threat Detection and Incident Response. Priyank helps customers meet their security visibility and response objectives by building architectures using AWS security services and tools. Before AWS, he spent eight years advising customers on global networking and security operations.

Praveen Haranahalli

Praveen is a Solutions Architect at AWS, based in Orlando, Florida. Praveen has helped a diverse set of customers design and operate various workloads using AWS and has a keen interest in Security and Governance. He loves being outdoors with his family outside of work.