AWS Security Blog

How to Set Up Continuous Golden AMI Vulnerability Assessments with Amazon Inspector

As companies mature in their cloud journey, they implement layered security capabilities and practices in their cloud architectures. One such practice is to continually assess golden Amazon Machine Images (AMIs) for security vulnerabilities. AMIs provide the information required to launch an Amazon EC2 instance, which is a virtual server in the AWS Cloud. A golden AMI is an AMI that contains the latest security patches, software, configuration, and software agents that you need to install for logging, security maintenance, and performance monitoring. You can build and deploy golden AMIs in your environment, but the AMIs quickly become dated as new vulnerabilities are discovered.

A security best practice is to perform routine vulnerability assessments of your golden AMIs to identify if newly found vulnerabilities apply to them. If you identify a vulnerability, you can update your golden AMIs with the appropriate security patches, test the AMIs, and deploy the patched AMIs in your environment. In this blog post, I demonstrate how to use Amazon Inspector to set up such continuous vulnerability assessments to scan your golden AMIs routinely.

Solution overview

Amazon Inspector performs security assessments of Amazon EC2 instances by using AWS managed rules packages such as the Common Vulnerabilities and Exposures (CVEs) package. The solution in this post creates EC2 instances from golden AMIs and then runs an Amazon Inspector security assessment on the created instances. When the assessment results are available, the solution consolidates the findings and advises you about next steps. Furthermore, the solution schedules an Amazon CloudWatch Events rule to run the golden AMI vulnerability assessments on a regular basis.

The following solution diagram illustrates how this solution works.

Solution diagram showing how this post's solution works

Here’s how this solution works, as illustrated in the preceding diagram:

  1. A scheduled CloudWatch Events event triggers the StartContinuousAssessment AWS Lambda function, which starts the security assessment of your golden AMIs.
  2. The StartContinuousAssessment Lambda function performs the following actions:
    1. It reads a JSON parameter stored in the AWS Systems Manager (Systems Manager) Parameter Store. This JSON parameter contains the following metadata for each golden AMI:
      1. InstanceType – A valid instance-type for launching an EC2 instance of the golden AMI.
      2. Ami-Id – The ID of the golden AMI.
      3. UserData – An operating system–compatible user-data script for installing the Amazon Inspector agent.

Later in this blog post, I provide instructions for creating this JSON parameter.

    1. For each AMI specified in the JSON parameter, the Lambda function creates an EC2 instance. When each instance starts, it installs the Amazon Inspector agent by using the user-data script provided in the JSON. The Lambda function then copies each golden AMI’s tags (you will assign custom metadata in the form of tags to each golden AMI when you set up the solution) to the corresponding EC2 instance. The function also adds a tag with the key of continuous-assessment-instance and value as true. This tag identifies EC2 instances that require regular security assessments. The Lambda function copies the AMI’s tags to the instance (and later, to the security findings found for the instance) to help you identify the golden AMIs for each security finding. After you analyze security findings, you can patch your golden AMIs.
    2. The first time the StartContinuousAssessment function runs, it creates:
      1. An Amazon Inspector assessment target: The target identifies EC2 instances to assess by using the continuous-assessment-instance tag.
      2. An Amazon Inspector assessment template: The template contains a reference to the Amazon Inspector assessment target created in the preceding step and the following AWS managed rules packages to evaluate:

    For subsequent assessments, the StartContinuousAssessment function reuses the target and the template created during the first run of StartContinuousAssessment function.

    Note: Amazon Inspector can start an assessment only after it finds at least one running Amazon Inspector agent. To allow EC2 instances to boot and the Amazon inspector agent to start, the Lambda function waits four minutes. Because the assessment runs for approximately one hour and boot time for EC2 instances typically takes a few minutes, all Amazon Inspector agents start before the assessment ends.

      1. The Lambda function then runs the assessment. The Amazon Inspector agents collect behavior and configuration data, and pass it to Amazon Inspector. Amazon Inspector analyzes the data and generates Amazon Inspector findings, which are possible security findings you may need to address.
      2. After the Lambda function completes the assessment, Amazon Inspector publishes an assessment-completion notification message to an Amazon SNS topic called ContinuousAssessmentCompleteTopic. SNS uses topics, which are communication channels for sending messages and subscribing to notifications.
      3. The notification message published to SNS triggers the AnalyzeInspectorFindings Lambda function, which performs the following actions:
        1. Associates the tags of each EC2 instance with security findings found for that EC2 instance. This enables you to identify the security findings using the app-name tag you specified for your golden AMIs. You can use the information provided in the findings to patch your golden AMIs.
        2. Terminates all instances associated with the continuous-assessment-instance=true tag.
        3. Aggregates the number of findings found for each EC2 instance by severity and then publishes a consolidated result to an SNS topic called ContinuousAssessmentResultsTopic.

How to deploy the solution

To deploy this solution, you must set it up in the AWS Region where you build your golden AMIs. If that AWS Region does not support Amazon Inspector, at the end of your continuous integration pipeline, you can copy your AMIs to an AWS Region where Amazon Inspector assessments are supported. To learn more about continuous integration pipelines, see What is Continuous Integration?

To deploy continuous golden AMI vulnerability assessments in your AWS account, follow these steps:

  1. Tag your golden AMIs – Tagging your golden AMIs lets you search assessment result findings based on tags after Amazon Inspector completes an assessment.
  2. Store your golden AMI metadata in the Systems Manager Parameter Store – Prepare and store the golden AMI metadata in the Systems Manager Parameter Store. The StartContinuousAssessment Lambda function reads golden AMI metadata and starts assessing for vulnerabilities.
  3. Run the supplied AWS CloudFormation template and subscribe to an SNS topic to receive assessment results – Set up the infrastructure required to run vulnerability assessments and subscribe to an SNS topic to receive assessment results via email.
  4. Test golden AMI vulnerability assessments – Ensure you have successfully set up the required resources to run vulnerability assessments.
  5. Set up a CloudWatch Events rule for triggering continuous golden AMI vulnerability assessments – Schedule the execution of vulnerability assessments on a regular basis.

1.  Tag your golden AMIs

You can search assessment findings based on golden AMI tags after Amazon Inspector completes an assessment.

To tag a golden AMI by using the AWS Management Console:

  1. Sign in to the AWS Management Console and then navigate to the EC2 console.
  2. In the navigation pane, choose AMIs.
  3. Choose your AMI from the list, and then choose Actions > Add/Edit Tags.
  4. Choose Create Tag. In the Key column, type app-name. In the Value column, type your application name. Following the same steps, create the app-version and app-environment tags. Choose Save.

Now that you have tagged your golden AMIs, you need to create golden AMI metadata, which will be read by the StartContinuousAssessment function to initiate vulnerability assessments. You will store the golden AMI metadata in the Systems Manager Parameter Store.

2.  Store your golden AMI metadata in the Systems Manager Parameter Store

This solution reads golden AMI metadata from a parameter stored in the Systems Manager Parameter Store. The metadata must be in JSON format and must contain the following information for each golden AMI:

  • Ami-Id
  • InstanceType
  • UserData

Step A: Find the AMI ID of your golden AMI.

An AMI ID uniquely identifies an AMI in an AWS Region and is a required parameter for launching an EC2 instance from a golden AMI. To find the AMI ID of your golden AMI:

  1. Sign in to the AWS Management Console and navigate to the EC2 console.
  2. In the navigation pane, choose AMIs.
  3. Choose your AMI from the list and then note the corresponding value in the AMI ID column.

Step B: Find a compatible InstanceType for your golden AMI.

Each AMI has a list of compatible InstanceTypes. The InstanceType is a required parameter for launching an EC2 instance from a golden AMI. To find a compatible InstanceType for your golden AMI:

  1. Sign in to the AWS Management Console and navigate to the EC2 console.
  2. Choose Launch Instance. On the Choose an Amazon Machine Image (AMI) page, choose My AMIs.
  3. Type the AMI ID that you noted in Step A in the Search my AMIs box, and then choose Enter.
  4. The search result will contain your golden AMI. To choose it, choose Select.
  5. Locate any available Instance Type and then note the corresponding value in the Type column.
  6. Choose Cancel.

Note: Amazon Inspector will launch the chosen InstanceType every time the vulnerability assessment runs.

Step C: Create the user-data script to install and start the Amazon Inspector agent.

The user-data script automates the installation of software packages when an EC2 instance launches for the first time. In this step, you create an operating system specific, JSON-compatible user-data script that installs and starts the Amazon Inspector agent.

  1. Identify the command that installs the Amazon Inspector agent

Based on Installing Amazon Inspector Agents, the following shell command installs the Amazon Inspector agent on an Amazon Linux-based EC2 instance.

wget https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install
bash install

To find this command for other operating systems, see Installing Amazon Inspector Agents.

  1. Identify the command that starts the Amazon Inspector agent

The following shell command starts the Amazon Inspector agent on an Amazon Linux-based EC2 instance.

sudo /etc/init.d/awsagent start

To find this command for other operating systems, see Amazon Inspector Agents.

  1. Create a script by concatenating the commands from the preceding two steps

The following is a sample concatenated script for the Amazon Linux operating system that installs and starts an Amazon Inspector agent.

wget https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install
bash install
sudo /etc/init.d/awsagent start
  1. Make the script user-data compatible

Based on Running Commands on Your Linux Instance at Launch, you make a Linux shell script user-data compatible by prefixing it with a #!/bin/bash. In this step, you add the #!/bin/bash prefix to the script from the preceding step. The following is the user-data compatible version of the script from the preceding step. 

#!/bin/bash
wget https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install
bash install
sudo /etc/init.d/awsagent start

To make your script user-data compatible for Windows, see Running Commands on Your Windows Instance at Launch.

The user-data script provided in the JSON metadata must be JSON-compatible, which you will do next.

  1. Make the user-data script JSON compatible

To make the user-data script JSON compatible, you must replace all new-line characters with a \r\n\r\n sequence. The following is the JSON-compatible user-data script that you specify for your Amazon Linux-based golden AMI in Step D.

JSON-compatible-user-data-for-Amazon-Linux-AMI

#!/bin/bash \r\n\r\nwget https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install \r\n\r\nbash install \r\n\r\nsudo /etc/init.d/awsagent start

Repeat Steps A, B, and C to find the Ami Id, InstanceType, and UserData for each of your golden AMIs. When you have this metadata, you can create the JSON document of metadata for all your golden AMIs. The StartContinuousAssessment Lambda function reads this JSON to start golden AMI vulnerability assessments.

Step D: Create a JSON document of metadata of all your golden AMIs.

Use the following template to create a JSON document:

[	
    { 
        "instanceType": "instance-type-of-first-AMI", 
        "ami-id": "AMI-ID-of-first-AMI", 
        "userData": "JSON-compatible-user-data-of-first-AMI"
     },
    { 
        "instanceType": "instance-type-of-second-AMI",
        "ami-id": "AMI-ID-of-second-AMI",
        "userData": "JSON-compatible-user-data-of-second-AMI" 
    }
]

Replace all placeholder values with values corresponding to your first golden AMI. If your golden AMI is Amazon Linux-based, you can specify the userData as the JSON-compatible-user-data-for-Amazon-Linux-AMI from Step C.5. Next, replace the placeholder values for your second golden AMI. You can add more entries to your JSON document, if you have more than two golden AMIs.

Note: The total number of characters in the JSON document must be fewer than or equal to 4,096 characters, and the number of golden AMIs must be fewer than 500. You must verify whether your account has permissions to run one on-demand EC2 instance for each of your golden AMIs. For information about how to verify service limits, see Amazon EC2 Service Limits.

Now that you have created the JSON document of your golden AMIs, you will store the JSON document in a Systems Manager parameter. The StartContinuousAssessment Lambda function will read the metadata from this parameter.

Step E: Store the JSON in a Systems Manager parameter.

To store the JSON in a Systems Manager parameter:

  1. Sign in to the AWS Management Console and navigate to the EC2 console.
  2. Expand Systems Manager Shared Resources in the navigation pane, and then choose Parameter Store.
  3. Choose Create Parameter.
  4. For Name, type ContinuousAssessmentInput.
  5. In the Description field, type Continuous golden AMI vulnerability assessment process metadata.
  6. For Type, choose String.
  7. Paste the JSON that you created in Step D in the Value field.
  8. Choose Create Parameter. After the system creates the parameter, choose Close.

To set up the remaining components required to run assessments, you will run a CloudFormation template and perform the configuration explained in the next section.

3.  Run the CloudFormation template and subscribe to an SNS topic to receive assessment results

Next, create a CloudFormation stack using the provided CloudFormation template. Before you start, download the CloudFormation template to your computer.

To create a stack:

  1. Sign in to the AWS Management Console and choose CloudFormation in the Services menu.
  2. Click Create Stack.
  3. On the Select Template page, choose Upload a template to Amazon S3.
  4. Choose Choose File and then choose the CloudFormation template you just downloaded. Choose Next.
  5. On the Specify Details page, specify the Stack Name as AmazonInspectorAssessment. Choose Next.
  6. On the Options page, choose Next.
  7. On the Review page, choose the check box next to the following message: “I acknowledge that AWS CloudFormation might create IAM resources.”
  8. Choose Create. The CloudFormation template creates SNS topics, AWS Identity and Access Management (IAM) roles, and Lambda functions.
  9. On the Stacks page, choose AmazonInspectorAssessment.
  10. In the Detail pane, choose Outputs to view the output of your stack.

After CloudFormation successfully creates a stack, the Outputs tab displays following results:

  • StartContinuousAssessmentLambdaFunction – The Value box displays the name of the StartContinuousAssessment function. You will run this function to trigger the entire workflow.
  • ContinuousAssessmentResultsTopic – The Value box displays the ContinuousAssessmentResultsTopic topic’s Amazon Resource Name (ARN), which you will use later.

To receive consolidated vulnerability assessment results in email, you must subscribe to ContinuousAssessmentResultsTopic.

To subscribe to ContinuousAssessmentResultsTopic:

  1. Sign in to the AWS Management Console and navigate to the SNS console.
  2. Choose Create subscription. In the Topic ARN field, paste the ARN of ContinuousAssessmentResultsTopic that you noted in the previous section.
  3. In the Protocol drop-down, choose Email.
  4. In the Endpoint box, type the email address where you will receive notifications.
  5. Choose Create subscription.
  6. Navigate to your email application and open the message from AWS Notifications. Click the link to confirm your subscription to the SNS topic.

4.  Test golden AMI vulnerability assessments

Before you schedule vulnerability assessments, you should test the process by running the StartContinuousAssessment function. In this test, you trigger a security assessment and monitor it. You then receive an email after the assessment has completed, which shows that vulnerability assessments have been successfully set up.

To start golden AMI vulnerability assessments:

  1. Sign in to the AWS Management Console and choose Lambda in the Services menu.
  2. Choose Functions. In the Functions pane, choose the StartContinuousAssessment function.
  3. Choose the Select a test event drop-down, and choose Configure test events.
  4. On the Configure test event page, choose Create new test event and specify the event name as test.
  5. Paste the following JSON in the editor box. 
    {
"AMIsParamName": "ContinuousAssessmentInput"
}
  6. Choose Create. Choose Test.

The StartContinuousAssessment function runs for approximately five minutes and then displays the following message.
Message showing the function has run successfully

Next, open Amazon Inspector and monitor the progress of the assessment:

  1. Sign in to the AWS Management Console and navigate to the Amazon Inspector console.
  2. On Dashboard under Recent Assessment Runs, you will see an entry with the status, Collecting Data. This status indicates that Amazon Inspector agents are collecting data from instances running your golden AMIs. The agents collect data for an hour and then Amazon Inspector analyzes the collected data.

After Amazon Inspector completes the assessment, the status in the console changes to Analysis complete. Amazon Inspector then publishes an SNS message that triggers the AnalyzeInspectionReports Lambda function. When AnalyzeInspectionReports publishes results, you will receive an email containing consolidated assessment results. You also will be able to see the findings.

To see the findings in Amazon Inspector’s Findings section:

  1. Sign in to the AWS Management Console and navigate to the Amazon Inspector console.
  2. In the navigation pane, choose Assessment Runs. In the table on the Amazon Inspector – Assessment Runs page, choose the findings of the latest assessment run.
  3. Choose the settings (Gear icon) icon and choose the appropriate tags to see the details of findings, as shown in the following screenshot. The findings also contain information about how you can address each underlying vulnerability.
    Screenshot showing details of findings

Having verified that you have successfully set up all components of golden AMI vulnerability assessments, you now will schedule the vulnerability assessments to run on a regular basis to give you continual insight into the health of instances created from your golden AMIs.

5.  Set up a CloudWatch Events rule for triggering continuous golden AMI vulnerability assessments

The last step is to create a CloudWatch Events rule to schedule the execution of the vulnerability assessments on a daily or weekly basis.

To set up a CloudWatch Events rule:

  1. Sign in to the AWS Management Console and navigate to the CloudWatch console.
  2. In the navigation pane, choose Rules > Create rule.
  3. On the Event Source page, choose Schedule. Choose Fixed rate of and specify the interval (for example, 1 day).
  4. For Targets, choose Add target and then choose Lambda function.
  5. For Function, choose the StartContinuousAssessment function.
  6. Choose Configure Input.
  7. Choose Constant (JSON text).
  8. In the box, paste the following JSON code. 
    {
     "AMIsParamName": "ContinuousAssessmentInput"
}
  9. Choose Configure details.
  10. For Rule definition, type ContinuousGoldenAMIAssessmentTrigger for the name, and type as the description, This rule triggers the continuous golden AMI vulnerability assessment process.
  11. Choose Create rule.

The vulnerability assessments are executed on the first occurrence of the schedule you chose while setting up the CloudWatch Events rule. After the vulnerability assessment is executed, you will receive an email to indicate that your continuous golden AMI vulnerability assessments are set up.

Summary

To get visibility into the security of your EC2 instances created from your golden AMIs, it is important that you perform security assessments of your golden AMIs on a regular basis. In this blog post, I have demonstrated how to set up vulnerability assessments, and the results of these continuous golden AMI vulnerability assessments can help you keep your environment up to date with security patches. To learn how to patch your golden AMIs, see Streamline AMI Maintenance and Patching Using Amazon EC2 Systems Manager.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about implementing the solution in this post, start a new thread on the Amazon Inspector forum or contact AWS Support.

– Kanchan and David

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.