AWS Security Blog

Continuous compliance monitoring with Chef InSpec and AWS Security Hub

In this post, I will show you how to run a Chef InSpec scan with AWS Systems Manager and Systems Manager Run Command across your managed instances. InSpec is an open-source runtime framework that lets you create human-readable profiles to define security, compliance, and policy requirements and then test your Amazon Elastic Compute Cloud (Amazon EC2) instances against those profiles. InSpec profiles can also be used to make sure certain network ports aren’t reachable, to verify that certain packages are not installed, and/or to confirm that certain processes are running on your instances.

InSpec is integrated within AWS Systems Manager, an AWS service that you can use to view and control your infrastructure on AWS. InSpec compliance scans are run by using an AWS Systems Manager document (SSM document), which installs InSpec on your servers and removes InSpec after scans are completed.

In this post, you will create the supporting infrastructure to collect non-compliant findings from AWS Systems Manager Configuration Compliance and send them to AWS Security Hub via a Lambda function. You will also explore methods to correlate finding information in Security Hub for non-compliant resources.

Note: AWS Systems Manager (Systems Manager) was formerly known as “Amazon Simple Systems Manager (SSM)” and “Amazon EC2 Systems Manager (SSM).” The original abbreviated name of the service, “SSM,” is still reflected in various AWS resources. For more information, see Systems Manager Service Name History.

Solution overview

The following diagram shows the flow of events in the solution I describe in this post.
 

Figure 1: Architecture diagram

Figure 1: Architecture diagram

  1. Invoke an AWS-RunInSpecChecks document on-demand by using Run Command against your target instances (State Manager is another option for scheduling InSpec scans, but is not covered in this post).
  2. Systems Manager downloads the InSpec Ruby files from Amazon Simple Storage Service (Amazon S3), installs InSpec on your server, runs the scan, and removes InSpec when complete.
  3. AWS Systems Manager pushes scan results to the Compliance API and presents the information in the Systems Manager Compliance console, to include severity and compliance state.
  4. A CloudWatch Event is emitted for Compliance state changes.
  5. A CloudWatch Event Rule listens for these state changes and when detected, invokes a Lambda function.
  6. Lambda calls the Compliance APIs for additional data about which InSpec check failed.
  7. Lambda calls the EC2 APIs to further enrich the data about the non-compliant instance.
  8. Lambda maps these details to the AWS Security Finding Format and sends them to Security Hub.

To support the steps above, you will deploy a CloudFormation template that creates a CloudWatch Event Rule and a placeholder Lambda function. You will then create an InSpec profile, upload it to Amazon S3, and use Run Command to invoke an InSpec compliance scan.

When the scan completes, you can then search for the findings in Security Hub. You can create saved searches with insights in AWS Security Hub, and use different filtering to correlate InSpec compliance failures with other information from Amazon Inspector and Amazon GuardDuty.

Prerequisites

Getting started

Before creating your CloudFormation stack, you’ll upload your InSpec profile and Lambda function package to an S3 bucket.

To upload the Lambda package:

  1. Download the Lambda function ZIP file from GitHub.
  2. Save the file to your workstation. In your S3 bucket, choose Upload to upload the InSpecToSecurityHub.zip file.

The next step is to create and upload an InSpec profile to Amazon S3.

To create and upload an InSpec profile:

  1. From your workstation, create a file named Inspec_SSH.rb and paste in the code that follows: 
    
	control 'Linux instance check' do
    title 'SSH access'
    desc 'SSH port should not be open to the world'
    impact 0.9
    require 'rbconfig'
    is_windows = (RbConfig::CONFIG['host_os'] =~ /mswin|mingw|cygwin/)
    if ! is_windows
      describe port(22) do
        it { should be_listening }
        its('addresses') {should_not include '0.0.0.0'}
      end
    end
  end

  2. Save your file, and in your S3 bucket, choose Upload to upload the Inspec_SSH.rb file.

Note: The InSpec profile in the example code above ensures that SSH (Port 22) is listening on your instance and that SSH access is not publicly available, as noted by {should_not include ‘0.0.0.0’}. For other InSpec profiles, see the DevSec chef-os-hardening project on GitHub for profiles to help you secure your instances and servers, or you can learn more about Compliance Automation with InSpec on the Chef website.

You will now deploy the CloudFormation template to finish setting up the solution.

To deploy the CloudFormation template:

  1. Download the CloudFormation template from GitHub and create a CloudFormation stack.

    Note: For more information about how to create a CloudFormation stack, see Getting Started with AWS CloudFormation in the AWS CloudFormation User Guide.

  2. Under Parameters, enter the name of the bucket you uploaded the package to, as shown in Figure 2, and finish creating your stack
     
    Figure 2: CloudFormation parameters

    Figure 2: CloudFormation parameters

After your stack is successfully created, you will run an InSpec scan.

Run an InSpec scan

Now that you have your Lambda function and InSpec profile ready, you will run your InSpec scan against one or more managed instances with Run Command.

  1. Navigate to the AWS Systems Manager Console and in the navigation pane, choose Run Command.
  2. In the search box, enter AWS-RunInspecChecks. In the search results, select AWS-RunInspecChecks.
  3. For Source Type, select S3. The Source Info format is: 
    
	{"path":"https://s3.amazonaws.com/BUCKET_NAME/PATH/ Inspec_SSH.rb"}

    Change BUCKET_NAME, PATH, and the InSpec profile name as appropriate, and enter it in the Source Info box, as shown in Figure 3.

  4. Under Targets, select Choose instances manually and select one or more instances, as shown in Figure 3.
     
    Figure 3: Run Command interface

    Figure 3: Run Command interface

  5. Scroll down to Output options, and select S3, CloudWatch Logs, or both, then select Run.
  6. In the Run Command console for the specific Command invocation you just sent, you will see your instances and their status, as shown in Figure 4. Refresh the page after a few minutes to see if the invocation was successful.
     
    Figure 4: Run Command invocation status

    Figure 4: Run Command invocation status

  7. If you see anything other than Success, you can find information about different statuses in Understanding Command Statuses. You can also see Troubleshooting Systems Manager Run Command.
  8. In the AWS Systems Manager Console, navigate to the Compliance menu to see your full Compliance resources summary, as shown in Figure 5. You may also see additional compliance types and statuses. Only failed Custom:InSpec compliance types will make their way to Security Hub.
     
    Figure 5: SSM Compliance dashboard

    Figure 5: SSM Compliance dashboard

Analyze with Security Hub

After Lambda has sent your non-compliant InSpec results to Security Hub, you can create saved searches by using Security Hub insights to do basic correlation.

  1. Navigate to the Security Hub console and in the navigation pane, choose Findings.
  2. To find InSpec-related findings, select the search bar, and choose Product fields.
  3. For the Key field, enter Provider Name and for the Value field enter AWS Systems Manager Compliance, then choose Apply, as shown in Figure 6.
     
    Figure 6: User defined field - filter

    Figure 6: User defined field – filter

    This will give you your InSpec-related findings, as shown in Figure 7, but there is not enough data to correlate across to other findings from GuardDuty or Amazon Inspector.
     

    Figure 7: Security Hub InSpec findings

    Figure 7: Security Hub InSpec findings

  4. Navigate the Insights menu on the navigation pane and choose Create insight on the top-right.
  5. Select the search bar, enter Resource type, enter AwsEc2Instance and select Apply, as shown in Figure 8. This will group all findings related to EC2 instances together.
     
    Figure 8: Filter insights for EC2 instances

    Figure 8: Filter insights for EC2 instances

  6. Select the search bar again, select Group by, scroll down to select Severity label and select Apply. We now can see our total findings by severity across all EC2 instances, as shown in Figure 9.
     
    Figure 9: Insights grouped by severity

    Figure 9: Insights grouped by severity

  7. On the top right select Create insight, enter a name such as EC2 Instances by Severity and choose Create insight to save your insight.
  8. Navigate back to the main Insights menu and select your new Insight for more detailed graphics to appear on the right-hand side, as shown in Figure 10.
     
    Figure 10: Detailed Insight menu

    Figure 10: Detailed Insight menu

    You can hover over the values to get the exact count and select them to filter down further on details such as the ARN of the resource, the name of the product, or what account the finding originated from. This view can be used by security analysts who are looking to do fast correlation of findings for a resource or to take action on high severity findings first.

Conclusion

In this post, I showed you how to run InSpec scans to monitor the compliance of your instances against your policy requirements, as defined by InSpec profiles. InSpec can help identify when certain points are improperly configured or publicly accessible. By using Systems Manager, you can continuously monitor the compliance against these profiles with State Manager, and run these checks on demand by using Run Command. Systems Manager allows you to rapidly scale across your managed instances, and enriches instance data through the SSM Agent.

You also learned how to use Security Hub to perform correlation within the findings menu, so you can quickly search based on resource type, or the VPC and subnet that the EC2 instances are in. When you have an instance that fails compliance against an InSpec profile, you can look for any threat-related findings from GuardDuty, or vulnerability information from Amazon Inspector or partner integrations.

To avoid incurring additional charges from the resources created in this blog, delete the CloudFormation stack you deployed. For more information on Security Hub pricing, see the AWS Security Hub Pricing page.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the Security Hub forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Jonathan Rau

Jonathan is the Senior TPM for AWS Security Hub. He holds an AWS Certified Specialty-Security certification and is extremely passionate about cyber security, data privacy, and new emerging technologies, such as blockchain. He devotes personal time into research and advocacy about those same topics.