Building STIG-compliant AMIs for Amazon EKS

As more organizations required to run hardened virtual machines to increase security to meet the internal compliance adopt Kubernetes, there is a need for hardened Amazon Machine Images (AMIs) that work with Amazon Elastic Kubernetes Service (Amazon EKS).

There are multiple options to choose from. One solution is to use Bottlerocket, a special-purpose OS from AWS designed for running Linux containers. Bottlerocket includes only the essential software required to run containers, thereby effectively reducing the attack surface and ensuring that the underlying software is always secure. Another option is to consider using hardened images from the AWS Marketplace.

In this post, we discuss the different approaches you can take to achieve a hardened AMI that works with Amazon EKS. We demonstrate how to create an Amazon Elastic Compute Cloud (Amazon EC2) Image Builder pipeline that applies operating system security requirements defined by the Defense Information Systems Agency (DISA) Security Technical Implementation Guides (STIGs) to an Amazon Linux 2 image. You can then add EKS binaries and other related files on the custom image. Finally, the custom image is used to create an EKS cluster.

Approach 1: Apply hardening scripts onto an EKS Optimized Amazon Linux AMI

In this approach, you have existing hardening scripts and a pipeline that you can apply on top of the EKS default AMI to build a hardened image. The following diagram depicts this approach.

From top to bottom, customer "golden" AMI, OS hardening scripts and EKS optimized Amazon Linux AMI

Figure 1. OS Hardening scripts on EKS Optimized Amazon Linux AMI

The Amazon EKS Optimized Amazon Linux AMI is built on top of Amazon Linux 2 and is configured to serve as the base image for Amazon EKS nodes. The AMI is configured to work with Amazon EKS, and it includes Docker, kubelet, and the AWS Identity and Access Management (IAM) Authenticator.

If you have an existing OS hardening script, you can create a pipeline to run these scripts onto the EKS Optimized Amazon Linux AMI to produce a custom AMI to be used with Amazon EKS.

Because AWS regularly applies the latest security patches and operating system updates as part of the latest AMI release version, the AMI ID for Amazon EKS Optimized AMIs is available by querying the AWS Systems Manager Parameter Store API. You can then programmatically run the pipeline to update the custom AMI and update the EKS cluster to use the AMI.

Approach 2: Install EKS-related binaries on a hardened image

In the second approach, instead of using the Amazon EKS Optimized AMI, you can use an existing custom “golden” AMI. The AMI is typically hardened to meet STIG or Center for Internet Security (CIS) standards and might include features such as a vulnerability scanning agent and a couple of monitoring agents. STIGs and CIS are the two primary third-party baselines adopted across public and private organizations. Some enterprises might be inclined to use STIGs as the baseline because they address US government requirements. STIGs are the configuration standards submitted by OS or software vendors to DISA for approval.

Once approved, the configuration standards are used to configure security hardened information systems and software. STIGs contain technical guidance to help secure information systems or software that might otherwise be vulnerable to a malicious attack. DISA develops and maintains STIGs and defines the vulnerability Severity Category Codes (CAT), which are referred to as CAT I, II, and III. Once you have the hardened base image, you can install EKS-related binaries to create a custom hardened AMI. The following diagram depicts this approach where the EKS-related binaries are layered on top of the existing golden AMI.

From top to bottom, EKS binaries on enterprise custom AMI

Figure 2. EKS binaries on Enterprise custom AMI

An Amazon Linux 2 CIS image from the AWS Marketplace can be used as the base image as well. CIS pre-hardens these virtual machine (VM) images to CIS Benchmark standards, and you can add EKS-related binaries. These images do incur some additional cost on the AWS Marketplace.

If you do not have an existing baseline AMI for use with Amazon EKS and wish to create one based on STIG, you can follow the steps in this post to see how to build a golden Linux operating system image that follows STIG compliance guidelines using Amazon EC2 Image Builder.

Create AMI with STIG standards using Amazon EC2 Image Builder

Amazon EC2 Image Builder provides a step-by-step wizard covering the steps to build a golden image that follows STIG compliance guidelines.

Define the source image

The first step is to create a recipe in Amazon EC2 Image Builder. Give the recipe a name.

SCreenshot of Create recipe screen with Name, Version, Description fields

Figure 3. Create recipe

Next define the base OS image to use as the foundation layer of the golden image. You can select an existing image that you own, an image owned by Amazon, or an image shared with you. Under Source Image, select Amazon Linux as the image operating system.

Screenshot of a grid of select managed images

Figure 4. Select managed images

screen shot of selecting image origin radio button options

Figure 5. Select image origin

Under Components, choose the stig-build-linux-high component. Image Builder provides STIG components that you can leverage to quickly build STIG-compliant images on standalone servers by applying local Group Policies. The STIG components of Image Builder scan for misconfigurations and run a remediation script. Image Builder defines the STIG components as low, medium, and high, which align with DISA CAT I, II, and III respectively.

Screenshot of the components with Build Components - Amazon Linux

Figure 6. Choose build components

Once done, select Create Recipe.

Next, we choose Create pipeline from this recipe.

Screenshot of selecting recipe names from list to create pipeline

Input the Pipeline name.

Screenshot of specifying pipeline details

Figure 8. Specify Pipeline Details

Once done, select Create Pipeline and run the pipeline.

Screenshot with dropdown showing Run pipeline selected

Figure 9. Run pipeline

After some time, the output image becomes available.

Screenshot shows list of output images

Install EKS binaries on top of hardened AMI

To create a custom Amazon Linux AMI for Amazon EKS, you must use the following:

HashiCorp Packer (available from the HashiCorp website)
A build specification with resources and configuration scripts from the Amazon EKS AMI repository on AWS GitHub

Note: Packer works using an AWS CloudFormation stack. The stack runs an m4.large or a1.large Amazon Elastic Compute Cloud (Amazon EC2) instance (depending on the target AMI architecture). The instance is provisioned by Packer. After the instance is provisioned with packages and binaries, Packer creates an AMI from the running instance.

Install Packer from the HashiCorp website into your preferred system. Packer supports Windows, Mac, or Linux. In this example, we use an EC2 Linux.

$ sudo yum -y install packer

SCreenshot of Packer available commands

To clone the Amazon EKS AMI repository to your workstation, run the following command:

$ git clone https://github.com/awslabs/amazon-eks-ami && cd amazon-eks-ami

Note: Packer is executed through a series of makefile targets with eks-worker-al2.json as the build specification. The build process uses the amazon-ebs Packer builder (from the HashiCorp website) and launches an instance. The Packer shell provisioner (from the HashiCorp website) runs the install-worker.sh script on the instance to install software and perform other configuration tasks. Then, Packer creates an AMI from the instance and terminates the instance after the AMI is created.

To configure a custom source AMI, set the variables source_ami_id, source_ami_owners, source_ami_filter_name and aws_region in the Packer configuration file eks-worker-al2.json.

An example follows:

"aws_region": "ap-southeast-1",
"source_ami_id": "ami-0cc93a9132225a429",
"source_ami_owners": "<AWS Account ID>",
"source_ami_filter_name": "AL2-STIG-High-Recipe*",

Start the build process by providing the Kubernetes version as the parameter, and run the following command.

$ make 1.21

Screenshot of output of packer build process

The new image will be available in the EC2 AMIs.

Screenshot of AMIs with 1.21 filtered into list

Launch EKS cluster with custom AMI

You can now make use of the custom AMI for Amazon EKS. In this example, eksctl is used to create the EKS cluster.

Download and extract the latest release of eksctl with the following command.

$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

Move the extracted binary to /usr/local/bin.

$ sudo mv /tmp/eksctl /usr/local/bin

Test that your installation was successful with the following command:

$ eksctl version

Create the cluster.yaml with the custom AMI ID:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
 name: managed-cluster
 region: ap-southeast-1

managedNodeGroups:
 - name: custom-ng
   ami: ami-0c51983d4f4f751e7
   overrideBootstrapCommand: |
    #!/bin/bash
    /etc/eks/bootstrap.sh managed-cluster

Next, create the cluster with the cluster.yaml as input.

$ eksctl create cluster --config-file=cluster.yaml

The cluster with the hardened AMIs is now ready for use.

Output $ kubectl get nodes

From the EC2 console, we can see that these custom nodes are using the custom AMI ID.

screenshot showing list of managed cluster nodes

Summary

In this post, we showed how to build a custom STIG compliant hardened AMI that works with Amazon EKS. We used the Amazon EC2 Image Builder to apply STIG components, installed Kubernetes binaries on the custom AMI and created an EKS cluster with the new AMI. This solution is an option for organizations looking to deploy compliant Amazon EKS environments. The resulting images should still be independently audited and scanned by your internal security organization to ensure they meet the latest standards.

If you are looking to learn more about Amazon EKS best practices, please check out the EKS Best Practices Guides.

Containers