Building STIG-compliant AMIs for Amazon EKS
As more organizations required to run hardened virtual machines to increase security to meet the internal compliance adopt Kubernetes, there is a need for hardened Amazon Machine Images (AMIs) that work with Amazon Elastic Kubernetes Service (Amazon EKS).
There are multiple options to choose from. One solution is to use Bottlerocket, a special-purpose OS from AWS designed for running Linux containers. Bottlerocket includes only the essential software required to run containers, thereby effectively reducing the attack surface and ensuring that the underlying software is always secure. Another option is to consider using hardened images from the AWS Marketplace.
In this post, we discuss the different approaches you can take to achieve a hardened AMI that works with Amazon EKS. We demonstrate how to create an Amazon Elastic Compute Cloud (Amazon EC2) Image Builder pipeline that applies operating system security requirements defined by the Defense Information Systems Agency (DISA) Security Technical Implementation Guides (STIGs) to an Amazon Linux 2 image. You can then add EKS binaries and other related files on the custom image. Finally, the custom image is used to create an EKS cluster.
Approach 1: Apply hardening scripts onto an EKS Optimized Amazon Linux AMI
In this approach, you have existing hardening scripts and a pipeline that you can apply on top of the EKS default AMI to build a hardened image. The following diagram depicts this approach.
The Amazon EKS Optimized Amazon Linux AMI is built on top of Amazon Linux 2 and is configured to serve as the base image for Amazon EKS nodes. The AMI is configured to work with Amazon EKS, and it includes Docker, kubelet, and the AWS Identity and Access Management (IAM) Authenticator.
If you have an existing OS hardening script, you can create a pipeline to run these scripts onto the EKS Optimized Amazon Linux AMI to produce a custom AMI to be used with Amazon EKS.
Because AWS regularly applies the latest security patches and operating system updates as part of the latest AMI release version, the AMI ID for Amazon EKS Optimized AMIs is available by querying the AWS Systems Manager Parameter Store API. You can then programmatically run the pipeline to update the custom AMI and update the EKS cluster to use the AMI.
Approach 2: Install EKS-related binaries on a hardened image
In the second approach, instead of using the Amazon EKS Optimized AMI, you can use an existing custom “golden” AMI. The AMI is typically hardened to meet STIG or Center for Internet Security (CIS) standards and might include features such as a vulnerability scanning agent and a couple of monitoring agents. STIGs and CIS are the two primary third-party baselines adopted across public and private organizations. Some enterprises might be inclined to use STIGs as the baseline because they address US government requirements. STIGs are the configuration standards submitted by OS or software vendors to DISA for approval.
Once approved, the configuration standards are used to configure security hardened information systems and software. STIGs contain technical guidance to help secure information systems or software that might otherwise be vulnerable to a malicious attack. DISA develops and maintains STIGs and defines the vulnerability Severity Category Codes (CAT), which are referred to as CAT I, II, and III. Once you have the hardened base image, you can install EKS-related binaries to create a custom hardened AMI. The following diagram depicts this approach where the EKS-related binaries are layered on top of the existing golden AMI.
An Amazon Linux 2 CIS image from the AWS Marketplace can be used as the base image as well. CIS pre-hardens these virtual machine (VM) images to CIS Benchmark standards, and you can add EKS-related binaries. These images do incur some additional cost on the AWS Marketplace.
If you do not have an existing baseline AMI for use with Amazon EKS and wish to create one based on STIG, you can follow the steps in this post to see how to build a golden Linux operating system image that follows STIG compliance guidelines using Amazon EC2 Image Builder.
Create AMI with STIG standards using Amazon EC2 Image Builder
Amazon EC2 Image Builder provides a step-by-step wizard covering the steps to build a golden image that follows STIG compliance guidelines.
Define the source image
The first step is to create a recipe in Amazon EC2 Image Builder. Give the recipe a name.
Next define the base OS image to use as the foundation layer of the golden image. You can select an existing image that you own, an image owned by Amazon, or an image shared with you. Under Source Image, select Amazon Linux as the image operating system.
Under Components, choose the stig-build-linux-high component. Image Builder provides STIG components that you can leverage to quickly build STIG-compliant images on standalone servers by applying local Group Policies. The STIG components of Image Builder scan for misconfigurations and run a remediation script. Image Builder defines the STIG components as low, medium, and high, which align with DISA CAT I, II, and III respectively.
Once done, select Create Recipe.
Next, we choose Create pipeline from this recipe.
Input the Pipeline name.
Once done, select Create Pipeline and run the pipeline.
After some time, the output image becomes available.
Install EKS binaries on top of hardened AMI
To create a custom Amazon Linux AMI for Amazon EKS, you must use the following:
- HashiCorp Packer (available from the HashiCorp website)
- A build specification with resources and configuration scripts from the Amazon EKS AMI repository on AWS GitHub
Note: Packer works using an AWS CloudFormation stack. The stack runs an m4.large or a1.large Amazon Elastic Compute Cloud (Amazon EC2) instance (depending on the target AMI architecture). The instance is provisioned by Packer. After the instance is provisioned with packages and binaries, Packer creates an AMI from the running instance.
Install Packer from the HashiCorp website into your preferred system. Packer supports Windows, Mac, or Linux. In this example, we use an EC2 Linux.
$ sudo yum -y install packer
To clone the Amazon EKS AMI repository to your workstation, run the following command:
Note: Packer is executed through a series of makefile targets with eks-worker-al2.json as the build specification. The build process uses the amazon-ebs Packer builder (from the HashiCorp website) and launches an instance. The Packer shell provisioner (from the HashiCorp website) runs the
install-worker.sh script on the instance to install software and perform other configuration tasks. Then, Packer creates an AMI from the instance and terminates the instance after the AMI is created.
To configure a custom source AMI, set the variables
aws_region in the Packer configuration file
An example follows:
Start the build process by providing the Kubernetes version as the parameter, and run the following command.
$ make 1.21
The new image will be available in the EC2 AMIs.
Launch EKS cluster with custom AMI
You can now make use of the custom AMI for Amazon EKS. In this example,
eksctl is used to create the EKS cluster.
Download and extract the latest release of
eksctl with the following command.
Move the extracted binary to /usr/local/bin.
Test that your installation was successful with the following command:
cluster.yaml with the custom AMI ID:
Next, create the cluster with the
cluster.yaml as input.
The cluster with the hardened AMIs is now ready for use.
From the EC2 console, we can see that these custom nodes are using the custom AMI ID.
In this post, we showed how to build a custom STIG compliant hardened AMI that works with Amazon EKS. We used the Amazon EC2 Image Builder to apply STIG components, installed Kubernetes binaries on the custom AMI and created an EKS cluster with the new AMI. This solution is an option for organizations looking to deploy compliant Amazon EKS environments. The resulting images should still be independently audited and scanned by your internal security organization to ensure they meet the latest standards.
If you are looking to learn more about Amazon EKS best practices, please check out the EKS Best Practices Guides.