Containers

Building Amazon Linux 2 CIS Benchmark AMIs for Amazon EKS

Introduction

The Center for Internet Security (CIS) Benchmarks are best practices for the secure configuration of a target system. They define various Benchmarks for Kubernetes control plane and the data plane. For Amazon EKS clusters, it is strongly recommended to follow the CIS Amazon EKS Benchmark. If the data plane of an Amazon EKS cluster uses Amazon Linux 2 as a node group Operating System, it is recommended to implement the CIS Amazon Linux 2 Benchmark. This blog provides detailed, step-by-step instructions on how customers can build an Amazon EKS Amazon Machine Image (AMI) compliant with the CIS Amazon Linux2 Benchmarks.

As Kubernetes adoption grows, many organizations are choosing it as their platform to build and host their modern and secure applications. Security is one of the primary design criteria for many workloads, especially those dealing with sensitive data, such as financial data processing. These workloads have a stringent requirement to adhere to various security and compliance controls.

Many Amazon EKS customers, especially enterprise customers from Banking and Finance, are looking for guidance from AWS on hardening Amazon EKS. This is primarily for meeting the security and compliance requirements, such as Amazon Linux 2 (AL2) CIS Benchmark Level 1 or Level 2.

Update: This blog post is updated on May 3rd 2023 with additional instructions to modify the build scripts and configurations to accommodate the changes in the Amazon EKS AMI github file structure, CIS Amazon Linux 2 AMI and also to use the latest Amazon EKS Version as of this date.

Amazon EKS AMI Hardening Process

The Amazon Linux 2 (AL2) CIS Benchmarks define two profiles for hardening i.e. Level1 and Level 2.

  • A Level 1 profile is intended to be practical and prudent, provide a clear security benefit, and not inhibit the utility of the technology beyond acceptable means.
  • A Level 2 profile is intended for environments or use cases where security is paramount, acts as a defense in depth measure, and may negatively inhibit the utility or performance of the technology.

There are two approaches for hardening the Amazon EKS AMI for CIS Benchmark Level 1 or Level 2 profiles.

  1. Use the standard Amazon EKS Optimized AMI as a base and add hardening on top of it. This process requires someone to apply all of configuration mentioned in the Amazon Linux 2 CIS Benchmark specification.
  2. Use the Amazon Linux 2 (AL2) CIS Benchmark Level 1 and Level 2AMI from the AWS Marketplace as a base, and add Amazon EKS specific components on top of it. This blog addresses this approach and provides step by step instructions on how build an Amazon EKS hardened AMI, leveraging the Amazon AL2 CIS Benchmark AMI.

The following is a proposed solution for hardening an Amazon EKS AMI and deploying in an Amazon EKS Cluster.

Diagram of the EKS AMI Hardening Process: AWS Marketplace > Github > EKS Hardened AMI > Custom EC2 template > amazon EKS Cluster

The proposed approach is outlined below.

  1. Subscribe to the CIS Amazon Linux 2 Benchmark – Level 2 AMI from the market place.

Note: this will incur costs for the software subscription in addition to the EC2 instance type used.

  1. Build a custom Amazon EKS AMI using above Amazon Linux (AL2) AMI as the base AMI.
  2. Create an Amazon EKS Cluster along with an Amazon EKS managed node group which uses the above custom Amazon EKS AMI.
  3. Deploy a sample application on this node group.

Solution walk through

Pre-requisites

You will need the following to complete the tutorial:

Note: We have tested the CLI steps in this post on Amazon Linux 2.

Let’s start by setting a few environment variables.

export CLUSTER_NAME=eks-cluster
export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

Subscribe to CIS Amazon Linux 2 Benchmark – Level 2 AMI

Note: the below section mentions Level 2 but the same procedure can be used for Level 1.

Go to the CIS Amazon Linux 2 Benchmark – Level 2 AWS Marketplace page. Click on “Continue to Subscribe” on the top-right of the page.

Screenshot of the aws marketplace with redbox around continue to subscribe to CIS Amazon Linux 2 Benchmark-Level 2

On the next page, Accept Terms and Conditions and follow the instructions.

Ensure that the newly subscribed CIS Amazon Linux 2 Benchmark – Level 2 AMI appears in your EC2 console. Sort the AMIs based on the Creation date and select the latest AMI for x86_64 Architecture.

Make a note of the following properties of the AMI and export them into an environment variables

  • AMI ID
  • AMI name
  • Owner account ID
export AMI_ID="ami-xxxxxxxxxxxxxxx"
export AMI_NAME="CIS Amazon Linux 2 xxxxxxxxxxxx"
export AMI_OWNER_ACCOUNT_ID="xxxxxxxxxxx"

Building a custom Amazon EKS AMI

Clone the repo for building a custom Amazon EKS AMI

git clone https://github.com/awslabs/amazon-eks-ami.git
cd amazon-eks-ami/

Install the Packer tool for your platform. Below instructions assume you are using an Amazon Linux based machine to build the AMI.

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install packer

Get the default Amazon Virtual Private Cloud (VPC) ID and get one of the subnet IDs and replace them in eks-worker-al2.json. The packer tool will launch a temporary Amazon EC2 Instance in this subnet to create the custom Amazon EKS AMI.

VPC_ID=$(aws ec2 describe-vpcs --filters=Name=isDefault,Values=true --query 'Vpcs[].VpcId' --output text)
echo $VPC_ID

SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" | jq '.Subnets[0].SubnetId')
SUBNET_ID=`sed -e 's/^"//' -e 's/"$//' <<<"$SUBNET_ID"`
echo $SUBNET_ID

Run the following commands to modify few configuration and script files.

sed -i -e 's#"remote_folder": ""#"remote_folder": "/home/ec2-user"#g' eks-worker-al2-variables.json
sed -i -e 's#"launch_block_device_mappings_volume_size": "4"#"launch_block_device_mappings_volume_size": "20"#g' eks-worker-al2-variables.json
sed -i -e 's#/tmp#/home/ec2-user#g' eks-worker-al2.json
sed -i -e 's#/tmp#/home/ec2-user#g' scripts/install-worker.sh
sed -i -e 's#mktemp -d#mktemp -d --tmpdir=/home/ec2-user/#g' scripts/install-worker.sh
sed -i -e 's#sudo chmod +x $binary#sudo chmod 755 $binary#g' scripts/install-worker.sh
sed -i -e 's#aws --version#sudo /bin/aws --version#g' scripts/generate-version-info.sh
sed -i -e 's#sudo rm -rf /tmp/worker#sudo rm -rf /home/ec2-user/worker#g' scripts/cleanup.sh

Apart from the above changes, add below command to the shell provisioner configurations in the file eks-worker-al2.json as per the githbub issue.

"execute_command": "{{.Vars}} bash '{{.Path}}'"

The updated eks-worker-al2.json file looks like below. The additional changes are highlighted below.

 "provisioners": [
    {
      "type": "shell",
        ...
        "ADDITIONAL_YUM_REPOS={{user `additional_yum_repos`}}"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
      ...
      "inline": [
        "mkdir -p /home/ec2-user/worker/log-collector-script/"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
....
{
      "type": "shell",
        ...
        "sudo mv /home/ec2-user/worker/bin/* /usr/bin/"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
        ...
        "KUBERNETES_VERSION={{user `kubernetes_version`}}",
        "KERNEL_VERSION={{user `kernel_version`}}"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
      ...
        "PAUSE_CONTAINER_VERSION={{user `pause_container_version`}}",
        "CACHE_CONTAINER_IMAGES={{user `cache_container_images`}}"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
      ...
      "script": "{{template_dir}}/scripts/cleanup.sh", "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
       ...
      "environment_vars": [
        "ADDITIONAL_YUM_REPOS={{user `additional_yum_repos`}}"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },
    {
      "type": "shell",
        ...
      "environment_vars": [
        "KERNEL_VERSION={{user `kernel_version`}}"
      ], "execute_command": "{{.Vars}} bash '{{.Path}}'"
    },

Run the following Make command with command line options, which are passed to packer tool as configuration parameters.

make 1.26 aws_region=$AWS_REGION source_ami_id=$AMI_ID source_ami_owners=$AMI_OWNER_ACCOUNT_ID source_ami_filter_name="$AMI_NAME" subnet_id="$SUBNET_ID"

The successful build looks something like below. Make a note of the custom Amazon EKS AMI.

==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
==> amazon-ebs: Running post-processor: manifest
==> amazon-ebs: Running post-processor: manifest
Build 'amazon-ebs' finished after 8 minutes 31 seconds.

==> Wait completed after 8 minutes 31 seconds

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-0f92014aa12e5ab96

Set an environment variable with the above custom Amazon EKS AMI.

export EKS_AMI_ID="ami-xxxxxxxxxxx"

Create Amazon EKS Cluster with EKS Managed Node group using custom EKS AMI

Run the below command to create an Amazon EKS Cluster along with a managed node group using custom Amazon EKS AMI.

cat > cluster.yaml <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: $CLUSTER_NAME
  region: $AWS_REGION
  version: "1.26"

managedNodeGroups:
- name: custom-ng
  desiredCapacity: 2
  amiFamily: AmazonLinux2
  ami: $EKS_AMI_ID
  
  overrideBootstrapCommand: |
      #!/bin/bash
      set -ex
      iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
      /etc/eks/bootstrap.sh $CLUSTER_NAME
EOF


eksctl create cluster -f cluster.yaml

Once the cluster is created, ensure that kubectl is working.

kubectl get nodes

NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-1-4.ec2.internal      Ready    <none>   3m7s   v1.26.2-eks-a59e1f0
ip-192-168-34-254.ec2.internal   Ready    <none>   3m7s   v1.26.2-eks-a59e1f0

Deploy a sample application

Let’s deploy a sample application to these new nodes.

cat > nginx-deploy.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        eks.amazonaws.com/nodegroup: custom-ng
      containers:
      - image: public.ecr.aws/nginx/nginx:latest
        imagePullPolicy: Always
        name: nginx
        resources:
          limits:
            cpu:  100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi        
        ports:
        - name: http
          containerPort: 80
---
EOF

kubectl apply -f nginx-deploy.yaml

Ensure the Nginx pod is running fine on these new nodes.

kubectl get pod -lapp=nginx
NAME                      READY   STATUS    RESTARTS   AGE

nginx-68485cdd49-lwrmm   1/1     Running   0          9s

Run the below command to exec into the pod and run a curl command:

 POD_NAME=$(kubectl get pods -l=app=nginx -o=jsonpath={.items..metadata.name})
kubectl exec -it  $POD_NAME -- /bin/bash
root@nginx-65d78895bd-tfp9s:/# curl 127.0.0.1:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@nginx-65d78895bd-tfp9s:/# exit
exit

Run the below command to access the logs from the pod and then Ctrl + C to exit from the container log output:

kubectl logs -f  $POD_NAME
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/05/03 06:53:22 [notice] 1#1: using the "epoll" event method
2023/05/03 06:53:22 [notice] 1#1: nginx/1.23.4
2023/05/03 06:53:22 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2023/05/03 06:53:22 [notice] 1#1: OS: Linux 5.10.177-158.645.amzn2.x86_64
2023/05/03 06:53:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/05/03 06:53:22 [notice] 1#1: start worker processes
2023/05/03 06:53:22 [notice] 1#1: start worker process 29
2023/05/03 06:53:22 [notice] 1#1: start worker process 30
127.0.0.1 - - [03/May/2023:06:53:46 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/7.74.0" "-"
q^X^C

Cleanup

Use these commands to delete the resources created during this post:

kubectl delete -f nginx-deploy.yaml
eksctl delete cluster -f cluster.yaml --wait

Follow this link to cancel the Amazon Marketplace subscription for the CIS Amazon Linux 2 Benchmark Level 2 AMI.

Conclusion

In this blog, we showed you a detailed procedure on how to build an Amazon EKS AMI based on the AWS Market place AMI for CIS Benchmark Amazon Linux 2 profile and also on how to leverage the custom AMI with EKS Managed node groups. These instructions remain same for both CIS Benchmark Amazon Linux 2 Level 1 as well.

Check out our Containers Roadmap!

If you have ideas about how we can improve Amazon container services, please use our Containers Roadmap to provide feedback and review our existing roadmap items.

TAGS: , ,
Jayaprakash Alawala

Jayaprakash Alawala

Jayaprakash Alawala is a Sr Container Specialist Solutions Architect at AWS. He helps customers on Applications Modernization and build large scale applications leveraging various AWS services. He has expertise in the area of Containers, Micro-services, Dev Ops, Security, Cost Optimization including EC2 Spot, Technical Training. Outside of work, he loves spending time reading and traveling. You can reach him on twitter @JP_Alawala

Paavan Mistry

Paavan Mistry

Paavan is a Senior Manager, Security and Compliance - Compute Services at AWS. Paavan has previously led teams of Developer Advocates, Solutions Architects, and Security Engineers at AWS. You can reach him on Twitter as @98pm and GitHub as @paavan98pm.