Validating Amazon EKS optimized Bottlerocket AMI against the CIS Benchmark
As Kubernetes adoption grows, many organizations are choosing it as their platform to build and host their modern and secure applications. Security is one of the primary design criteria for many workloads, especially those dealing with sensitive data such as financial data processing. These workloads have a stringent requirement to adhere to various security and compliance controls.
Many Amazon Elastic Kubernetes Service (Amazon EKS) customers, especially enterprise customers from the Banking and Finance industry, are looking for guidance from AWS on hardening Amazon EKS.
The Center for Internet Security (CIS) Benchmarks are best practices for the secure configuration of a target system. They define various Benchmarks for the Kubernetes control plane and the data plane. For Amazon EKS clusters, it’s strongly recommended to follow the CIS Amazon EKS Benchmark. However, many organizations also need to harden the operating system on the worker nodes for security and compliance purposes. Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for containers. If an organization needs to ensure compliance, the organization must implement the CIS Benchmark for Bottlerocket. This post provides detailed, step-by-step instructions on how customers can bootstrap an Amazon EKS optimized Bottlerocket Amazon Machine Image (AMI) for the requirements of the CIS Bottlerocket Benchmarks. This post also illustrates how to continuously validate the worker nodes against the Benchmark after deployment to minimize the risk of security configuration drift.
Amazon EKS optimized Bottlerocket AMI hardening process
The CIS Bottlerocket Benchmark defines two profiles for hardening (i.e., Level 1 and Level 2):
- A Level 1 profile is intended to be practical and prudent, provide a clear security benefit, and not inhibit the utility of the technology beyond acceptable means.
- A Level 2 profile is intended for environments or use cases where security is paramount, acts as a defense in depth measure, and may negatively inhibit the utility or performance of the technology.
The following walkthrough is a solution for hardening and validating an Amazon EKS optimized Bottlerocket AMI against Level 2.
Amazon EKS optimized Bottlerocket AMI support for CIS Benchmark
The Amazon EKS optimized Bottlerocket AMI (as of this writing) supports 18 out of 28 Level 1 and 2 recommendations specified in the CIS Benchmark for Bottlerocket, without a need for any additional configuration effort. For the remaining 10 recommendations to adhere to Level 2, six recommendations can be addressed via a bootstrap container and four recommendations can be addressed via kernel sysctl configurations in the user data of the Amazon EKS worker nodes.
|Logging and auditing
The code used in this solution is available in GitHub. Please clone the repository to prepare for the walkthrough.
You’ll need the following to complete the tutorial:
Note: We have tested the CLI steps in this post on Amazon Linux 2.
You’ll also need to configure the following environment variables:
Building a bootstrap container image
Bootstrap containers on Bottlerocket are specialized host containers that run on a new node as it is launched. Bootstrap containers are used to configure host OS settings that are not available via the Bottlerocket API. For the CIS Benchmark for Bottlerocket, a bootstrap container is needed to update the iptables rules of the host to be in compliance with Level 2 of the CIS Benchmark. The bootstrap container’s Entrypoint executes the following bash script, which, aside from the rules allowing inbound kubelet traffic to the instance, is sourced directly from the Benchmark documentation.
For our walkthrough, we’ll use a script from the repository to create an Amazon Elastic Container Registry (Amazon ECR) repository to store the container image. You can safely ignore the error message An error occurred (RepositoryNotFoundException) output while running the script.
Next, let’s build the bootstrap container image and push it to the newly created Amazon ECR repository.
The output from the make command should look like the following code:
Create Amazon EKS cluster with Amazon EKS-managed node group with Bottlerocket AMI
With the bootstrap container created and ready for use in Amazon ECR, we can create an Amazon EKS cluster with a managed node group running Bottlerocket configured to CIS Bottlerocket Benchmark. We’ll use the following cat command to insert the environment variables defined earlier into the cluster.yaml file located in the root of the GitHub repository.
Bottlerocket configuration settings are passed through to the managed nodes through user data in TOML format as referenced above to include the setting referencing the bootstrap container we created. To provision the cluster, run the following:
The cluster provisioning takes 10 to 15 minutes — now would be a great time to grab a cup of coffee.
Once the cluster is created, ensure that kubectl is functional:
Once the cluster has been provisioned, you can verify the bootstrap container ran successfully on the Bottlerocket host. Since we configured our node group without SSH access, we’ll use AWS Systems Manager Session Manager to connect to the Bottlerocket control container, enter the admin container, and through the admin container obtain access to a host root shell:
Deploy a sample nginx pod on the Bottlerocket nodes
With a working cluster and managed node group, we can deploy a sample application to make sure everything is running properly. For this example, we’ll use a simple nginx deployment defined in the GitHub repository to deploy pods to the cluster. We can then verify the pods are running, and the nginx webserver started correctly:
The output looks like below:
Run the below command to exec into the pod and run a curl command:
Run the below command to access the logs from the pod and then Ctrl + C to exit from the container log output:
The output looks like the following:
Validating the Bottlerocket AMI against the CIS Benchmark
After deploying a managed node group that adheres to the CIS Benchmark, we can use the commands outlined in the benchmark to verify the configuration, as 26 of the 28 checks are able to be automatically audited. For this step, we created a container to execute a script to perform the validation and a Kubernetes pod configuration to deploy the container to the cluster. This deployment method is idempotent in nature, so it can be executed once to verify initial configuration and regularly afterwards to detect any configuration drift.
First, we’ll need to again create an Amazon ECR repository to house the validation image:
Second, we’ll build the validation container and put it in the newly created Amazon ECR reposition:
Third, we will use another cat command to insert the environment variables defined previously into the job-eks.yaml file located at the root of the GitHub repository. This file is used to deploy the Kubernetes batch job object which references the validation image onto the cluster:
Apply the batch job using kubectl and check if the pod completed the execution:
The output looks like the following:
Once the batch job has completed, we can view the pod logs to verify the CIS Bottlerocket Benchmark compliance status of the node:
The output should look like the following:
Use these commands to delete the resources created during this post:
To comply with regulatory requirements, many organizations need the ability to deploy infrastructure that implements the recommendations from the Center for Internet Security Benchmarks. In this post, we showed you how an organization can bootstrap the Amazon EKS optimized Bottlerocket AMI and validate it against the CIS Bottlerocket Benchmark v1.0.0. This post used Amazon EKS Managed Node Groups to bootstrap Bottlerocket nodes; however, you can use the solution proposed above with self-managed node groups as well. After deployment, the validating container described above can be used. We hope this guide simplifies adopting Bottlerocket.
Check out our Containers Roadmap!
If you have ideas about how we can improve Amazon container services, then please use our Containers Roadmap to provide us feedback and review our existing roadmap items.