Run Amazon EKS on RHEL Worker Nodes with IPVS Networking
Amazon Elastic Kubernetes Services (Amazon EKS) provides excellent abstraction from managing the Kubernetes control plane and data plane nodes that are responsible for operating and managing a cluster. AWS offers managed Amazon Machine Images, or AMIs, for Amazon Linux 2, Bottlerocket, and Windows Server. Many customers have requirements, or simply prefer, to use Red Hat Enterprise Linux (RHEL) for all their Linux machines. Although RHEL isn’t officially supported for Amazon EKS, customers can build Amazon EKS worker nodes on RHEL using the scripts in this post.
An Amazon EKS cluster has built-in add-ons for networking: The Amazon Virtual Private Cloud (Amazon VPC) container network interface (CNI) plugin for Kubernetes, CoreDNS and kube-proxy components. Each of these components plays a vital role for Kubernetes networking. Of these components, kube-proxy is responsible for maintaining the network rules that allow network communication to your Pods from network sessions inside or outside of your cluster. IPVS and iptables are two proxy modes for kube-proxy on Linux, with the default being iptables. While it is most common and acceptable to use the default mode, there are known limitations on iptables that can ultimately impacts a cluster’s performance.
With RHEL 8.6, Red Hat transitioned from iptables to nftables for network filtering. This presents a challenge when running Kubernetes components such as the kube-proxy daemonset, which at the time of writing doesn’t support nftables on RHEL. One work around for this challenge is to use IP Virtual Server (IPVS).
IPVS can enhance the performance of a Kubernetes cluster as it was created for load balancing. It uses hash tables for efficient data structuring, making it more suitable for use in larger clusters. To enable IPVS, you must configure this at the Kubernetes cluster level as well as on the individual worker nodes. In this post, we’ll show you how to configure your Amazon EKS cluster for IPVS networking. We’ll also demonstrate how to build RHEL 8.x or RHEL 9.x worker nodes and join them to your cluster.
Note: There is a network routing issue caused by the nm-cloud-setup service that comes preinstalled on RHEL machines. In this guide, we will disable this service and reboot the EC2 instances as recommended by Red Hat in the following KB article.
This walkthrough requires the following prerequisites (versions of each are listed as of the time of writing).
- AWS Command Line Interface (AWS CLI) (walkthrough version 2.13.21)
- eksctl (walkthrough version 0.160.0)
- kubectl (walkthrough version 1.28.1)
- git (walkthrough version 2.40.1)
- HashiCorp Packer (walkthrough version 1.9.4)
First, we’ll create an Amazon EKS cluster without any workers nodes and configure IPVS networking. Next, we’ll build out the Amazon EKS worker nodes on RHEL. And finally, we’ll join the Amazon EKS worker nodes to the cluster.
Amazon EKS cluster creation and configuration
Create an Amazon EKS cluster without any nodes.
eksctl create cluster --name <CLUSTER_NAME> --without-nodegroup --version=<VERSION>
Edit the kube-proxy-config ConfigMap to use IPVS. This can be done using the Amazon EKS VPC CNI add-on using instructions from this post.
kubectl -n kube-system edit cm kube-proxy-config
Set the ipvs → scheduler value to your preferred method. Example: rr
- rr: round-robin
- lc: least connection
- dh: destination hashing
- sh: source hashing
- sed: shortest expected delay
- nq: never queue
Set the mode to “ipvs”. Save and exit.
hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "rr" syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 0.0.0.0:10249 mode: "ipvs" nodePortAddresses: null oomScoreAdj: -998 portRange: "" udpIdleTimeout: 250ms
With IPVS enabled at the cluster level, your worker nodes need to have the proper kernel modules installed and loaded. This is taken care of by the installation script referenced in the next portion of the walkthrough, but you can enable these modules manually if you don’t want to run the installation script.
# Install ipvsadm $ sudo yum install -y ipvsadm # Verify no entries are present $ sudo ipvsadm -L # Load kernel modules $ sudo modprobe ip_vs $ sudo modprobe ip_vs_rr $ sudo modprobe ip_vs_wrr $ sudo modprobe ip_vs_sh $ sudo modprobe nf_conntrack
Building the RHEL Worker Node AMI
We’ll use Packer to build the worker nodes. Packer can be setup on your local workstation with Application Programming Interface (API) keys for AWS credentials. Packer can also be setup on an Amazon Elastic Compute Cloud (Amazon EC2) instance with an AWS Identity and Access Management (AWS IAM) Instance Profile with the necessary permissions, or it can be setup in AWS CloudShell. We recommend using AWS CloudShell because it is free to use and is preconfigured with the same credentials of the user logged into the AWS Management Console.
To setup Packer in AWS CloudShell, you can run the following commands.
sudo yum install -y yum-utils shadow-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo sudo yum -y install packer
Next, clone the GitHub repository for building Amazon EKS RHEL AMIs and cd into the directory.
git clone https://github.com/aws-samples/amazon-eks-ami-rhel.git cd amazon-eks-ami-rhel/
Launch the Packer build process using a Make command similar to one of the ones below, specifying your Kubernetes cluster version as well as any other custom values you desire.
Example basic command:
Example command for building a customized Defense Information Systems Agency (DISA) Security Technical Implementation Guide (STIG) compliant AMI, owned by a specific AWS Account in AWS GovCloud us-gov-east-1 Region, with binaries stored in a private Amazon Simple Storage Service (Amazon S3) bucket, an AWS IAM instance profile attached, and using AWS Systems Manager Session Manager for Packer terminal access:
make 1.28 source_ami_owners=123456789012 source_ami_filter_name=RHEL9_STIG_BASE*2023-04-14* ami_regions=us-gov-east-1 aws_region=us-gov-east-1 binary_bucket_name=my-eks-bucket binary_bucket_region=us-gov-east-1 iam_role=EC2Role pull_cni_from_github=false ssh_interface=session_manager
The build process can take anywhere from 5 to 20 minutes and should provide an output similar to the one below at the end. Copy down the resulting AMI ID for use in the next steps.
Build 'amazon-ebs' finished after 15 minutes 25 seconds. ==> Wait completed after 15 minutes 25 seconds ==> Builds finished. The artifacts of successful builds are: --> amazon-ebs: AMIs were created: us-gov-east-1: ami-0c6d588be9ce412ef
Now that you have successfully built a custom AMI, it’s time to join worker nodes to your cluster. In the root of the directory of the GitHub repository cloned at the beginning of the walkthrough, there are bash (Linux) and zsh (MacOS) scripts that you can execute to join nodes to your cluster. These scripts use eksctl to create Amazon EKS-managed node groups. Modify these scripts with any custom tags or labels as desired.
Note: These scripts contain custom Amazon EC2 User Data scripts that disable the nm-cloud-setup service to resolve the issue with this service mentioned earlier in the post.
./create_nodegroup.sh <CLUSTER_NAME> <AMI_ID> <NODE_GROUP_NAME> <REGION> <KEY_PAIR> <INSTANCE_TYPE> <MIN_GROUP_SIZE> <DESIRED_GROUP_SIZE> <MAX_GROUP_SIZE>
./create_nodegroup.sh rhel-cluster ami-0c6d588be9ce412ef rhelnodegroup us-gov-east-1 govcloudkeypair t3.large 5 5 5
At this point, we want to perform some validation of our Amazon EKS cluster. First, we’ll verify our worker nodes have joined the cluster.
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-192-168-138-204.us-gov-east-1.compute.internal Ready <none> 46m v1.28.1-eks-43840fb ip-192-168-168-184.us-gov-east-1.compute.internal Ready <none> 46m v1.28.1-eks-43840fb
Next, we’ll verify kube-system namespace pods are running.
$ kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE aws-node-qqw24 1/1 Running 0 47m aws-node-xpbzl 1/1 Running 0 47m coredns-76bcb79755-6vfsd 1/1 Running 0 47m coredns-76bcb79755-zvbvq 1/1 Running 0 47m kube-proxy-454wj 1/1 Running 0 47m kube-proxy-tmgln 1/1 Running 0 47m
Note: We want to confirm that coredns is working as expected as this is one of the main issues seen when moving from iptables to nftables or IPVS.
Lastly, we’ll verify IPVS entries are present on one of our worker nodes.
$ sudo ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP ip-10-100-0-1.us-gov-east-1. rr -> ip-192-168-113-81.us-gov-eas Masq 1 0 0 -> ip-192-168-162-166.us-gov-ea Masq 1 1 0 TCP ip-10-100-0-10.us-gov-east-1 rr -> ip-192-168-104-215.us-gov-ea Masq 1 0 0 -> ip-192-168-123-227.us-gov-ea Masq 1 0 0 UDP ip-10-100-0-10.us-gov-east-1 rr -> ip-192-168-104-215.us-gov-ea Masq 1 0 0 -> ip-192-168-123-227.us-gov-ea Masq 1 0 0
Note that the first entry is the Transmission Control Protocol (TCP) entry for the Kubernetes ClusterIP service (from
kubectl get service kubernetes).
The second and third entries are the TCP and User Datagram Protocol (UDP) entries for the kube-dns service (from
kubectl get service kube-dns -n kube-system). The endpoints are the coredns pod IPs.
To avoid incurring future charges, delete the two AWS CloudFormation Stacks created in the steps above. You’ll need to delete the Managed Node Group stack first, and then you can delete the Amazon EKS cluster stack.
In this post, we showed you how to create Amazon EKS worker nodes that run on RHEL 8 and RHEL 9 Amazon EC2 instances and how to run your Amazon EKS clusters in IPVS networking mode.