Karpenter now supports Windows containers
In November 2021, AWS introduced Karpenter, an open-source high-performance Kubernetes Cluster Autoscaler licensed under the Apache License 2.0. Karpenter helps improve your application availability and cluster efficiency by rapidly launching right-sized compute resources in response to changing application load. Since its release, we’ve been seeing an increase in customers migrating from Kubernetes Cluster Autoscaler to Karpenter. However, for customers running a heterogeneous Amazon Elastic Kubernetes Service (Amazon EKS) cluster with Windows workloads, it became a showstopper as Karpenter didn’t support Windows nodes, until now.
The OSS community did a great job starting development on Windows workloads in Karpenter. The AWS team took it a step further to review the proposed design, add enhancements to improve the customer experience, and integrate it with our internal continuous integration (CI) process.
When Karpenter is installed in your cluster, it observes the aggregate resource requests of unscheduled pods and decides to launch new nodes when additional capacity is needed, while deciding to deprovision nodes when that capacity is no longer needed. By doing this, Karpenter reduces the scheduling latencies and infrastructure costs of your cluster.
Figure 1: Karpenter high-level scheduling
In this post, we focus on scaling out/in Windows Server 2019 and Windows Server 2022 using Karpenter for Amazon EKS. To learn more about Karpenter architecture and components, access the Karpenter website.
- Ensure you are running eksctl commands with an AWS Identify and Access Management (AWS IAM) profile that has permissions to create and manage Amazon EKS. This AWS IAM security principal is used in the Getting Started section below for the AWS Command Line Interface (AWS CLI) configuration.
- Ensure you are using eksctl v0.124.0 or higher to operate Karpenter.
- Follow the Getting Started section in the Amazon EKS documentation to install aws cli, kubectl, and eksctl on your development machine.
- Alternatively, you could leverage Cloud9 or Cloudshell to handle deployment and maintenance tasks.
- Create OS variables to be used throughout the post.
- Deploy Karpenter service requirements.
- Create an Amazon EKS cluster with the necessary iamIdentityMappings for Karpenter.
- Enable Amazon EKS Windows support.
- Install Karpenter with Helm.
- Create Karpenter provisioner and NodeTemplate.
- Test Karpenter for Windows – scale out.
- Test Karpenter for Windows – scale in.
- Cleanup test resources.
1. Create OS variables to be used throughout the post
2. Create Karpenter service requirements
Karpenter directly integrates with the Amazon Elastic Compute Cloud (Amazon EC2) API endpoint to take specific actions based on events such as spot interruption or instance state changes. The following command automatically deploys the necessary AWS services/components such as Amazon EventBridge rules applied to messages being sent over an Amazon SQS queue using AWS CloudFormation.
Upon successful execution of the AWS CloudFormation template, you’ll be presented with the following output:
3. Create an Amazon EKS cluster with the necessary iamIdentityMappings for Karpenter
Next, we deploy a temporary Amazon EKS cluster using eksctl in order to test Karpenter integration with Windows. The necessary AWS IAM and IdentityMapping are created as ServiceAccounts and added to the Kubernetes ConfigMap.
eksctl uses AWS CloudFormation to create all the necessary resources to build an Amazon EKS cluster. Upon successful creation of your cluster, you’ll be presented with a similar output. If the cluster creation fails, then the failure reason is provided in the AWS CLI output (or AWS CloudFormation console).
4. Enable Amazon EKS Windows support
To deploy Windows nodes to our cluster, we need to enable Amazon EKS Windows support.
5. Install Karpenter with Helm
Next, we will use Helm to install Karpenter.
Upon successful installation, you‘ll see the following output.
6. Create provisioner as required
Now we create two Karpenter provisioners to support Windows Server 2019 and Windows Server 2022 in the same Amazon EKS cluster. The Karpenter provisioner sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes.
7. Scale out the deployment
We now have our Amazon EKS cluster prepped for running Windows nodes and all the necessary components of Karpenter. We scale a sample application to see Karpenter automatically add nodes to the Amazon EKS cluster based on demand.
7.1 Run the following code to create your Windows Server 2022 sample application.
The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelector fields. Kubernetes automatically adds a label to the Windows node, named node.kubernetes.io/windows-build to simplify this.
This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used for each Windows Server version:
|1||Windows Server 2019||10.0.17763|
|2||Windows Server 2022||10.0.20348|
Based on the build version specified in the Pod nodeSelector, Karpenter launches new Windows nodes with the operating system accordingly. For example, if the build version is specified as 10.0.17763, then Karpenter uses the Windows 2019 provisioner to launch Windows nodes. For more information, please refer to the Guide for Running Windows Containers in Kubernetes.
7.2 Run the following command to scale your Windows Server 2022 sample application.
7.3 You can use the Karpenter logs to track the scaling progress.
The following output shows the Windows Server 2022 Karpenter provisioner scaling from 0 nodes to 1 to support the 10 replicas we requested be run.
7.4 Run the following command to track the deployment progress of your pods.
You’ll see the 10 replicas being created on our new Karpenter provisioned Windows worker nodes.
7.5 Run the following code to scale out your Windows Server 2019 deployment.
7.6 Run the following command to scale your Windows Server 2019 sample application.
A new Windows Server 2019 worker is launched by Karpenter as more pods are requested to be scheduled. This process is identical to Windows Server 2022 and you can reuse the steps above to track the progress of launching Windows Server 2019 worker node.
8. Scale in the deployment
Karpenter handles scale out and scale in of Windows nodes based on demand. We’ll now tear down our sample applications and watch Karpenter terminate our Windows nodes.
8.1 Run the following commands to delete your sample application deployments.
The Windows instances launched earlier by Karpenter will now be terminated. You can use the Karpenter logs to track the scale down progress.
Once all pods have been terminated, Karpenter deletes all idle instances.
When you’ve finished, clean up the resources associated with the example cluster deployment to avoid incurring unwanted charges.
If this command times out, then you can run the command above again to show the cluster has been successfully removed.
In this post, we showed you can leverage Karpenter to seamlessly scale out/in your Windows worker nodes on Amazon EKS. Customers no longer need to maintain two auto-scaler solutions on a heterogeneous Amazon EKS cluster with Windows and Linux nodes.
A big shout-out to topikachu, who proactively started the development of the add-on.