Containers

Karpenter v1.15: GPU-aware provisioning, smarter node repair, and more

Kubernetes runs in production at over 80% of organizations that use containers, according to the Cloud Native Computing Foundation (CNCF) 2025 Annual Cloud Native Survey. But the workloads have changed. Distributed training jobs, stateful databases, and mixed-instance fleets now require GPU topology awareness, high-speed networking, and persistent storage. These requirements sit below the abstraction layer that Kubernetes resource requests were designed to express. When provisioning cannot account for the physical hardware layer, the result is unnecessary node launches, scheduling failures, and underutilized capacity. The community has responded. According to Datadog’s State of Containers and Serverless report, Karpenter adoption surged 22% in two years, reflecting growing demand for hardware-aware node provisioning.

AWS created Karpenter in 2021 and open sourced it as a Kubernetes node lifecycle manager that provisions right-sized compute in real time. In 2023, AWS contributed the vendor-neutral core to the Kubernetes Special Interest Group (SIG) Autoscaling under the CNCF. Karpenter reached general availability (GA) with v1.0 in August 2024, establishing the first stable release.

Building on that foundation, v1.15 addresses three areas. Provisioning now understands the physical hardware layer before a node launches. Repair coordinates with your existing availability controls instead of overriding them. Consolidation aligns with your own scaling decisions. The following sections describe what changed, why it matters, and how to activate each capability.

Throughout this post, version references use “1.15” rather than a specific patch. Always run the latest patch release for your version.

1.0 Provisioning that understands the hardware

Three enhancements give Karpenter visibility into the physical hardware layer before it launches a node.

NVIDIA GPU and EFA DRA support

GPU and Elastic Fabric Adapter (EFA) workloads have a placement requirement that ordinary resource counting cannot capture: The devices must share the same Peripheral Component Interconnect Express (PCIe) root complex. When an NVIDIA GPU and an EFA adapter share a PCIe root complex, they transfer data directly over the PCIe fabric without routing through the CPU and system memory. This peer-to-peer path reduces latency and frees memory bandwidth for collective operations such as AllReduce that underpin large-scale distributed training.

Karpenter 1.15 adds Dynamic Resource Allocation (DRA) support for the AWS provider, accounting for NVIDIA GPUs with system-mediated time-slicing and EFA networking through DRANET when sizing new nodes. Without DRA, Karpenter and the Kubernetes scheduler can only reason about resources based on generic counts, with no ability to specify constraints between devices.With DRA support, Karpenter evaluates device availability and topology as part of the provisioning decision, before the node is requested.

This DRA path also establishes the infrastructure for GPU time-slicing scenarios, where multiple pods share a single physical GPU by dividing its execution time. Time-slicing requests flow through the same DRA path. Karpenter can then reason about how many virtual devices a node can serve, rather than treating the GPU as a single indivisible unit.

DRA support is available on EKS 1.34+ and time-slicing support is available on EKS 1.36+. To turn on DRA support, use the DRANVIDIAGPU and DRAEFA AWS feature gates, or set settings.enableDRA in Helm. To learn more, refer to Dynamic Resource Allocation.

For the full DRA data model, refer to the Kubernetes DRA documentation.

Volume-limit-aware provisioning

Workloads with many PersistentVolumeClaims (PVCs) previously triggered a provisioning loop: Karpenter would launch a node, discover it could not attach enough Amazon Elastic Block Store (Amazon EBS) volumes, and then launch another. This happened because Karpenter did not account for instance-level volume attach limits until after a node was running.

In 1.15, Karpenter checks those limits before requesting an instance. It knows that an m5.xlarge supports up to 25 attached Amazon EBS volumes and factors that into scheduling decisions from the start. Workloads with many persistent volumes land on appropriately sized nodes the first time.

Nitro Enclaves support on EC2NodeClass

With the new spec.enclaveOptions field on EC2NodeClass, you can run confidential computing workloads on Amazon Elastic Compute Cloud (Amazon EC2) instances using AWS Nitro Enclaves through Karpenter. Karpenter filters automatically for enclave-capable instance types and passes the enclave configuration through to the Amazon EC2 launch template.

Before this change, enclave workloads required a separate, static node group maintained outside Karpenter, which meant giving up dynamic scaling and cost optimization for that part of the fleet. Now those workloads benefit from the same provisioning and consolidation as everything else.

To learn more, see Nitro Enclaves.

2.0 Repair that respects your workloads

When a node becomes unhealthy, how Karpenter responds determines whether your workloads experience a planned, coordinated transition or an abrupt interruption. Karpenter 1.15 gives the repair system the same availability controls that govern intentional disruption and adds a targeted reboot path for faults that do not require a full replacement.

Node repair as voluntary disruption

In 1.15 is that repair is now classified as a voluntary disruption. This means it participates in the same budget and Pod Disruption Budget (PDB) machinery that governs intentional operations like drift and consolidation. With this change, the repair system now provides three key guarantees for workloads with strict availability requirements such as distributed training jobs, stateful services, or multi-replica applications with tight PDB windows:

  1. Karpenter honors NodePool disruption budgets when repairing nodes, surfacing a new Unhealthy reason in budget accounting.
  2. Karpenter respects Pod Disruption Budgets during the drain.
  3. Karpenter applies a per-condition termination grace period to bound how long each drain can take.

In earlier releases, a repair action could exceed your intended concurrency budget or drain pods faster than a PDB permitted. The voluntary disruption model closes that gap by routing repair through the same controls as every other planned operation.

To exclude a specific node from repair, annotate it with karpenter.sh/do-not-repair: "true".

To learn more, see Node Auto Repair.

Reboot as a repair action

Not every node fault warrants replacing the instance. Transient GPU XID errors, wedged kubelets, and hung container runtimes are often cleared by a reboot. Replacing the instance in those cases incurs unnecessary latency, risks losing a capacity reservation slot, and generates more churn than the situation requires.

Repair policies can now match on condition reason and select either RebootNode or ReplaceNode. On AWS, the Amazon Elastic Kubernetes Service (Amazon EKS) Node Monitoring Agent identifies some GPU XID errors as reboot-clearable. Those errors trigger an in-place ec2:RebootInstances call instead of a full instance replacement.

The node stays in the fleet and the capacity reservation slot is preserved. Recovery through a reboot is typically faster than launching and initializing a new instance, which can take several minutes depending on the Amazon Machine Image size and bootstrap scripts.

Repair coverage also now extends to nodes that register with the cluster but never complete initialization. Those nodes are now included in the repair lifecycle.

Repair is available in alpha and can be enabled with the RebootForRepair feature gate, or set settings.awsFeatureGates.rebootForRepair in helm.

Terminate-first disruption

When a NodePool is at its node limit or backed by a full Capacity Reservation, Karpenter cannot stage a replacement before removing the current node because there is no room. The TerminateFirstDrift and TerminateFirstRepair feature gates (alpha) activate a terminate-first path for those environments. Karpenter terminates the drifted or unhealthy node first, then launches the replacement.

A new metric surfaces the reason a replacement could not be staged, so you can distinguish intentional terminate-first behavior from unexpected capacity shortfalls.

To learn more, see Terminate First Disruption.

3.0 Consolidation and configuration

Karpenter’s consolidation logic and your workload scaling logic can pull in different directions. When they do, the result is nodes that remain occupied longer than intended or kubelet configurations that require more NodeClass resources to manage than necessary. Karpenter 1.15 introduces two features that align these systems.

Pod deletion cost management

Karpenter identifies which nodes to remove during consolidation, but the Kubernetes ReplicaSet controller independently decides which pods to terminate during a scale down. When those decisions conflict, the nodes Karpenter wants to consolidate remain occupied by pods that the ReplicaSet controller chose to keep. Karpenter 1.15 introduces a pod deletion cost management controller (feature gate: PodDeletionCostManagement) that resolves this conflict. Karpenter manages the controller.kubernetes.io/pod-deletion-cost annotation so that Deployment scale down preferentially removes pods from the nodes Karpenter has targeted for consolidation.

You can provide additional guidance using the new karpenter.sh/disruption-cost annotation to steer which pods are removed first. To learn more, see Pod Deletion Cost Management.

CEL expressions in kubelet configuration

Heterogeneous fleets typically require separate EC2NodeClass resources for each distinct kubelet configuration, because fields like maxPods, kubeReserved, and systemReserved vary by instance type. The NodeClassCEL AWS feature gate introduces Common Expression Language (CEL) support in kubelet configuration. You can compute those values dynamically per instance type within a single EC2NodeClass.

A single expression can scale reserved memory proportionally to instance size, eliminating the need to maintain parallel NodeClass definitions for each instance family.

Get started

To start using these capabilities on Amazon EKS, see Device management. Before upgrading, review the Upgrade Guide for breaking changes. Key changes in 1.15 include standardized lowercase values for the reason metric label and updated consolidation_type label behavior on queue_failures_total. Several 1.15 capabilities ship behind feature gates. Core gates are set with --feature-gates. AWS specific gates are set with --aws-feature-gates.

Feature Gate Purpose
Node Auto Repair (voluntary disruption) NodeRepair (core) Replace unhealthy nodes under NodePool Disruption Budgets and PDBs instead of forceful termination
Reboot as repair action RebootForRepair (AWS) Clear transient GPU XID faults with an in-place reboot instead of a full replacement. Requires NodeRepair also enabled and ec2:RebootInstances on the controller role
Terminate-first on drift TerminateFirstDrift (core) Allow drift replacement when a NodePool is at its node limit or a Capacity Reservation is full
Terminate-first on repair TerminateFirstRepair (core) Allow repair in the same no-headroom cases as above
Pod deletion cost management PodDeletionCostManagement (core) Steer Deployment scale-in to remove pods from nodes Karpenter wants to consolidate, so whole nodes empty
DRA (NVIDIA GPU + EFA) DRANVIDIAGPU, DRAEFA (AWS gates) Include DRA devices from the NVIDIA GPU and EFA (DRANET) drivers when sizing new nodes. Also needs IGNORE_DRA_REQUESTS=false (Helm settings.enableDRA handles both gates and that setting)
CEL in kubelet config NodeClassCEL (AWS) Compute maxPods, kubeReserved, systemReserved per instance type in one EC2NodeClass instead of maintaining parallel NodeClasses
Nitro Enclaves on EC2NodeClass Spec.enclaveOptions.enabled (no gate) Run confidential-computing workloads on Karpenter-managed nodes instead of static side group

Resources:


About the authors

Sajjan Gundapuneedi

Sajjan Gundapuneedi

Sajjan is a Senior Manager in AWS Kubernetes, where he leads the EKS data plane team. He is passionate about AI/ML infrastructure, and his teams build the EKS infrastructure for AI/ML workloads across training, fine-tuning, inference, and agents.

Trishi Mehta

Trishi Mehta

Trishi is a Senior Product Manager, Technical at AWS.

Jason Deal

Jason Deal

Jason is a Software Development Engineer at AWS.

Dharam Shah

Dharam Shah

Dharam is a Cloud Infrastructure Architect at AWS, based in Bangalore, India. With three years of experience at Amazon Web Services, he specializes in DevOps, Networking and Generative AI solutions, helping customers navigate their cloud transformation journeys. Beyond work, he enjoys playing badminton and listening to music.