Under the hood: how Amazon EKS Auto Mode detects, repairs, and diagnoses node failures

A GPU drops off the PCIe bus, a network interface goes dark, or the container runtime wedges. On most Kubernetes clusters, that is the start of a pager story: someone wakes up, reads dashboards, SSHes in, cordons, drains, and terminates the instance by hand. With Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, the failing node is detected, drained, and replaced automatically before anyone reaches for a laptop.

Two components work together to make this happen. The Amazon EKS Node Monitoring Agent (NMA) detects the fault and records it as a Kubernetes node condition. Karpenter reads that condition and replaces the node. On EKS Auto Mode, both run automatically with no configuration. On other EKS compute (managed node groups or self-managed Karpenter), you can install the Node Monitoring Agent as an EKS add-on to get the same detection signals. This gives you the same repair behavior. The difference on Auto Mode is that the agent runs as a systemd service baked into the AMI rather than a DaemonSet. That means it keeps running even when pod scheduling is degraded and cannot be accidentally evicted or misconfigured.

In this post, we show how the repair cycle works step by step and why specific faults trigger node replacement while others stay informational. We also explain how the safety thresholds prevent cascading failures and how you collect diagnostics from any node without ever logging in.

How detection and repair work together

The cycle has two halves, and a clean contract between them.

Detection: the Node Monitoring Agent. The agent watches the kernel, the container runtime, networking, storage, and accelerated hardware. It translates messy, low-level signals into a small set of Kubernetes node conditions. When a fault is terminal enough to warrant replacement, the agent flips the matching condition to `False`:

status:
  conditions:
  - type: AcceleratedHardwareReady
    status: "False"
    reason: NvidiaDeviceCountMismatch
    message: "Expected 8 GPUs, found 7."

Remediation: Karpenter. Karpenter is the managed compute controller that already provisions and scales all EKS Auto Mode nodes. It consumes those conditions directly. There is no separate repair backend to switch on and no per-node-group opt-in. The same controller that launched the node is the one that replaces it. Its cloud provider declares repair policies. Each policy pairs a condition type with a status that means “replace this node.”

Replacement is Karpenter’s only repair action. This differs from managed node group auto repair, which chooses between rebooting and replacing an instance based on the condition. On EKS Auto Mode, an unhealthy node is always replaced with a fresh one. When a node trips a policy and stays tripped past the grace period, Karpenter acts. It taints the node, drains it, terminates the instance, and launches a replacement.

The node condition is the entire interface between the two halves. The agent does not call Karpenter, and Karpenter does not know how the agent reached its verdict. The agent writes a Kubernetes construct, Karpenter acts on it, and the Kubernetes API server is the only thing they share. That decoupling is what makes the cycle reliable.

What the agent detects

The agent groups its detections under five monitoring conditions. These are `KernelReady`, `ContainerRuntimeReady`, `NetworkingReady`, `StorageReady`, and `AcceleratedHardwareReady`. Every detection carries one of two severities. A Condition-severity detection is a terminal fault: it flips the matching condition to False and makes the node eligible for replacement. An Event-severity detection is a transient problem or a sub-optimal setting, surfaced as a Kubernetes event for visibility while the node stays in service. Severity is the switch that decides whether repair fires.

The faults that trigger repair are the ones you cannot ride out. On accelerated instances that means GPU device-count mismatches, critical XID errors, and double-bit ECC errors. It also covers NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA, HBM, and SRAM uncorrectable errors. These are hardware faults that will not recover on their own and waste GPU-hours on every training step while the node stays active. On the network side it means the Amazon Virtual Private Cloud (Amazon VPC) Container Networking Interface (CNI) process down or IPAMD unable to reach the API server. It also covers an interface that will not come up or a missing loopback. The kernel and runtime add their own terminal cases: a fork failure from PID or memory exhaustion, and pods stuck in the Terminating state behind a broken container runtime.

The agent also raises Event-severity detections. These, by contrast, stay informational: the agent posts a Kubernetes event and the node stays in service. These cover bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, and I/O delays. They also surface filesystem fragmentation, clock drift, probe failures, kube-proxy anomalies, and GPU thermal and power warnings. They give you a read on trouble building before it turns terminal. The full catalog of reason codes and their severities lives in the node health documentation. NVIDIA coverage is extensive, powered by DCGM. DCGM is built on NVML and adds push-based policy events, diagnostics, and NVSwitch fabric health. On multi-GPU instances like P5e and P6, a single bad device can stall an entire training job, making this coverage critical.

From a False condition to a fresh node

The following section explains how detection translates into node replacement. Karpenter’s AWS cloud provider declares a set of repair policies. Each policy pairs a condition type with the status that counts as broken and a toleration window the condition must stay broken before repair begins.

The flow, step by step:

1. The agent flips a condition. A `Condition`-severity detection sets the matching node condition to `False`, with a reason and a human-readable message. `Event`-severity detections never reach this step. They post a Kubernetes event and the node stays `True`.
2. Karpenter’s health controller clocks it. Karpenter watches node conditions and starts a timer from the condition’s `LastTransitionTime`. A blip that clears on its own before the toleration window expires costs nothing. The timer resets and the node is never touched.
3. The toleration window decides the speed. Accelerated hardware faults wait 10 minutes. Kernel, runtime, networking, storage, and the kubelet-reported NodeReady wait 30.
4. A safety gate guards the fleet. Karpenter will not repair past a threshold of unhealthy nodes in a NodePool or cluster (20 percent). Auto repair also stands down while an Amazon Application Recovery Controller (ARC) zonal shift is active. The intent is to fix the one bad node, never to amplify a widespread problem.
5. Repair runs. Past the window and inside the safety threshold, Karpenter taints the node to stop new scheduling and drains the running pods, respecting Pod Disruption Budgets. It then terminates the instance and provisions a replacement sized for the displaced pods. For stateful workloads like distributed training, successful recovery depends on the workload’s ability to restart from a checkpoint. The system replaces the node and reschedules the pod, but resuming from the last saved state is the workload’s responsibility.

The whole sequence runs without a human in it. The replacement node comes up registered, monitored by a fresh agent, and ready for work.

Lessons from operating at scale

Running the Node Monitoring Agent across thousands of clusters taught us design principles worth calling out.

Why specific faults are terminal or informational. A GPU that drops off the PCIe bus will not recover without a reboot. Leaving it as a Warning meant workloads kept scheduling onto a degraded node. A bandwidth ceiling, by contrast, is transient and workload dependent. The classification rule: if the fault persists regardless of what pods are running and cannot self-heal, it is Condition-severity and triggers replacement. If it depends on workload behavior or clears on its own, it is Event-severity and stays informational.

Why the 20 percent safety threshold. Without a fleet-wide cap, a correlated false positive could flag dozens of healthy nodes simultaneously. A bad AMI, a control-plane hiccup, or a misconfigured monitor could trigger mass replacement. The 20 percent cap means Karpenter will not terminate more than one-fifth of a NodePool at once, preserving the majority of capacity while still repairing genuine faults. The threshold is fixed in Karpenter’s health controller. It is not configurable through NodePool settings. When the cap blocks a repair, Karpenter publishes a NodeRepairBlocked event on the node and rechecks every five minutes.

Why toleration windows differ. Accelerated hardware faults get 10 minutes because they are unambiguous (a GPU is either present or absent on the bus) and expensive to ignore. Kernel, runtime, networking, and storage faults get 30 minutes because transient conditions in these subsystems can self-resolve. A brief network blip or a temporary runtime restart should not cost a node.

Reason codes are a public API. Every downstream consumer (repair controllers, dashboards, customer automation) keys on reason strings by literal match. In v1.6.2, we changed `NvidiaDeviceCountMismatch` from Warning to Fatal severity. The technical reasoning was sound, but customers had repair configs keyed on the old severity. Their automation broke. From that point, we treat reason code additions as feature work and severity changes as breaking changes.

“Absent” must not equal “healthy.” When a monitor is disabled, the agent emits no condition at all rather than writing `True` or `Unknown`. Writing `True` would tell auto-repair the node is healthy when the agent is not watching. Writing `Unknown` is ambiguous and might trigger unnecessary repair. Emitting nothing is the only safe choice.

Seeing the repair cycle in action

The following example shows the repair cycle running end to end on a live Amazon EKS Auto Mode cluster.

The cluster has a GPU node, a `g6e.xlarge` on the Bottlerocket EKS Auto Nvidia variant. With the GPU healthy, NMA reports every node condition green:

$ kubectl get node i-0981...e7d \
    -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'
AcceleratedHardwareReady=True (NvidiaGPUIsReady)
ContainerRuntimeReady=True (ContainerRuntimeIsReady)
NetworkingReady=True (NetworkingIsReady)
KernelReady=True (KernelIsReady)
StorageReady=True (DiskIsReady)

To simulate a real GPU fault without waiting for silicon to fail, we inject a double-bit ECC error into the node’s DCGM host engine. This is the same telemetry source NMA reads. EKS Auto Mode nodes are managed instances with no SSH access. To inject the fault, we run a privileged pod that carries the DCGM tooling:

apiVersion: v1
kind: Pod
metadata:
  name: dcgm-inject
spec:
  nodeName: i-0981...e7d
  hostPID: true
  containers:
  - name: dcgm
    image: nvcr.io/nvidia/cloud-native/dcgm:3.3.7-1-ubuntu22.04
    command: ["sleep", "3600"]
    securityContext:
      privileged: true
    volumeMounts:
    - name: dev
      mountPath: /dev
  volumes:
  - name: dev
    hostPath:
      path: /dev
  tolerations:
  - operator: Exists

The pod needs `privileged: true` and `hostPID: true` so it can communicate with the host’s DCGM engine. Once it is running, we exec in and inject:

$ kubectl exec dcgm-inject -- dcgmi test --inject --gpuid 0 -f 319 -v 4
Successfully injected field info.

NMA picks it up on its next poll, seconds later, and flips the condition to `False`:

$ kubectl get node i-0981...e7d -o jsonpath='{range .status.conditions[?(@.type=="AcceleratedHardwareReady")]}{.status} {.reason}: {.message}{"\n"}{end}'
False NvidiaDoubleBitError: detected 4 Nvidia Double Bit error(s) on location Device

The agent records the flip on the node as an event. This is the signal Karpenter watches:

$ kubectl get events --field-selector involvedObject.name=i-0981...e7d
... AcceleratedHardwareReady  Status condition transitioned, Type: AcceleratedHardwareReady,
    Status: True -> False, Reason: NvidiaDoubleBitError,
    Message: detected 4 Nvidia Double Bit error(s) on location Device

From here no one touches anything. Ten minutes after the condition went `False` (the toleration window for accelerated hardware), Karpenter starts the repair on its own. It launches a replacement while draining the unhealthy node:

$ kubectl get nodeclaims
NAME             TYPE          ZONE         NODE           READY
gpu-demo-fdthm   g6e.xlarge   us-west-2c                  Unknown   # replacement, launching
gpu-demo-xfqcb   g6e.xlarge   us-west-2b   i-0981...e7d     True      # unhealthy, draining

A couple of minutes later the unhealthy node is gone, a fresh node has taken its place, and the workload is running again:

$ kubectl get node i-0bb2...fb4 -o jsonpath='{range .status.conditions[?(@.type=="AcceleratedHardwareReady")]}{.status} {.reason}{"\n"}{end}'
True NvidiaGPUIsReady
$ kubectl get pods -l app=gpu-demo -o wide
NAME                          READY   STATUS    NODE
gpu-demo-workload-...-j97n5     1/1     Running   i-0bb2...fb4

One injected fault, a condition flip in seconds, ten minutes of deliberate toleration, and roughly 90 seconds to reprovision and rejoin. Start to finish, hands-off.

Clean up resources

After running the walkthrough, remove the test resources:

kubectl delete pod dcgm-inject

Diagnostics without SSH: Understanding why a node failed

Auto-repair handles the common case: unhealthy node gets replaced, workloads keep running. But sometimes you need to understand why a node failed. Maybe it is a recurring pattern. Maybe you need evidence for a support case.

On traditional EKS nodes, collecting diagnostics meant SSH access to the node and a manual log-collection script. The NodeDiagnostic CRD provides an alternative. The previous approach requires SSH access that compliance often forbids, and it cannot run if networking failed. On EKS Auto Mode it does not work at all. Auto Mode nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no SSH access by design. The NodeDiagnostic CRD and `kubectl ekslogs` are your only window into what happened on these nodes.

How kubectl ekslogs works

The kubectl-ekslogs plugin wraps the NodeDiagnostic workflow into a single command:

$ kubectl ekslogs i-0abc123def456
✔ Transfer mode: node proxy API
⟳ Validating node(s)...
✔ 1 node validated
⟳ Creating NodeDiagnostic resources...
⟳ Waiting for log collection to complete (timeout: 300s)...
✔ Log collection completed on 1 node
⟳ Downloading log bundles...
✔ Saved: ./i-0abc123def456-logs.tar.gz (288K)
✔ Done: 1 log bundle downloaded to ./

Under the hood, the plugin:

1. Creates a `NodeDiagnostic` custom resource with the node name and `destination: node`.
2. The agent on the target node detects the resource through a Kubernetes watch.
3. The agent collects system logs, kernel messages, networking state, and runtime diagnostics into a compressed tarball.
4. The agent stores the tarball temporarily on the node (available for 10 minutes).
5. The plugin downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36).

SSH is not required. Security group changes are not required. Key pairs are not required. Works on Auto Mode managed instances where you have no shell access.

If you need logs persisted to Amazon Simple Storage Service (Amazon S3) rather than streamed through the kubelet API, set the NodeDiagnostic destination to a pre-signed S3 PUT URL. The Retrieve node logs documentation walks through that flow step by step. You can also capture network traffic through the same mechanism: the agent runs `tcpdump` on the node, compresses the capture files, and uploads them to your bucket.

Detection and diagnosis are separate concerns

The architecture keeps these apart by design. Detection answers “is this node healthy?” and runs continuously with minimal overhead, writing NodeConditions. Diagnosis answers “what went wrong?” and runs on demand, collecting detailed artifacts through NodeDiagnostic. The two share an agent binary but serve different consumers. Detection feeds the repair cycle. Diagnosis feeds humans investigating after the fact.

This separation means detection does not slow down to collect evidence and diagnosis does not need to be always-on. You can diagnose a node that auto-repair already flagged but has not yet terminated.

What you get on EKS Auto Mode

Node health monitoring is one of several capabilities baked into the Auto Mode AMI. The Auto Mode architecture post covers the broader picture. Everything described in this post is active by default. The only optional piece is the diagnostics tooling on your workstation.

For on-demand diagnostics, install the kubectl-ekslogs plugin and run kubectl ekslogs <node-name> to retrieve a full log bundle. For network captures, create a NodeDiagnostic resource with a tcpdump destination.

On other EKS compute you can assemble this yourself: install the Node Monitoring Agent as an EKS add-on, keep its version current, and opt each node group into repair. The node monitoring and auto repair documentation covers the install and opt-in steps.

Conclusion

In this post, we showed how Amazon EKS Auto Mode detects, repairs, and diagnoses node failures without a human in the path. The Node Monitoring Agent turns low-level faults into Kubernetes node conditions, and Karpenter turns those conditions into replacements. The node condition is the entire contract between the two, and it is nothing more than a plain Kubernetes construct. That is what makes the cycle dependable. Severity decides what fires: Condition-severity detections trigger replacement, while Event-severity detections stay informational and leave the node in service.

The system is deliberately measured rather than reactive. Toleration windows ride out transient faults, a fleet-wide safety threshold keeps a correlated failure from being amplified, and reason codes are treated as a public API so the automation you build on them doesn’t break underneath you. And when you need to understand why a node failed, the NodeDiagnostic CRD and kubectl ekslogs retrieve full log bundles through the Kubernetes API, with no SSH required, even on managed instances you can’t log into.

On EKS Auto Mode, none of this is a feature you turn on. There’s no add-on, no opt-in flag, and no version to manage: detection, repair, and diagnostics are part of the data-plane lifecycle AWS runs for you, active from the moment the cluster exists.

EKS Auto Mode handles the compute so you can spend your time on the workloads. And because the Node Monitoring Agent is open source, the failure modes you hit at scale feed directly into new detections through pull requests.

Learn more

• Under the hood: Amazon EKS Auto Mode (architecture overview).
• Amazon EKS introduces node monitoring and auto-repair (launch announcement).
• Node health and auto repair in the Amazon EKS User Guide.
• Retrieve node logs with NodeDiagnostic.
• Capture network traffic on a managed node.
• kubectl-ekslogs plugin.
• EKS Node Monitoring Agent (GitHub).
• Karpenter provider-aws (GitHub).

About the authors