Containers

ARC zonal shift support for EKS Auto Mode and Karpenter

When an Availability Zone (AZ) experiences a failure, workloads running across multiple AZs need a fast, automated way to shift traffic and capacity away from the impaired zone. The AWS provider for Karpenter recently added support for Zonal Shift and Zonal Autoshift, features of Amazon Application Recovery Controller (ARC) designed to help you recover from these types of failures. In this post, we walk through how zonal shift integrates with Amazon Elastic Kubernetes Service (Amazon EKS) and what happens when a shift is triggered. We also show how to enable it on both self-managed Karpenter and EKS Auto Mode (EKS Auto) clusters.

When Zonal Shift integration is enabled, it performs the following functions automatically: first, it cordons the worker nodes running in the affected zone, preventing new pods from being scheduled there. Second, it deregisters the IP addresses of pods running on worker nodes in the impaired zone from application and network load balancers. It also removes the endpoints of those pods from their corresponding endpoint slices. This effectively prevents those pods from receiving network traffic. Third, it temporarily prevents Managed Node Groups (MNG), EKS Auto, and self-managed Karpenter from provisioning new capacity in the impaired zone. When using EKS Auto or self-managing Karpenter, voluntary disruptions (consolidation, drift, empty, and underutilized) are also temporarily suspended in the affected zone.

How it works

Like MNGs, Karpenter avoids provisioning capacity in impaired zones. It detects the type of shift (manual or auto) and which zones are currently affected by polling the ARC GetManagedResources API every 30 seconds. When service to the zones has been restored or when a zonal shift has expired, Karpenter resumes provisioning capacity in all the node pool’s configured zones. If your node pools don’t specify the topology.kubernetes.io/zone key, but subnet auto-discovery is enabled, Karpenter continues provisioning capacity in those subnets.

During zonal impairments, nodes in the affected zone are tainted and unable to have pods scheduled onto them. Because Karpenter also cannot provision in the impaired AZ, it doesn’t disrupt nodes in the affected zone until the zonal shift expires or services in that zone are restored. If a disruption requires replacement capacity that depends on the impaired zone, validation fails and the disruption does not proceed.

Best practices

There are several things you can do to improve the reliability of your applications.

Distribute pods across Availability Zones. Spread pods across worker nodes in at least two Availability Zones using topology spread constraints or node affinity. When using either option, allow the scheduler to ignore constraints if they cannot be met. Otherwise, pods never get scheduled during an outage. For node affinity, use preferredDuringSchedulingIgnoredDuringExecution instead of requiredDuringSchedulingIgnoredDuringExecution. For topology spread, set whenUnsatisfiable to ScheduleAnyway instead of DoNotSchedule, or increase maxSkew to allow more variance between the number of pods in targeted zones.

Avoid singletons. Don’t run a single replica in a Deployment. If your only replica is running in a zone that experiences an issue, that application will be unreachable for the duration of the outage.

Configure pod disruption budgets (PDBs). With PDBs, you can specify the minimum number of pods in a set that must always be running. Pods can’t be evicted unless replacement pods from that set are scheduled first. This prevents cluster autoscalers like Karpenter from voluntarily terminating a node before replacement pods are ready.

Note: Forceful terminations will still be processed, for example, Amazon Elastic Compute Cloud (Amazon EC2) Spot interruption events.

Be cognizant of where you run stateful services. Amazon Elastic Block Store (Amazon EBS) volumes are zonal, in that they’re confined to a particular Availability Zone. If that zone experiences an issue, you might lose access to the volumes in that zone. Create periodic snapshots of Amazon EBS volumes (using the Amazon EBS CSI Snapshotter API) and replicate them to other zones so those snapshots can be used to hydrate volumes in alternative zones. You can also investigate whether the stateful service you’re using natively supports replication. If replication isn’t an option, create a backup of the data, for example, with AWS Backup.

Knowing where you’re exposed

The practices above matter most for workloads that actually have zonal redundancy gaps, and at portfolio scale, finding those manually is hard. The next generation of AWS Resilience Hub runs generative-AI failure mode assessments against your services, evaluating them with the AWS Well-Architected best practices and the AWS Resilience Analysis Framework. It surfaces exactly the issues this section addresses: singleton deployments, missing multi-AZ spread, zonal stateful storage, and absent pod disruption budgets. Each finding maps to a requirement in a reusable resilience policy (for example, an availability SLO or an AZ-failure recovery time objective) and comes with how-to-fix guidance you can mark as resolved after you’ve addressed it. Use it to decide which workloads warrant Zonal Shift, then use the following test to prove the control behaves as expected.

Enabling Zonal Shift support (OSS Karpenter)

To enable zonal shift in Karpenter the following conditions must be met:

  • Karpenter version 1.12.0 or higher.
  • The flag for Zonal Shift, --enableZonalShift, needs to be set to true on the Karpenter controller.
  • The AWS Identity and Access Management (IAM) role assigned to the Karpenter controller needs a policy that allows the ARC GetManagedResources API to be called.
  • Zonal Shift must be enabled on the EKS cluster.

We’ve created a GitHub repository that includes Terraform examples and scripts that will help you install Karpenter and enable Zonal Shift in new and existing clusters. The module detects your current Karpenter state (not installed, < v1.12.0, or >= v1.12.0) and takes the appropriate action: install, upgrade, or attach the required IAM permission.

Enabling Zonal Shift on Auto Mode clusters

On Auto Mode, enabling Zonal Shift is a single step. There’s no Helm flag to set, no IAM policy to add, and no service-linked role to create. Auto Mode detects shift signals internally, so your cluster’s IAM configuration doesn’t change.

aws eks update-cluster-config \
  --name <cluster-name> \
  --region eu-north-1 \
  --zonal-shift-config enabled=true

That’s it. Compare this to the four steps required for OSS Karpenter (upgrade, Helm flag, IAM policy, enable on cluster). The observable behavior during a Zonal Shift is identical.

Note: Enabling Zonal Shift on the cluster makes it eligible for manual shifts and registers it as a managed resource in ARC. If you also want AWS to automatically trigger shifts when it detects a zonal impairment, you need to separately enable Zonal Autoshift through the ARC console or API. Without autoshift, you trigger shifts manually.

Running a test

Here’s the full cycle running end to end on a live EKS Auto Mode cluster: deploy a multi-AZ workload, trigger a manual Zonal Shift, observe the behavior, and verify recovery.

Prerequisites

  • An EKS Auto Mode cluster with Zonal Shift enabled (see the earlier section Enabling Zonal Shift on Auto Mode clusters).
  • kubectl configured to talk to the cluster.
  • AWS CLI v2.

Step 1: Deploy a multi-AZ workload

Deploy a 9-replica nginx deployment with topology spread constraints across all Availability Zones, and expose it with a LoadBalancer service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zonal-shift-test
  namespace: default
spec:
  replicas: 9
  selector:
    matchLabels:
      app: zonal-shift-test
  template:
    metadata:
      labels:
        app: zonal-shift-test
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: zonal-shift-test
      containers:
      - name: nginx
        image: public.ecr.aws/nginx/nginx:latest
        resources:
          requests:
            cpu: 500m
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: zonal-shift-test
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: zonal-shift-test
  ports:
  - port: 80
    targetPort: 80
kubectl apply -f zonal-shift-test.yaml

Step 2: Verify pre-shift state

Wait for pods to spread across zones, then capture the baseline:

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,TAINTS:.spec.taints
NAME                  ZONE          TAINTS
i-002fb7e2742fcc16d   eu-north-1a   <none>
i-00670d3ba819818ac   eu-north-1b   <none>
i-0f85cd1eb97a38312   eu-north-1c   <none>

$ kubectl get pods -l app=zonal-shift-test -o wide
NAME                                READY   STATUS    IP                NODE
zonal-shift-test-5794c8bbf7-6g8qp   1/1     Running   192.168.115.112   i-002fb7e2742fcc16d   # eu-north-1a
zonal-shift-test-5794c8bbf7-jvdd4   1/1     Running   192.168.115.114   i-002fb7e2742fcc16d   # eu-north-1a
zonal-shift-test-5794c8bbf7-xbft2   1/1     Running   192.168.115.113   i-002fb7e2742fcc16d   # eu-north-1a
zonal-shift-test-5794c8bbf7-9szmj   1/1     Running   192.168.142.2     i-00670d3ba819818ac   # eu-north-1b
zonal-shift-test-5794c8bbf7-qntd2   1/1     Running   192.168.142.0     i-00670d3ba819818ac   # eu-north-1b
zonal-shift-test-5794c8bbf7-vfj8j   1/1     Running   192.168.142.1     i-00670d3ba819818ac   # eu-north-1b
zonal-shift-test-5794c8bbf7-8465p   1/1     Running   192.168.178.192   i-0f85cd1eb97a38312   # eu-north-1c
zonal-shift-test-5794c8bbf7-dmf4w   1/1     Running   192.168.178.193   i-0f85cd1eb97a38312   # eu-north-1c
zonal-shift-test-5794c8bbf7-kjw54   1/1     Running   192.168.178.194   i-0f85cd1eb97a38312   # eu-north-1c

Three pods per zone, all nodes untainted, and all nine pod IPs listed as ready in the endpoint slices.

Step 3: Trigger a manual zonal shift

The ARC API uses AZ IDs (for example, eun1-az1) rather than AZ names (for example, eu-north-1a). Map names to IDs first:

$ aws ec2 describe-availability-zones --region eu-north-1 \
  --query "AvailabilityZones[].{Name:ZoneName,Id:ZoneId}" --output table
+-------------+----------+
| eu-north-1a | eun1-az1 |
| eu-north-1b | eun1-az2 |
| eu-north-1c | eun1-az3 |
+-------------+----------+

Start a Zonal Shift with a 30-minute expiry:

$ CLUSTER_ARN=$(aws eks describe-cluster --name zonal-shift-auto-test \
  --query "cluster.arn" --output text)

$ aws arc-zonal-shift start-zonal-shift \
  --resource-identifier "${CLUSTER_ARN}" \
  --away-from "eun1-az1" \
  --expires-in "30m" \
  --comment "Testing zonal shift behavior with EKS Auto Mode"
{
  "awayFrom": "eun1-az1",
  "status": "ACTIVE",
  "zonalShiftId": "54cf82a6-31d1-2401-52e5-1d005a08bce7",
  ...
}

Save the zonalShiftId for canceling the shift later.

Step 4: Observe zonal shift behavior

Wait 60–90 seconds for the shift to take effect, then observe:

Nodes in the affected zone are cordoned and tainted:

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,TAINTS:.spec.taints
NAME                  ZONE          TAINTS
i-002fb7e2742fcc16d   eu-north-1a   [eks-arc-zonal-shift/impaired-zone=eu-north-1a:NoSchedule, node.kubernetes.io/unschedulable]
i-00670d3ba819818ac   eu-north-1b   <none>
i-0f85cd1eb97a38312   eu-north-1c   <none>

The taint eks-arc-zonal-shift/impaired-zone=<zone>:NoSchedule prevents new pods from being scheduled onto these nodes. The node.kubernetes.io/unschedulable taint cordons them as well, providing a redundant guarantee that no new work lands in the impaired zone.

Pods are still running (replica count is unchanged):

$ kubectl get pods -l app=zonal-shift-test --field-selector status.phase=Running --no-headers | wc -l

All nine pods remain Running. Zonal shift doesn’t evict existing pods. Instead, it prevents new scheduling and removes them from load balancing.

Note: When pods that are part of a Kubernetes Service are deregistered from an AWS Load Balancer or removed from their corresponding EndpointSlices, the carrying capacity of that service temporarily decreases. If the replicas in healthy zones can’t process the incoming traffic for the service, you might need to scale out those services. After the Zonal Shift expires or the impairment is resolved, you can scale the services back in.

Endpoints in the impaired zone are deregistered:

$ kubectl get endpointslices -l kubernetes.io/service-name=zonal-shift-test \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} zone= ready={.conditions.ready}{"\n"}{end}'
192.168.142.1 zone=eu-north-1b ready=true
192.168.142.0 zone=eu-north-1b ready=true
192.168.142.2 zone=eu-north-1b ready=true
192.168.178.192 zone=eu-north-1c ready=true
192.168.178.194 zone=eu-north-1c ready=true
192.168.178.193 zone=eu-north-1c ready=true

Only six of nine pod IPs remain in the endpoint slices, all from eu-north-1b and eu-north-1c. The three pods running on the eu-north-1a node are excluded from load balancing. Traffic no longer routes to the impaired zone.

New capacity respects the shift:

$ kubectl scale deployment zonal-shift-test --replicas=15
$ sleep 90
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,TAINTS:.spec.taints
NAME                  ZONE          TAINTS
i-002fb7e2742fcc16d   eu-north-1a   [eks-arc-zonal-shift/impaired-zone=eu-north-1a:NoSchedule, ...]
i-00670d3ba819818ac   eu-north-1b   <none>
i-036491d47fd5cb299   eu-north-1b   <none>   # new
i-069e1fb376f8251bd   eu-north-1c   <none>   # new
i-0f85cd1eb97a38312   eu-north-1c   <none>

New nodes launched to handle the scale-up are placed only in healthy zones (eu-north-1b and eu-north-1c). Karpenter doesn’t provision new capacity in the impaired zone. Voluntary disruptions (consolidation, drift) are also suspended in the affected zone, so existing workloads remain stable until recovery.

Step 5: Cancel the shift and verify recovery

$ aws arc-zonal-shift cancel-zonal-shift \
  --zonal-shift-id "54cf82a6-31d1-2401-52e5-1d005a08bce7"
{
  "status": "CANCELED",
  ...
}

Wait 60–90 seconds for recovery to propagate:

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,TAINTS:.spec.taints
NAME                  ZONE          TAINTS
i-002fb7e2742fcc16d   eu-north-1a   <none>
i-00670d3ba819818ac   eu-north-1b   <none>
i-036491d47fd5cb299   eu-north-1b   <none>
i-069e1fb376f8251bd   eu-north-1c   <none>
i-0f85cd1eb97a38312   eu-north-1c   <none>

$ kubectl get endpointslices -l kubernetes.io/service-name=zonal-shift-test \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} zone={"\n"}{end}' | sort -t= -k2
192.168.115.112 zone=eu-north-1a
192.168.115.115 zone=eu-north-1a
192.168.115.116 zone=eu-north-1a
192.168.103.224 zone=eu-north-1a
192.168.103.225 zone=eu-north-1a
192.168.103.226 zone=eu-north-1a
192.168.142.0 zone=eu-north-1b
192.168.142.1 zone=eu-north-1b
192.168.142.2 zone=eu-north-1b
192.168.137.224 zone=eu-north-1b
192.168.178.192 zone=eu-north-1c
192.168.178.193 zone=eu-north-1c
192.168.178.194 zone=eu-north-1c
192.168.182.144 zone=eu-north-1c
192.168.182.145 zone=eu-north-1c

Taints are removed, all 15 pod IPs are restored across all three zones, and Karpenter resumes normal provisioning everywhere.

Step 6: (Optional) Test with AWS Fault Injection Service

To simulate how zonal autoshift would trigger automatically during a real impairment using the AWS Fault Injection Service (FIS):

  1. Enable Zonal Autoshift on the cluster through the ARC console.
  2. Create a FIS experiment using the “AZ Availability: Power Interruption” scenario from the scenario library.
  3. The aws:arc:start-zonal-autoshift recovery action triggers automatically 5 minutes into the experiment.
  4. Observe the same previous behavior, but triggered by AWS rather than a manual API call.

See the FIS Zonal Autoshift testing documentation for detailed instructions.

Clean up

This test uses Amazon EKS and Amazon EC2 resources that incur costs. Complete the following steps to avoid ongoing charges.

Delete the sample workload

kubectl delete deployment zonal-shift-test
kubectl delete service zonal-shift-test

(Optional) Disable zonal shift on the cluster

aws eks update-cluster-config \
  --name <cluster-name> \
  --region eu-north-1 \
  --zonal-shift-config enabled=false

Conclusion

ARC Zonal Shift gives you a single-API recovery mechanism when an Availability Zone goes bad. On EKS Auto Mode, that integration is one configuration flag away. No IAM changes, no Helm values, no controller upgrades. Enable it, and Karpenter handles the rest: tainting nodes in the affected zone, deregistering endpoints, blocking new capacity there, and suspending voluntary disruptions until the zone recovers. Pair it with Zonal Autoshift and you don’t even need to press the button. AWS detects the impairment and shifts for you.

Zonal Shift is the recovery control. The next generation of AWS Resilience Hub is how you find the workloads that need it and confirm they’re built to survive an AZ loss. Assess with Resilience Hub, implement the multi-AZ practices it recommends, validate with a zonal shift (or FIS-driven autoshift), and you’ve closed the resilience loop: assess, implement, test, recover.

To get started, see the following resources:

Enable EKS zonal shift (EKS User Guide)

Amazon Application Recovery Controller (service page)

Karpenter ARC Zonal Shift Terraform module (GitHub)

Enhance Kubernetes high availability with ARC and Karpenter integration (related blog)

End-to-end recovery from AZ impairments in Amazon EKS using EKS Zonal shift and Istio (related blog)

About the authors

Jeremy Cowan

Jeremy Cowan

Jeremy is a Principal Solutions Architect for containers at AWS. Prior to joining AWS, Jeremy worked for several large software vendors, including VMware, Microsoft, and IBM.

Ajay Desai

Ajay Desai

Ajay is a Senior Technical Account Manager (TAM) at AWS with over 17 years of experience in the IT industry. He is passionate about containers, cost optimization, and operational excellence, focusing on helping customers design and maintain secure, reliable, efficient, and scalable solutions.

Sajjan Gundapuneedi

Sajjan Gundapuneedi

Sajjan is a Sr Manager of Software Development at AWS. He leads EKS compute infrastructure including EKS Auto Mode, OSS Karpenter, Managed Node Groups, Fargate, EKS Hybrid Nodes, and core EKS data-plane components including networking and runtime AMIs.