AWS Partner Network (APN) Blog

Building Production-Grade Kubernetes Clusters with Amazon EKS Anywhere on Nutanix

By Elamaran Shanmugam, Sr. Partner Solutions Architect, Containers – AWS
By Yannick Struyf, Sr. Product Manager – Nutanix


Amazon EKS Anywhere (Amazon EKS-A) allows customers to run containerized workloads on customer-managed infrastructure. It provides an installable software package for creating and operating Kubernetes clusters on-premises and automation tooling for cluster lifecycle support.

Amazon EKS Anywhere brings a consistent Amazon Web Services (AWS) management experience to your data center, building on the strengths of Amazon EKS Distro, an open-source distribution for Kubernetes used by Amazon Elastic Kubernetes Service (Amazon EKS).

In December 2022, the general availability of Amazon EKS Anywhere on Nutanix was announced with AHV virtualization. This expands the choice of infrastructure options for customers running Kubernetes on-premises.

Nutanix enhances the list of deployment options for EKS Anywhere customers, which already includes bare metal servers, VMware vSphere, Snow, Apache CloudStack, and Docker.

Nutanix is an AWS Partner that provides hyperconverged infrastructure software with enterprise-grade data services for stateful application needs including volumes, objects, files, and Nutanix Database Services (NDB).

AWS collaborated with Nutanix to integrate Amazon EKS Anywhere with the Cluster API provider for Nutanix (CAPX) to provide customers with declarative, Kubernetes-style APIs for cluster creation, configuration, and management.

In this post, we’ll show you how to build a production-grade Kubernetes cluster with EKS-A on Nutanix.

Solution Overview

Here is the overall architecture for setting up an Amazon EKS Anywhere cluster on the Nutanix platform.


Figure 1 – Architecture of setting up an EKS-A cluster on Nutanix.

  • The cluster creation workflow begins from an EKS-A Admin instance running on the Nutanix environment, where a kind bootstrap cluster running inside a Docker container ingests a cluster configuration file to stand up the corresponding EKS-A cluster.
  • Cilium is used as a container network interface (CNI) plugin, and direct network interfaces (DNIs) are created and associated with each cluster node, allowing pods to communicate with each other without network address translation (NAT). Avoiding NAT helps to achieve better network performance, lower latencies, and improve security.
  • kube-vip is used as a control plane load balancer. In this context, kube-vip is running as a static pod on the control plane nodes and will use address resolution protocol (ARP) to update the route mapping between the cluster endpoint virtual IP address (VIP) and the corresponding hardware media access control (MAC) address upon failover. kube-vip also uses the Kubernetes Go client library to perform leader election for new control plane nodes in the event of failover.


Note that Amazon EKS Anywhere currently supports Ubuntu as a node image for your EKS-A cluster on Nutanix.

Let’s start by setting a few environment variables in your administrative machine:

# EKSA Environment variables
export EKSA_NUTANIX_USERNAME='<Your-Nutanix-Username>'
export EKSA_NUTANIX_PASSWORD='<Your-Nutanix-Password>'

# Curated Packages
export AWS_REGION=us-west-2 # Region of Availability for Curated Packages.
export EKSA_AWS_REGION="us-west-2" # Region of Availability for Curated Packages.
export CLUSTER_NAME=mgmt

Creating Your EKS-A Nutanix Cluster

Use the following eksctl anywhere command to generate the Nutanix EKS-A cluster configuration:

eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider nutanix > eksa-mgmt-cluster.yaml

Next, modify the initial cluster config (eksa-mgmt-cluster.yaml) as follows:

  • NutanixDatacenterConfig fields:
    • endpoint with Prism Central server fully qualified domain name or IP address.
  • NutanixMachineConfig fields:
    • with Prism Element cluster name.
    • with name of the OS image.
    • with subnet name to be assigned to the virtual machines.
  • The rest of the configurations in the eksa-mgmt-cluster.yaml file can be updated per your needs. Refer to Nutanix configuration for information on configuring this cluster config for a Nutanix provider.
  • Add Optional configuration settings as needed.

We recommend at least three control plane nodes and three worker nodes for a production cluster to provide high availability and rolling upgrades with minimal disruption.

Next, create an EKS-A cluster for Nutanix by executing the following:

eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml

This output provides progress information as the cluster is created. If you see the below output, your EKS-A cluster on Nutanix is up and running.

To understand more about the cluster creation process, review the cluster creation workflow documentation.

2023-02-22T16:34:51.081Z    V0    🎉 Cluster created!
2023-02-22T16:34:51.081Z    V4    Task start    {"task_name": "install-curated-packages"}
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
2023-02-22T16:35:21.563Z    V4    Tasks completed    {"duration": "14m43.618820376s"}

After the EKS-A cluster is created, you can set an environment variable pointing to the kubeconfig file that was generated as part of the same process.

The kubeconfig file will then be used by kubectl CLI to access the Amazon EKS-A cluster. For example, you can verify the workload cluster creation process was completed by listing the machines to view the status of both the control plane and data plane nodes.

> export KUBECONFIG=$PWD/$CLUSTER_NAME_A/$CLUSTER_NAME_A-eks-a-cluster.kubeconfig

> kubectl get machinedeployments -A
eksa-system   mgmt-md-0    mgmt      3          3       3         0             Running     18m   v1.23.15-eks-1-23-12

> kubectl get machines -A
NAMESPACE     NAME                          CLUSTER   NODENAME                                          PROVIDERID                                       PHASE         AGE   VERSION
eksa-system   mgmt-flx8k                    mgmt      mgmt-control-plane-template-1676996603686-zmg9q   nutanix://c58172ff-e5b4-4bea-a3c9-5d552f99d517   Running       24h   v1.23.15-eks-1-23-12
eksa-system   mgmt-jr9cd                    mgmt      mgmt-control-plane-template-1676996603686-ph82c   nutanix://c474b4b4-1b18-4556-a30e-41d4e3367e77   Running       24h   v1.23.15-eks-1-23-12
eksa-system   mgmt-l84ls                    mgmt      mgmt-control-plane-template-1676996603686-sjhql   nutanix://89759877-7ea5-40f4-8d42-c5165bdbb769   Running       24h   v1.23.15-eks-1-23-12
eksa-system   mgmt-md-0-5955dd4ff4-7dhwz    mgmt      mgmt-md-0-1676996603687-f7jm8                     nutanix://6e59cda9-50e0-40d9-8470-697e75399513   Running       21h   v1.23.15-eks-1-23-12
eksa-system   mgmt-md-0-5955dd4ff4-f454l    mgmt      mgmt-md-0-1676996603687-lfh4d                     nutanix://52c47485-9779-432c-83dc-78e17a4bde02   Running       21h   v1.23.15-eks-1-23-12
eksa-system   mgmt-md-0-5955dd4ff4-jkpxw    mgmt      mgmt-md-0-1676996603687-nzxvk                     nutanix://23526d43-fcb7-4438-92a7-3b775df7cde0   Running       21h   v1.23.15-eks-1-23-12

Deploying a Workload

Now, let’s deploy a workload to test it:

kubectl apply -f ""

You’ll see a hello-eks-a deployment and service created in your cluster:

deployment.apps/hello-eks-a created
service/hello-eks-a created

Verify the workload by forwarding the service port to the host’s port 3000 with the following command:

kubectl port-forward svc/hello-eks-a 3000:80

Check that the service is running and available with:

curl localhost:3000

You should see output that looks like this:


Thank you for using

███████╗██╗  ██╗███████╗
██╔════╝██║ ██╔╝██╔════╝
█████╗  █████╔╝ ███████╗
██╔══╝  ██╔═██╗ ╚════██║
███████╗██║  ██╗███████║
╚══════╝╚═╝  ╚═╝╚══════╝

 █████╗ ███╗   ██╗██╗   ██╗██╗    ██╗██╗  ██╗███████╗██████╗ ███████╗
██╔══██╗████╗  ██║╚██╗ ██╔╝██║    ██║██║  ██║██╔════╝██╔══██╗██╔════╝
███████║██╔██╗ ██║ ╚████╔╝ ██║ █╗ ██║███████║█████╗  ██████╔╝█████╗
██╔══██║██║╚██╗██║  ╚██╔╝  ██║███╗██║██╔══██║██╔══╝  ██╔══██╗██╔══╝
██║  ██║██║ ╚████║   ██║   ╚███╔███╔╝██║  ██║███████╗██║  ██║███████╗
╚═╝  ╚═╝╚═╝  ╚═══╝   ╚═╝    ╚══╝╚══╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚══════╝

You have successfully deployed the hello-eks-a pod hello-eks-a-866ff6bbc7-hs5pn

For more information check out


Upgrading Your EKS-A Nutanix Cluster

Amazon EKS Anywhere provides you with the upgrade command, which allows you to upgrade various aspects of your EKS-A cluster. After updating your configuration file, when you run the below commands EKS-A runs a set of preflight checks to ensure your cluster is ready to be upgraded.

EKS-A then performs the upgrade, modifying your cluster to match the updated specification and core components of EKS-A. EKS-A documentation on upgrade will provide you more information on upgrade of your EKS-A cluster on Nutanix.

Before you perform an upgrade, check the current and new versions of components that are ready to be upgraded by typing:

eksctl anywhere upgrade plan cluster -f eksa-mgmt-cluster.yaml

Run the below upgrade command to upgrade your EKS-A cluster on Nutanix:

eksctl anywhere upgrade cluster -f eksa-mgmt-cluster.yaml

Storage Management for Stateful Applications

Storage Management on Nutanix is an essential ingredient for running stateful applications on your EKS-A cluster. Nutanix distributed storage delivers storage using multiple protocols, including Network File System (NFS), Server Message Block (SMB), and Internet Small Computer System Interface (iSCSI).

The Nutanix Container Storage Interface (CSI) Volume Driver for Kubernetes uses Nutanix Volumes and Nutanix Files to provide scalable, persistent storage for stateful applications.

Kubernetes contains an in-tree CSI Volume plugin that allows the out-of-tree Nutanix CSI Volume Driver to gain access to containers and provide persistent-volume storage. The plugin runs in a pod and dynamically provisions requested PersistentVolumes (PVs) using Nutanix Files and Nutanix Volumes storage.

The Nutanix CSI Snapshot Controller is responsible for deploying the snapshot controller, webhook validation, as well as the related custom resource definitions. This is required for CSI snapshotting to work and is not specific to any CSI driver. For more information, see CSI Volume Snapshots.

Nutanix Volumes is enterprise-class, software-defined storage that exposes storage resources directly to virtualized guest operating systems or physical hosts using the iSCSI protocol.

Nutanix CSI driver provides ReadWriteOnce Persistent Volume (PV) using Volumes. You must first configure a storage container in an AOS cluster with the required storage properties such as deduplication, compression, erasure coding, and thin provisioning. Then, provide the storage container as input when creating the storage class in your Kubernetes cluster.

Now, all of the PVCs created with the newly-created storage class will have the configured storage properties. Nutanix Volumes expose the provisioned volumes as iSCSI targets. For more information about Nutanix Volumes, see the Volumes Guide.

Make sure systemctl iSCSID service is running on your worker nodes to enable storage management working on your EKS-A clusters to access Nutanix Volumes.

Nutanix Files is a software-defined, scale-out file storage solution that provides a repository for unstructured data, such as home directories, user profiles, departmental shares, application logs, backups, and archives. Flexible and responsive to workload requirements, Nutanix Files is a fully integrated, core component of Nutanix platform.

Things to Know

Launch Partners

AWS Partners are key to the success of Amazon EKS Anywhere. The following partners have validated their software solutions through the Amazon EKS Anywhere Conformance and Validation Framework, extending their GitOps-driven integrations to Amazon EKS-A on Nutanix environment.

Customers can deploy the validated solutions these partners provide to operate their EKS-A clusters on Nutanix environment, addressing common production readiness concerns such as secrets management, storage, and maintenance of third-party components across a distributed fleet of devices.

  • Dynatrace OneAgent provides you with operational and business performance metrics.
  • HashiCorp Vault helps you manage secrets and protect sensitive data.
  • Kubecost is a popular open-source cloud cost monitoring tool.
  • Sysdig enables you to confidently secure containers, Kubernetes, and cloud services.
  • SUSE NeuVector is an open-source, zero-trust container security platform.


Once you’re done, you can delete the cluster with the following command:

eksctl anywhere delete cluster $CLUSTER_NAME --kubeconfig $KUBECONFIG


In this post, we showed how to build a near- to production-grade Kubernetes cluster with Amazon EKS Anywhere on Nutanix. We also covered storage management as well as AWS launch partners for EKS-A on Nutanix.

For more information on getting started with Amazon EKS Anywhere, check out the EKS-A workshop, EKS-A documentation, or frequently asked questions.

Check out the EKS-A GitHub repo and join the community Slack channel in the Kubernetes workspace to contribute.


Nutanix – AWS Partner Spotlight

Nutanix is an AWS Partner that provides hyperconverged infrastructure software with enterprise-grade data services for stateful application needs including volumes, objects, files, and Nutanix Database Services (NDB).

Contact Nutanix | Partner Overview | AWS Marketplace