AWS Partner Network (APN) Blog
Building Production-Grade Kubernetes Clusters with Amazon EKS Anywhere on Nutanix
By Elamaran Shanmugam, Sr. Partner Solutions Architect, Containers – AWS
By Yannick Struyf, Sr. Product Manager – Nutanix
 |
| Nutanix |
 |
Amazon EKS Anywhere (Amazon EKS-A) allows customers to run containerized workloads on customer-managed infrastructure. It provides an installable software package for creating and operating Kubernetes clusters on-premises and automation tooling for cluster lifecycle support.
Amazon EKS Anywhere brings a consistent Amazon Web Services (AWS) management experience to your data center, building on the strengths of Amazon EKS Distro, an open-source distribution for Kubernetes used by Amazon Elastic Kubernetes Service (Amazon EKS).
In December 2022, the general availability of Amazon EKS Anywhere on Nutanix was announced with AHV virtualization. This expands the choice of infrastructure options for customers running Kubernetes on-premises.
Nutanix enhances the list of deployment options for EKS Anywhere customers, which already includes bare metal servers, VMware vSphere, Snow, Apache CloudStack, and Docker.
Nutanix is an AWS Partner that provides hyperconverged infrastructure software with enterprise-grade data services for stateful application needs including volumes, objects, files, and Nutanix Database Services (NDB).
AWS collaborated with Nutanix to integrate Amazon EKS Anywhere with the Cluster API provider for Nutanix (CAPX) to provide customers with declarative, Kubernetes-style APIs for cluster creation, configuration, and management.
In this post, we’ll show you how to build a production-grade Kubernetes cluster with EKS-A on Nutanix.
Solution Overview
Here is the overall architecture for setting up an Amazon EKS Anywhere cluster on the Nutanix platform.
Figure 1 – Architecture of setting up an EKS-A cluster on Nutanix.
- The cluster creation workflow begins from an EKS-A Admin instance running on the Nutanix environment, where a kind bootstrap cluster running inside a Docker container ingests a cluster configuration file to stand up the corresponding EKS-A cluster.
- Cilium is used as a container network interface (CNI) plugin, and direct network interfaces (DNIs) are created and associated with each cluster node, allowing pods to communicate with each other without network address translation (NAT). Avoiding NAT helps to achieve better network performance, lower latencies, and improve security.
- kube-vip is used as a control plane load balancer. In this context, kube-vip is running as a static pod on the control plane nodes and will use address resolution protocol (ARP) to update the route mapping between the cluster endpoint virtual IP address (VIP) and the corresponding hardware media access control (MAC) address upon failover. kube-vip also uses the Kubernetes Go client library to perform leader election for new control plane nodes in the event of failover.
Prerequisites
- Administrative machine with machine requirements as listed in the EKS-A product page with the following tools installed:
- AWS Command Line Interface (CLI) version 2.10 or higher
- Helm
- eksctl
- eks-anywhere (v0.14.0 or later)
- kubectl
- Nutanix Cloud Infrastructure with required resources as listed in EKS-A documentation.
- Nutanix user with Prism Central permissions to create EKS-A clusters on top of your Nutanix cluster.
- Build and import of Nutanix AHV Ubuntu node images to AOS image service per instructions in the artifacts documentation section.
- Amazon EKS-A Curated Packages are only available to customers with the Amazon EKS-A Enterprise Subscription. To request a free trial, connect with your AWS representative.
Note that Amazon EKS Anywhere currently supports Ubuntu as a node image for your EKS-A cluster on Nutanix.
Let’s start by setting a few environment variables in your administrative machine:
Creating Your EKS-A Nutanix Cluster
Use the following eksctl anywhere command to generate the Nutanix EKS-A cluster configuration:
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
--provider nutanix > eksa-mgmt-cluster.yaml
Next, modify the initial cluster config (eksa-mgmt-cluster.yaml) as follows:
NutanixDatacenterConfigfields:endpointwith Prism Central server fully qualified domain name or IP address.
NutanixMachineConfigfields:cluster.namewith Prism Element cluster name.image.namewith name of the OS image.subnet.namewith subnet name to be assigned to the virtual machines.
- The rest of the configurations in the
eksa-mgmt-cluster.yamlfile can be updated per your needs. Refer to Nutanix configuration for information on configuring this cluster config for a Nutanix provider. - Add Optional configuration settings as needed.
We recommend at least three control plane nodes and three worker nodes for a production cluster to provide high availability and rolling upgrades with minimal disruption.
Next, create an EKS-A cluster for Nutanix by executing the following:
eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml
This output provides progress information as the cluster is created. If you see the below output, your EKS-A cluster on Nutanix is up and running.
To understand more about the cluster creation process, review the cluster creation workflow documentation.
After the EKS-A cluster is created, you can set an environment variable pointing to the kubeconfig file that was generated as part of the same process.
The kubeconfig file will then be used by kubectl CLI to access the Amazon EKS-A cluster. For example, you can verify the workload cluster creation process was completed by listing the machines to view the status of both the control plane and data plane nodes.
Deploying a Workload
Now, let’s deploy a workload to test it:
kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"
You’ll see a hello-eks-a deployment and service created in your cluster:
deployment.apps/hello-eks-a created
service/hello-eks-a created
Verify the workload by forwarding the service port to the host’s port 3000 with the following command:
kubectl port-forward svc/hello-eks-a 3000:80
Check that the service is running and available with:
curl localhost:3000
You should see output that looks like this:
Upgrading Your EKS-A Nutanix Cluster
Amazon EKS Anywhere provides you with the upgrade command, which allows you to upgrade various aspects of your EKS-A cluster. After updating your configuration file, when you run the below commands EKS-A runs a set of preflight checks to ensure your cluster is ready to be upgraded.
EKS-A then performs the upgrade, modifying your cluster to match the updated specification and core components of EKS-A. EKS-A documentation on upgrade will provide you more information on upgrade of your EKS-A cluster on Nutanix.
Before you perform an upgrade, check the current and new versions of components that are ready to be upgraded by typing:
eksctl anywhere upgrade plan cluster -f eksa-mgmt-cluster.yaml
Run the below upgrade command to upgrade your EKS-A cluster on Nutanix:
eksctl anywhere upgrade cluster -f eksa-mgmt-cluster.yaml
Storage Management for Stateful Applications
Storage Management on Nutanix is an essential ingredient for running stateful applications on your EKS-A cluster. Nutanix distributed storage delivers storage using multiple protocols, including Network File System (NFS), Server Message Block (SMB), and Internet Small Computer System Interface (iSCSI).
The Nutanix Container Storage Interface (CSI) Volume Driver for Kubernetes uses Nutanix Volumes and Nutanix Files to provide scalable, persistent storage for stateful applications.
Kubernetes contains an in-tree CSI Volume plugin that allows the out-of-tree Nutanix CSI Volume Driver to gain access to containers and provide persistent-volume storage. The plugin runs in a pod and dynamically provisions requested PersistentVolumes (PVs) using Nutanix Files and Nutanix Volumes storage.
The Nutanix CSI Snapshot Controller is responsible for deploying the snapshot controller, webhook validation, as well as the related custom resource definitions. This is required for CSI snapshotting to work and is not specific to any CSI driver. For more information, see CSI Volume Snapshots.
Nutanix Volumes is enterprise-class, software-defined storage that exposes storage resources directly to virtualized guest operating systems or physical hosts using the iSCSI protocol.
Nutanix CSI driver provides ReadWriteOnce Persistent Volume (PV) using Volumes. You must first configure a storage container in an AOS cluster with the required storage properties such as deduplication, compression, erasure coding, and thin provisioning. Then, provide the storage container as input when creating the storage class in your Kubernetes cluster.
Now, all of the PVCs created with the newly-created storage class will have the configured storage properties. Nutanix Volumes expose the provisioned volumes as iSCSI targets. For more information about Nutanix Volumes, see the Volumes Guide.
Make sure systemctl iSCSID service is running on your worker nodes to enable storage management working on your EKS-A clusters to access Nutanix Volumes.
Nutanix Files is a software-defined, scale-out file storage solution that provides a repository for unstructured data, such as home directories, user profiles, departmental shares, application logs, backups, and archives. Flexible and responsive to workload requirements, Nutanix Files is a fully integrated, core component of Nutanix platform.
Things to Know
- Operational support: There are no up-front commitments or fees to use Amazon EKS Anywhere. Customers can optionally purchase Amazon EKS Anywhere Enterprise Subscriptions for access to EKS-A Curated Packages as well as 24/7 support from AWS subject matter experts for all bundled tooling.
- Version support: Refer to the official Amazon EKS Anywhere and Kubernetes version support policy page.
- Pricing: Amazon EKS Anywhere Enterprise Subscription option available with EKS-A, which is required to get support for the EKS-A clusters and access to additional paid features such as Amazon EKS Anywhere Curated Packages.
- ISV Partner validation: The Amazon EKS Anywhere Conformance and Validation Framework is available to help partners validate their solutions for EKS-A on Nutanix deployments.
Launch Partners
AWS Partners are key to the success of Amazon EKS Anywhere. The following partners have validated their software solutions through the Amazon EKS Anywhere Conformance and Validation Framework, extending their GitOps-driven integrations to Amazon EKS-A on Nutanix environment.
Customers can deploy the validated solutions these partners provide to operate their EKS-A clusters on Nutanix environment, addressing common production readiness concerns such as secrets management, storage, and maintenance of third-party components across a distributed fleet of devices.
- Dynatrace OneAgent provides you with operational and business performance metrics.
- HashiCorp Vault helps you manage secrets and protect sensitive data.
- Kubecost is a popular open-source cloud cost monitoring tool.
- Sysdig enables you to confidently secure containers, Kubernetes, and cloud services.
- SUSE NeuVector is an open-source, zero-trust container security platform.
Cleanup
Once you’re done, you can delete the cluster with the following command:
eksctl anywhere delete cluster $CLUSTER_NAME --kubeconfig $KUBECONFIG
Conclusion
In this post, we showed how to build a near- to production-grade Kubernetes cluster with Amazon EKS Anywhere on Nutanix. We also covered storage management as well as AWS launch partners for EKS-A on Nutanix.
For more information on getting started with Amazon EKS Anywhere, check out the EKS-A workshop, EKS-A documentation, or frequently asked questions.
Check out the EKS-A GitHub repo and join the community Slack channel in the Kubernetes workspace to contribute.
.
.
Nutanix – AWS Partner Spotlight
Nutanix is an AWS Partner that provides hyperconverged infrastructure software with enterprise-grade data services for stateful application needs including volumes, objects, files, and Nutanix Database Services (NDB).