Getting started with Amazon EKS Anywhere on Bare Metal

We are happy to announce the general availability of Amazon EKS Anywhere on Bare Metal. We released Amazon EKS Anywhere in 2021 with support to run on-premises Kubernetes clusters using VMware and today you can use EKS Anywhere to provision clusters without virtualization. The new functionality adds support for managing the full hardware lifecycle to boot, provision, and operate clusters.

Running Kubernetes clusters on top of virtualization technologies can provide flexibility for your compute needs with live migrations, dynamic scaling, and hardware abstractions for the operating system. The new bare metal capability is provided in addition to the existing support for EKS Anywhere running on VMware. Many customers have asked for the ability to run EKS Anywhere directly on hardware with no virtualization. Running Kubernetes on bare metal instances will take full advantage of the servers because the operating system is communicating with the physical hardware. This allows workloads to take advantage of compute accelerators, local storage, and native I/O speeds. The benefits virtualization brings to virtual machines Kubernetes does for workloads without additional abstractions, management, or performance overhead.

EKS Anywhere is fully open source and free to use. It builds on a variety of existing, open-source projects such as, Tinkerbell for provisioning servers, kind for bootstrapping, and Cluster API for Kubernetes lifecycle management. Let’s create our first bare metal cluster.

Hardware requirements

The minimum specs for an EKS Anywhere server is:

4 CPU
8 GB memory
20 GB available storage
Network card that can PXE boot

You can run highly available clusters with five servers for the control plane—3 etcd, and 2 Kubernetes API—or you can run a cluster without high availability on a single server with a minimum of 16 GB of memory. Cluster workloads are not scheduled on the control plane, and you can have different node configurations in your cluster as long as they meet the minimum specs.

We have worked with multiple hardware providers and integrators to make sure your on-premises clusters are fully supported.

You can see a list of launch partners near the end of this post.

Create a cluster

To create a bare metal cluster you need to create an inventory file with information about the hardware you want to use. The file is in CSV format and will be used by EKS Anywhere’s bootstrap process to power on and install an operating system to your servers. You can customize the provisioning workflow by adding your own packages, configuration, or files as shown here in the documentation.

The CSV file needs the following information at a minimum:

Hostname
Installation disk
Network configuration

An example CSV file will have these fields:

hostname,labels,disk,mac,ip_address,netmask,gateway,nameservers
eks-a-1,type=cp,/dev/sda,f0:d2:f1:00:00:01,10.10.1.110,255.255.255.0,10.10.10.1.1,ns1
worker1,type=worker,/dev/sda,f0:d2:f1:00:00:02,10.10.1.111,255.255.255.0,10.10.10.1.1,ns1

A full example of the inventory file can be found in the documentation.

Install the EKS Anywhere CLI to generate a cluster configuration. The installation will work on macOS or Linux. Windows and Windows subsystem for Linux (WSL) are not supported. If you would like to manually install the CLI you can read how in the documentation.

brew install aws/tap/eks-anywhere

Now that we have our hardware configuration and CLI installed we can generate the cluster configuration. This is a declarative specification to configure a cluster.

In the following commands you can replace the $VARIABLES with your own configuration:

eksctl anywhere generate clusterconfig $CLUSTER_NAME \
    --provisioner tinkerbell > clusterconfig.yaml

Edit the configuration to add information unique to your cluster and environment. Open the clusterconfig.yaml file in your favorite text editor and edit the metadata.name entry to give the cluster a unique name and modify controlPlaneConfiguration.endpoint.host entry to an available IP address on your network. This IP address will be the IP address of your Kubernetes API server, but it should not already be in use on your network. It will be a virtual IP address (VIP) that allows for highly available API servers.

You also need to add an IP address for spec.tinkerbellIP under the TinkerBellDatacenterConfig object. This IP address will be used for PXE booting machines in the cluster.

kind: Cluster
metadata:
  name: $CLUSTER_NAME
spec:
  controlPlaneConfiguration:
    count: 1
    endpoint:
      host: $CONTROL_PLANE_IP
...
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellDatacenterConfig
metadata:
  name: $CLUSTER_NAME
spec:
  tinkerbellIP: $TINKERBELL_IP

You can optionally edit other configurations in the file. Some common configurations include:

Pod and service CIDR blocks
sshAuthorizedKeys for control plane and worker nodes

Once the configuration is set, you can begin the cluster creation process. Just like with other providers, EKS Anywhere will run a bootstrap cluster on your local machine to prepare the initial cluster. You can read more about how the bootstrap process works in the documentation.

Use the hardware inventory along with your cluster configuration to create the bare metal cluster.

eksctl anywhere create cluster --file clusterconfig.yaml \
    --hardware-csv hardware.csv

EKS Anywhere works with out-of-band (OOB) and lights-out management (LOM) hardware in servers. This allows you to control the power of a server over the network even if it is powered off and you don’t have physical access. EKS Anywhere uses the Intelligent Platform Management Interface (IPMI) protocol or Redfish to communicate with your servers and manage their power state to provision them.

If your hardware has an out-of-band management, and you provided the connection information in the inventory file, then EKS Anywhere will automatically boot your hardware, install the operating system, configure Kubernetes, and boot into the full, installed operating system.

You should see output similar to this when the process is done:

Performing setup and validations
✅ Tinkerbell Provider setup is valid
✅ Validate certificate for registry mirror
✅ Create preflight validations pass
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Creating new workload cluster
Installing networking on workload cluster
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Installing resources on management cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Installing EKS-D components on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing AddonManager and GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!

You can now use your EKS Anywhere cluster using the generated kubeconfig file in the local directory.

export KUBECONFIG=$CLUSTER_NAME/$CLUSTER_NAME.kubeconfig
kubectl get nodes

You’ll see the nodes you declared in your inventory file.

NAME       STATUS   ROLES                  AGE   VERSION
never      Ready    control-plane,master   56m   v1.22.10-eks-7dc61e8
gonna      Ready    <none>                 48m   v1.22.10-eks-7dc61e8
give-you   Ready    <none>                 48m   v1.22.10-eks-7dc61e8
up         Ready    <none>                 49m   v1.22.10-eks-7dc61e8

You can deploy a test workload to the cluster using:

kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"

Launch Partners

We have worked with a lot of partners to validate their hardware with EKS Anywhere on Bare Metal. Here is a small list of some of the partners and validated hardware.

LG CNS: Application modernization with containers has become the key competency for the enterprise. Enterprise customers have requirements that address policies for privacy, compliance, and edge use cases. Amazon EKS Anywhere provides the flexibility to accommodate the modernization of diverse customer workloads while extending Amazon Elastic Kubernetes Service (Amazon EKS) capabilities on premises. Furthermore, it is noteworthy that not only VMware but also bare metal options have been added to enable more efficient EKS environments, and consistent management of on-premises EKS environments is possible on AWS. With Amazon EKS Anywhere, LG CNS provides solutions for customers to achieve business modernization goals beyond application modernization. – Tae Hoon Kim, VP of Cloud Business, LG CNS

Nirmata: Amazon EKS Anywhere on Bare Metal brings the proven EKS distribution to bare metal infrastructure. Now, AWS customers can get consistent Kubernetes experience in the cloud and on-premises, on virtual machines and bare metal. Nirmata supports EKS Anywhere Bare Metal and provides unified operations, governance and security for EKS and EKS Anywhere clusters independent of where and how they are deployed. As an APN Partner, we are super excited to participate in this launch and look forward to working closely with the AWS team on accelerating our customers’ cloud-native journey. – Ritesh Patel, VP Products at Nirmata

Rafay: Rafay is pleased to be partnered with AWS on EKS Anywhere Bare Metal environments. Rafay’s Kubernetes Operations Platform (KOP) is uniquely positioned to help customers deliver enterprise-grade automation, security, visibility and governance requirements. Customers planning to leverage EKS Anywhere for on-premises, bare metal environments can now operate both public cloud and on-premises based clusters easily with Rafay. – Mohan Atreya, SVP of Product and Solutions, Rayfay

Tech Mahindra: Tech Mahindra’s netOps.ai enables a fully managed hybrid telco cloud experience on AWS using EKS-Anywhere on-premises bare metal hardware through a single click. netOps.ai simplifies all operations, which are typically one of the most significant sources of expenditures for dedicated on-premises networks, by running such operations on EKS-Anywhere providing continuous integration and continuous delivery pipelines for the entire lifecycle of the Solution and a fully automated installation and maintenance. – Tulasi Ram Bora, Chief Architect for ai in Tech Mahindra

Tetrate: “Customers, while migrating their applications to the cloud and embracing containers, often choose to span their application between on-premises and cloud,” said Varun Talwar, CEO of Tetrate and founding PM of Istio. “With the launch of Amazon EKS Anywhere customers can get the symmetry of experience with their container platform in hybrid environments, while Tetrate Service Bridge built on Istio and Envoy provides end to end security, observability and consistent application networking policies with a unified management plane which spans multiple environments, clusters and tenants.” – Varun Talwar, CEO, Tetrate

Learn more about EKS Anywhere enabled partners:

Aqua – Securing Kubernetes Everywhere with EKS Anywhere (EKS-A) Bare Metal
Canonical – Ubuntu on AWS
Dell (PowerFlex) – Dell PowerFlex Bare Metal with Amazon Elastic Kubernetes Service Anywhere, and We Do Mean “Anywhere!”
Equinix – Run Your Infrastructure Where You Want
Harness – Harness AWS EKS Anywhere Bare Metal
Kong – Kong Enterprise for Amazon Web Services
Lenovo – Lenovo Solution for AWS ECS-A and EKS-A on ThinkAgile VX
LG CNS – Amazon EKS(Elastic Kubernetes Service) Anywhere with LG CNS
LTI – Amazon EKS Anywhere
nClouds – Amazon EKS Anywhere
New Relic – AWS and New Relic: Better Together
Rafay – Streamlining Kubernetes operations for Amazon EKS Anywhere for bare metal environments
SentinelOne – Securing Amazon EKS Anywhere Bare Metal with SentinelOne Singularity
Spectro Cloud – Extending Palette to support Amazon EKS Anywhere for bare metal Kubernetes!
Suse Rancher – Open Source Container Management Platform
Sysdig – Adapting Security to Amazon EKS Anywhere on Bare Metal
Tech Mahindra – netOps.ai Powering Telco Networks on AWS
Tetrate – Deployment in AWS Joining EKS Anywhere and EKS Cloud
Nirmata – Nirmata Kubernetes Policy Management
Pulumi – Cloud Engineering with Kubernetes
Vertical Relevance – AWS Cloud Transformation
Wipro – WiPro AWS Business Group

Conclusion

Creating a Kubernetes cluster on bare metal has typically taken a lot of work to automate the provisioning and lifecycle management for upgrades and high availability. With EKS Anywhere you can now create highly available clusters with automated upgrades with only two configuration files and a single command. You can use the same tools and workflow to create clusters on VMware, as well as on your local workstation, with more to come. If you’d like to try other provisioners, check out the full EKS Anywhere documentation here.

EKS Anywhere on Bare Metal has been the most requested feature from our customers to manage EKS clusters on-premises and get support directly from AWS. This allows you to run fully supported clusters on your own hardware, in your own datacenters, with all of the performance, compliance, and scalability you need. We’re excited to see how you use this new capability and love to hear your feedback directly on the EKS Anywhere GitHub repo or through your account representatives.

What’s next?

If you would like to watch a live demo of a cluster creation process, check out Containers from the Couch where we have videos and demos with more information about EKS Anywhere.
We also have a live stream planned for the week of July 6th to answer your questions and demonstrate a full cluster provisioning process.
And be sure to subscribe to Containers from the Couch to get notified when new videos are posted!

Containers