Use your on-premises infrastructure in Amazon EKS clusters with Amazon EKS Hybrid Nodes

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Hybrid Nodes, a new feature that you can use to attach your on-premises and edge infrastructure as nodes to EKS clusters in the cloud.

With Amazon EKS Hybrid Nodes, you can unify Kubernetes management across cloud and on-premises environments and take advantage of the scale and availability of Amazon EKS in all the places your applications need to run. You can use your existing on-premises hardware, while offloading the responsibility for managing Kubernetes control planes to EKS and conserving on-premises capacity for your workloads. Using Amazon EKS Hybrid Nodes, you can adopt consistent operational practices and tooling across your cloud and on-premises environments.

Amazon EKS Hybrid Nodes expands our support for hybrid Kubernetes deployments, adding to Amazon EKS on AWS Outposts and Amazon EKS Anywhere, which we introduced previously. You can compare how Kubernetes and hardware components are managed with each of the EKS hybrid deployment options.

When you use Amazon EKS Hybrid Nodes to attach your on-premises and edge infrastructure to EKS clusters, you can use other Amazon EKS features and integrations, including Amazon EKS add-ons, Pod Identities, cluster access entries, cluster insights, and extended Kubernetes version support. Amazon EKS Hybrid Nodes inherently integrates with AWS services including AWS Systems Manager, AWS IAM Roles Anywhere, Amazon Managed Service for Prometheus, Amazon CloudWatch, and Amazon GuardDuty for centralized monitoring, logging, and identity management.

Get started with Amazon EKS Hybrid Nodes
Here are steps to use Amazon EKS Hybrid Nodes. First, create an EKS cluster and specify your on-premises node and pod subnets. After setting up network connectivity and AWS Identity and Access Management (AWS IAM) permissions for your on-premises environment, run the Amazon EKS Hybrid Nodes CLI (nodeadm) on each host that will join the cluster. When hybrid nodes join your cluster, required networking components, such as kube-proxy and CoreDNS, are automatically installed. Before your hybrid nodes become ready to serve applications, you must install a compatible Container Network Interface (CNI) driver. The Cilium and Calico CNI drivers are supported for use with Amazon EKS Hybrid Nodes.

1. Prerequisites

You must have certain prerequisites in place before your on-premises infrastructure can join your EKS cluster as hybrid nodes, including the following:

• Hybrid network connectivity from your on-premises environment to and from AWS using with AWS Site-to-Site VPN, AWS Direct Connect, or another virtual private network (VPN) solution
• A virtual private cloud (VPC) with routes in its routing table for your on-premises node and, optionally, pod networks, with your virtual private gateway (VGW) or transit gateway (TGW) as the target
• Infrastructure in the form of physical or virtual machines
• Operating system that is compatible with hybrid nodes
• Either AWS IAM Roles Anywhere or AWS Systems Manager set up to authenticate your hybrid nodes with the control plane
• An EKS cluster IAM role and an EKS Hybrid Nodes IAM role

You can use Amazon Linux 2023, Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04, or Red Hat Enterprise Linux (RHEL) 8 and 9 as the node operating system for your hybrid nodes. AWS supports the hybrid nodes integration with these operating systems but doesn’t provide support for the operating systems themselves. You’re responsible for operating system provisioning and management.

To learn more, visit Prerequisites for EKS Hybrid Nodes in the Amazon EKS User Guide.

2. Create EKS cluster and enable hybrid nodes

Go to the Amazon EKS console and start to create your EKS cluster. In the Step 2 Specify networking screen, turn on Specify the CIDR blocks for your on-premises environments that you will use for hybrid nodes in the Configure remote networks to enable hybrid nodes option.

The Classless Inter-Domain Routing (CIDRs) of remote nodes and pods need to be RFC-1918 IPv4 IPv4 addresses, and they can’t overlap with the VPC CIDR or the EKS cluster Kubernetes service CIDR. Additionally, the remote node CIDR and the remote pod CIDR can’t overlap. Specifying a pod CIDR block is required if you will run webhooks on your nodes or if your CNI doesn’t use NAT for pod addresses as pod traffic leaves your nodes.

You can also create an EKS cluster using AWS Comand Line Interface (AWS CLI), eksctl, and AWS CloudFormation. To enable your cluster for Amazon EKS Hybrid Nodes, use the remote-network-config flag to specify your remote node and, optionally, your remote pod CIDR blocks.

$ aws eks create-cluster --name channy-hybrid-cluster --region=us-east-1 \
    --role-arn arn:aws:iam::012345678910:role/eks-cluster-role \
    --resources-vpc-config subnetIds=subnet-1234a11a,subnet-5678b11b \
    --remote-network-config \
{"remoteNodeNetworks":[{"cidrs":["10.80.0.0/16"]}],"remotePodNetworks":[{"cidrs":["10.85.0.0/16"]}]}}

Your cluster must be configured with API or API_AND_CONFIG_MAP cluster access authentication modes. Create an Amazon EKS access entry for your EKS Hybrid Nodes IAM role to enable nodes to join the cluster.

$ aws eks create-access-entry \
  --cluster-name my-hybrid-cluster \
  --principal-arn arn:aws:iam::012345678910:role/eksHybridNodesRole \ 
  --type HYBRID_LINUX

Amazon EKS Hybrid Nodes use temporary IAM credentials provisioned by AWS Systems Manager hybrid activations or AWS IAM Roles Anywhere to authenticate with the EKS cluster. Before connecting your on-premises nodes, you must either create an AWS Systems Manager hybrid activation or add certificates and keys to your nodes for use with AWS IAM Roles Anywhere. To learn more, visit Prepare credentials for EKS Hybrid Nodes in the Amazon EKS User Guide.

3. Connect your hybrid nodes to the EKS cluster

You’re now ready to connect Amazon EKS Hybrid Nodes to your EKS cluster. You can use the Amazon EKS Hybrid Nodes CLI (nodeadm) to simplify the installation, configuration, and registration of your hosts as hybrid nodes. nodeadm automatically installs the required AWS Systems Manager or IAM Roles Anywhere components when you run the nodeadm install command.

You can run the nodeadm install process on each running host, or you can run nodeadm install as part of your operating system build pipelines to produce an image with the components needed to join your host to an EKS cluster.

$ nodeadm install 1.31 --credential-provider <ssm, iam-ra>
{"level":"info","ts":...,"caller":"...","msg":"Loading configuration","configSource":"file://nodeConfig.yaml"}
{"level":"info","ts":...,"caller":"...","msg":"Validating configuration"}
{"level":"info","ts":...,"caller":"...","msg":"Validating Kubernetes version","kubernetes version":"1.30"}
{"level":"info","ts":...,"caller":"...","msg":"Using Kubernetes version","kubernetes version":"1.30.0"}
{"level":"info","ts":...,"caller":"...","msg":"Installing SSM agent installer..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubelet..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubectl..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing cni-plugins..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing image credential provider..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing IAM authenticator..."}
{"level":"info","ts":...,"caller":"...","msg":"Finishing up install..."}

Create a nodeConfig.yaml file on each host that contains the information required to connect to your EKS cluster. Here is an example nodeConfig.yaml that uses AWS Systems Manager hybrid activations.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
metadata:
  name: hybrid-node
spec:
  cluster:
    name: my-cluster
    region: us-east-1
  hybrid:
    ssm:
      activationCode: <activation-code>
      activationId: <activation-id>

Now, run nodeadm on each host.

$ nodeadm init -c file://nodeConfig.yaml

If the preceding command is completed successfully, your hybrid node has joined your EKS cluster. You can verify this in the Amazon EKS console or with the kubectl get nodes command. Before your hybrid nodes have status as Ready, you must install a compatible CNI. To learn more, visit Install CNI for EKS Hybrid Nodes in the Amazon EKS User Guide.

4. View and manage connected your hybrid nodes in EKS console

Now that the nodes are ready, you can view your hybrid nodes and the resources running on them in the EKS console.

You’re responsible for managing your hybrid nodes and updating the software they run. You can update to the latest version of the Amazon EKS Hybrid Nodes CLI to pull in the latest fixes and updates and upgrade Kubernetes versions. To learn more, visit Upgrade EKS Hybrid Nodes in the Amazon EKS User Guide.

Now available
Amazon EKS Hybrid Nodes is now available in all AWS Regions except the AWS GovCloud (US) Regions and the China Regions.

There are no upfront commitments or minimum fees, and you pay for the hourly usage of your EKS cluster and EKS Hybrid Nodes as you use them. EKS clusters with your hybrid nodes have the same per cluster per hour cost as EKS clusters with nodes running in AWS Cloud for both standard and extended support. Additionally, EKS clusters with your hybrid nodes incur an hourly fee per hybrid node vCPU. To learn more, visit the Amazon EKS pricing page.

Give EKS Hybrid Nodes a try in the Amazon EKS console. To learn more, visit the EKS Hybrid Nodes documentation and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

— Channy