Introducing Amazon EKS add-ons: lifecycle management for Kubernetes operational software

From the start, our goal with Amazon Elastic Kubernetes Service (Amazon EKS) has been to build a fully managed service that makes it easy for you to run Kubernetes on AWS without needing to be an expert in managing Kubernetes clusters. When Amazon EKS first launched, that meant a fully managed Kubernetes control plane. In time, our customers asked us to help manage the compute power required for their clusters. A year ago, we introduced this with EKS Managed Node Groups and support for AWS Fargate.

Our customers asked us again to go further, so today we want to tell you about EKS add-ons, a new feature that is a major step in our journey to provide fully managed Kubernetes clusters. EKS add-ons allows you to configure, deploy, and update the operational software, or add-ons, that provide key functionality to support your Kubernetes applications. These add-ons include critical tools for cluster networking like the Amazon VPC CNI, as well as operational software for observability, management, scaling, and security.

Starting today with the Amazon VPC CNI plugin, Amazon EKS now allows you to enable add-ons when you create a new cluster or at any time after the cluster is running. EKS will start the add-on software on the cluster and allow you to deploy new versions of the add-on with a single command. Every add-on includes the latest security patches and bug fixes, and is validated by AWS to work with Amazon EKS. This reduces the amount of work you need to do in order to start, manage, and upgrade production-ready Kubernetes clusters, which helps to keep your clusters stable and secure.

How it works

You can configure add-ons during cluster creation or anytime for an existing cluster. When you create the cluster, simply specify the version of the add-on to run.

Once the cluster is created, you can see the add-on running under the new Add-ons tab in cluster Configuration.

You may have noticed that this cluster was started with an older version of the VPC CNI plugin and an update is available. Add-ons can easily be edited at any time to change version or service account role.

Updates are started by simply choosing the version of the add-on you want to run. The optional override existing configuration checkbox will automatically resolve conflicts with any existing add-on configuration in favor of the new add-on configuration you are applying in case of any issues.

Changes are handled gracefully in your cluster, rolling out updates across related Kubernetes workloads. After a few minutes, you will see that the add-on has updated. Because add-ons are Kubernetes applications that are running on your cluster, you can inspect them directly during the update. In this example, monitoring the pods in the DaemonSet for the VPC CNI plugin (aws-node) will show these updates, and you’ll see that the image is updated to 1.7.5.

IAM Roles for service accounts

An important feature of EKS add-ons is support for AWS IAM roles for service accounts. When a cluster is first created, add-on software that requires IAM permissions (like the VPC CNI plugin) will inherit permissions from the Amazon EC2 instances (or nodes) where it runs. IAM for service accounts allows you to assign IAM permissions directly to the Kubernetes applications, instead of granting broad IAM permissions that applications share when they run on the cluster’s nodes.

Because your cluster must be active to configure IAM for service accounts, add-ons enabled during cluster creation will always start using the node role. You can change this any time after the cluster is started.

The first step is to create an OpenID Connect (OIDC) provider for a service account to use. First, go to the Overview tab for the cluster and select Details to see the OIDC provider URL.

Then use the URL you copied to create an OIDC provider in the AWS IAM console. Specify the audience sts.amazonaws.com:

Once the OIDC provider is created, you will create an IAM role that uses it. This is the IAM role that will give permissions to the add-on. For us, that will be the VPC-CNI’s ‘aws-node’ pods.

Search for and attach the managed AmazonEKS_CNI_Policy, which has the right permissions already defined for the VPC CNI plugin.

Once the role is created, the role’s trust policy must be modified to include your cluster and the Amazon VPC CNI’s service account (aws-node).

The correct format for the entity that can assume the role is "oidc.eks.<region>.amazonaws.com/id/<oidc provider ID>:sub": "system:serviceaccount:kube-system:aws-node"

The first part should be already filled out when you edit the trust relationship policy, just change :aud to :sub and add "system:serviceaccount:kube-system:aws-node".

While we just configured this role for a single cluster, you could use the same role for this application across multiple clusters by adding conditions to the trust relationship policy.

Now, with the role created, navigate back to your cluster and update the add-on to use the new IAM role. This is done by editing the add-on and selecting the role from the drop-down.

In a minute or so, the add-on is updated on your cluster to use the new IAM role.

Once this role is applied, you can remove the VPC CNI policy from any node roles you have defined. Additional detailed instructions on how to set up IAM roles for service accounts can be found in the EKS documentation.

Introducing server-side apply

So how do these add-ons get onto the cluster and how do you keep from overwriting important configuration changes, like CNI custom networking? EKS add-ons uses a new feature, available in Kubernetes version 1.18, called server-side apply. Server-side apply migrates the functionality of ‘kubectl apply’ to the Kubernetes API server and provides fine-grained control over resolving conflicts, which happen during configuration changes.

Server-side apply introduces a new configuration merging algorithm as well as field ownership tracking within the Kubernetes control plane, which allows Kubernetes to detect and resolve conflicts automatically. As Kubernetes resources may be changed over time by multiple users or other system components, this new feature now tracks which fields were changed by which entity and can use this information to resolve conflicts.

Using server-side apply, EKS add-ons is able to deploy and update operational software on the cluster, while allowing you to implement custom configuration changes to the add-on via the Kubernetes API that will not be accidentally overwritten by EKS. Server-side apply is an advanced Kubernetes feature; you can learn more about how it works on the Kubernetes blog.

Available now and coming soon

EKS add-ons is available today for all clusters running Kubernetes version 1.18 with platform version eks.3 and higher. Because server-side apply is a key feature that enables EKS add-ons and is only available starting with Kubernetes version 1.18, EKS add-ons is not compatible with older Kubernetes versions.

Today, EKS add-ons supports the Amazon VPC CNI and we plan to expand supported add-ons over the next year including kube-proxy, CoreDNS, and other common tools for observability, scaling, management, and security. If there’s a particular add-on you’d like us to support, please let us know on the AWS containers roadmap.

– Nate & Jesse