Introducing launch template and custom AMI support in Amazon EKS Managed Node Groups

Amazon Elastic Kubernetes Service (EKS) now supports EC2 Launch Templates and custom AMIs for managed node groups. When combined, these new features provide flexible configuration and customization options for Amazon EC2 instances which are managed as Kubernetes nodes by EKS. This enables you to leverage the simplicity of managed node provisioning and lifecycle management features while adhering to any level of customization, compliance, or security requirements.

Amazon EKS managed node groups

Amazon EKS is a fully managed service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. The control plane managed by EKS is scalable and highly available. It deploys control plane instances on your behalf across multiple Availability Zones, and automatically detects and replaces unhealthy control plane instances. It also provides patching for the cluster control plane, and control plane upgrades are automated.

Kubernetes clusters deploy your applications and services on nodes, and you have options to choose from with EKS. For complete control of your cluster compute resources, you can provision Amazon EC2 instances and manage their lifecycle yourself with self-managed nodes. At the opposite end of the spectrum, you can use AWS Fargate with EKS. This service provides on-demand, right-sized compute capacity for containers. With Fargate, you don’t need to provision or manage the lifecycle of nodes at all, as compute resources are allocated on demand within the Fargate service.

Using Amazon EKS managed node groups automates the provisioning and lifecycle management of Kubernetes nodes for you. This removes the undifferentiated heavy lifting of launching, configuring, patching, and upgrading instances, while still providing you some control over, and access to, your Kubernetes nodes.

Managed nodes are provisioned on your behalf as part of an EC2 Auto Scaling group that is managed for you by EKS. All resources including the EC2 instances and Auto Scaling groups run within your AWS account, and you have complete access to them. With managed nodes, you can create, update, and terminate nodes in your cluster with a single operation. EKS handles update and termination operations gracefully, automatically draining nodes to maintain availability of your deployed applications and services.

Managed node group customization with launch templates

Managed node groups were initially designed to provide the best out-of-the-box default experience. Managed nodes use the Amazon EKS-optimized Amazon Linux 2 AMI, and are provisioned with opinionated configuration which is not customizable.

Until now, managed node groups haven’t met the needs of customers that have node configuration requirements outside of the most basic elements. Specifying configuration and customization of the OS environment, installation and configuration of additional software, and other customization methods have not been possible. Additionally, there’s been no way to specify that a managed node group use an alternative to the Amazon Linux 2 EKS-optimized AMI.

Managed node groups in EKS now provide this node configuration and customization through support for EC2 Launch Templates. This enhancement provides for declarative configuration and customization for your managed node groups, including specifying your own AMI, through a well-defined and widely adopted Amazon EC2 feature.

Launch templates provide versioned, declarative configuration specification for instances within an EC2 Auto Scaling Group. The configuration is declarative and is specified via JSON. Pulling all of these features together provides good scaffolding for infrastructure-as-code practices for your EKS clusters.

Configuration elements such as instance type, instance tags, security groups, storage configuration and more can be specified in a launch template. The launch template can then be specified in the configuration for EKS managed node group create and update operations, via the EKS API, AWS CLI, CloudFormation, or the EKS Console. Support for launch templates has also been added to eksctl, which we’ll later show below.

Creating Managed Node Groups with launch templates

Starting with a simple example, you can see that it’s straightforward to create a new managed node group in EKS by specifying a launch template. Note, that you can optionally specify its version. The next example will make use of that.

Start by creating a launch template, or choose one that you’ve already created. Here, we’ll use the EKS Console.

Now, specify the launch template ID and optionally its version in the node group configuration panel. It has been updated to allow for launch template use, providing a drop-down of available launch templates.

Now, checking the status of our new node group, we can see that the launch template used is shown in the node group info console view.

Updating managed node groups with a new launch template version

Iterating from our example above, we’ll now create a new launch template version which specifies some additional configuration parameters, then use it to gracefully update this managed node group. Note that a node group can’t be updated to using a separate launch template, only from one version to another. To start using launch templates, create a new managed node group.

Using versioned declarative configuration along with automated node group updates simplifies iteration of your node configuration over time, and reflects modern application and infrastructure lifecycle workflows.

First, create a new launch template version from the initial version created above. This can be done in the EKS console, but we’ll shift to the aws CLI for this example to show that as well.

aws ec2 create-launch-template-version \
        --launch-template-name hermione-mng-custom \
        --version-description "add ssh key and security groups" \
        --source-version 1 \
        --launch-template-data '{"KeyName":"bastion","SecurityGroupIds":["sg-0e9b58499f42bcd4b","sg-0275026e71e1e7c9c"]}'

Here, we add an SSH key and specify an additional security group to allow ssh access. Note we also specify the clusterSecurityGroup. Anything not specified will carry over from the source version.

With this new launch template version, invoke an update with the launch template ID and version in its configuration.

aws eks update-nodegroup-version \
        --cluster-name hermione \
        --nodegroup-name hermione-mng-custom \
        --launch-template name=hermione-mng-custom,version=2

EKS will start gracefully cycling nodes, draining work from them as it goes, until the node group lands at the new configuration. This will take some time, but EKS will ensure that your applications and services remain available.

Support for launch templates in eksctl

Work has been underway by the team at Weave that maintains eksctl, the official open source command line utility for Amazon EKS. It now too supports launch templates in managed node specifications, for both creation and updates.

As with the examples above, a launch template version can be specified any time you’re creating a managed node group, or updating from one version to another in an existing group. Below is an example of eksctl configuration showing this.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: molly
  region: us-west-2
managedNodeGroups:
- name: molly-mng-custom
  desiredCapacity: 2
  labels: {role: worker}
  launchTemplate:
    id: lt-05bc7904721579ee2
    version: "1"

This can now be passed to eksctl and it will pass the launch template configuration as specified.

$ eksctl create nodegroup -f molly_mng_custom.yaml
[ℹ] eksctl version 0.25.0-dev+7491feef.2020-07-29T19:29:47Z
[ℹ] using region us-west-2
[ℹ] will use version 1.17 for new nodegroup(s) based on control plane version
[ℹ] 2 sequential tasks: { fix cluster compatibility, no tasks }
[ℹ] checking cluster stack for missing resources
[ℹ] cluster stack has all required resources
[ℹ] no tasks
...

What’s great about their implementation is that it uses a launch template whether you specify one or not. Under the covers, eksctl will take configuration information and create one for you, giving you a simple means of adopting this new feature and the workflows available with it. We’ll discuss using eksctl to specify customized node groups more below.

Customizing nodes with user data

The introduction of launch templates for managed node groups also unlocks the ability to define customizations for your managed nodes. Beyond specifying EC2 configuration data, you can also supply User Data in the template. This takes the form of shell scripts or cloud-init directives, providing one-shot configuration injection during instance launch.

With user data, you can specify actions such as software installation and configuration, OS configuration, enabling or disabling of system services, adding users and groups… really any type of customization you might need. Do keep in mind that your instances will need to fulfill the duties of an EKS node, so heavy modifications should always be tested thoroughly in a test cluster. If your managed node group is active and nodes have joined the cluster, you’re on a good path.

Let’s consider the following script which installs and enables the Amazon SSM Agent as an example.

#!/bin/bash
yum install -y amazon-ssm-agent
systemctl enable amazon-ssm-agent && systemctl start amazon-ssm-agent

We can encode this in base64, create a new launch template or a new launch template version, and use that to create or update a managed node group which will install the Amazon SSM Agent and enable its service. Each instance which scales in as a Kubernetes node in that group will have the SSM Agent installed and enabled.

$ cat config_install_ssm.json
{ 
  "LaunchTemplateData": {
  "EbsOptimized": false,
    "InstanceType": "t3.small",
    "KeyName": "bastion",
    "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSIvLyIKCi0tLy8KQ29udGVudC1UeXBlOiB0ZXh0L3gtc2hlbGxzY3JpcHQ7IGNoYXJzZXQ9InVzLWFzY2lpIgojIS9iaW4vYmFzaAoKeXVtIGluc3RhbGwgLXkgYW1hem9uLXNzbS1hZ2VudApzeXN0ZW1jdGwgZW5hYmxlIGFtYXpvbi1zc20tYWdlbnQgJiYgc3lzdGVtY3RsIHN0YXJ0IGFtYXpvbi1zc20tYWdlbnQKLS0vLy0tCg==",
    "SecurityGroupIds": [
      "sg-0e9b58499f42bcd4b",
      "sg-0275026e71e1e7c9c"
    ]
  }
}

aws ec2 create-launch-template \
        --launch-template-name hermione-mng-custom-ssm \
        --version-description "first version (ssh, ssm)" \
        --cli-input-json file://./config_install_ssm.json

aws eks create-nodegroup --cluster-name hermione \
        --nodegroup-name hermione-mng-custom-ssm \
        --subnets subnet-00e4fbabdbb93505c subnet-04831fd6485e95dd6 \
        --node-role 'arn:aws:iam::510431938379:role/node-instance-role' \
        --launch-template name=hermione-mng-custom-ssm

This aspect of launch template support in managed node groups provides a flexible solution for your own EKS managed node customization.

Note that user data is used by EKS to insert the EKS bootstrap invocation for your managed nodes. EKS will automatically merge this in for you, unless a custom AMI is specified. In that case, you’ll need to add that in. More on this point below.

Bringing custom AMIs to the party

By default, managed node groups are configured to use the Amazon EKS-optimized Amazon Linux 2 AMI for the specific version of your cluster. This continues to be the default behavior. Some EKS customers have deeper customization needs, and this is where custom AMI support comes into play.

Through the support of custom AMIs in managed node groups, EKS can now provision and managed the lifecycle of your Kubernetes nodes running any OS, in nearly any configuration. Again, it’s your responsibility to ensure that the required software is in place for your node to join an EKS cluster and fulfill its duties. Beyond this abstract requirement, the world is your customizable oyster. As with node customization with user data above, custom AMI configurations should be tested thoroughly. Also as noted above, if your nodes join your cluster, things are going well.

Many customers will already have EKS-capable AMIs ready to jump right in. For others, a great resource to get started with is the Amazon EKS AMI build specification. It contains resources and configuration scripts for building a custom Amazon EKS AMI based on Amazon Linux 2.

If you are in need of a more customized AMI, if you have enhanced security or compliance needs, if your organization is standardized on a different Linux distro, or for any other reason, you may find a great resource in the Amazon EKS Custom AMIs project. It provides some automation magic using make, Ansible, and Packer and provides examples of how to build custom AMIs with different operating systems such as CentOS and Debian for use with Amazon EKS.

Using custom AMIs with EKS Managed Node Groups

To use a custom AMI with an EKS managed node group, simply specify the AMI ID in your launch template. EKS will specify this AMI for the node group’s auto scaling group, and the instances will be launched with your specified AMI. Note that as with launch templates, a node group can’t be updated from the EKS-optimized AMI to a custom AMI. You will need to create a new node group to use custom AMIs.

There is one additional requirement, in that you need to supply the EKS bootstrap invocation in your launch template user data. Since using a custom AMI is a full customization use case, EKS will not merge its default bootstrap information into your user data as discussed above. You are free to fully configure your OS environment, and providing the bootstrap is now your responsibility.

Let’s go through an example here, using a RHEL 7-based AMI that’s been created for use with EKS.

First, define a simple bootstrap script for EKS nodes on a public network. See a note below about nodes on private subnets.

#!/bin/bash
set -ex
/etc/eks/bootstrap.sh hermione

Now, encode the script and create a new launch template, specifying your custom AMI.

cat config_custom_ami.json
{
  "LaunchTemplateData": {
  "EbsOptimized": false,
  "ImageId": "ami-0e00c1f097aff7fe8",
  "InstanceType": "t3.small",
  "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSIvLyIKCi0tLy8KQ29udGVudC1UeXBlOiB0ZXh0L3gtc2hlbGxzY3JpcHQ7IGNoYXJzZXQ9InVzLWFzY2lpIgojIS9iaW4vYmFzaApzZXQgLWV4Ci9ldGMvZWtzL2Jvb3RzdHJhcC5zaCBoZXJtaW9uZQoKLS0vLy0tCg==",
    "SecurityGroupIds": [
     "sg-0e9b58499f42bcd4b"
   ]
 }
}

aws ec2 create-launch-template \
        --launch-template-name hermione-mng-custom-ami \
        --version-description "first version (rhel-7)" \
        --cli-input-json file://./config_custom_ami.json

Now create a new managed node group.

aws eks create-nodegroup --cluster-name hermione \
        --nodegroup-name hermione-mng-custom-ami-rhel-7 \
        --subnets subnet-00e4fbabdbb93505c subnet-04831fd6485e95dd6 \
        --node-role 'arn:aws:iam::510431938379:role/node-instance-role' \
        --launch-template name=hermione-mng-custom-ami,version=1

Once the node group creation is complete, verify your managed nodes were launched from your custom AMI. In my example, I see the nodes are running RHEL 7.8, along with 2 other nodes from my node group using the EKS-optimized Amazon Linux 2 AMI.

$ kubectl describe nodes | grep OS
      OS Image: Amazon Linux 2
      OS Image: Red Hat Enterprise Linux Server 7.8 (Maipo)
      OS Image: Red Hat Enterprise Linux Server 7.8 (Maipo)
      OS Image: Amazon Linux 2

As mentioned above, if your nodes are configured to launch on private subnets you’ll need to include your cluster’s specific cluster endpoint and certificate authority data. These can be retrieved via the EKS DescribeCluster API and added to your bootstrap script. Without this data, your nodes will be unable to join the cluster. You can retrieve this data through the Details page for your cluster in the EKS Console, or using the API or AWS CLI.

aws eks describe-cluster --name hermione \
        --query 'cluster.[endpoint,certificateAuthority]'

Now add this to your bootstrap script, either in the console or using the CLI workflow shown above.

#!/bin/bash
set -ex
B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSRTR4cWRPTmFBM1NLaGt3b3QwK0thUFlTcHRjWApBT3lZMUl6ME9mZTA4dXJlNWVmM0IyTDdpQVBkQTJBWUhMbURienk5T1l4S2g0SDM3TjgwOWl4OENjMFh0ckhyClZjRE1QVis3ZUVwWEhERDEwS0R1aEpLRG43cUR4NVhFTHRxU2txTFoxZmdoa2NNSWFFVTVkSjIvanBzaE5UUFYKT0puUWw3Z1Y1TGNSZlpJeStRcWZUcjQxOEdQbWorMGdWdVJJYm40MW5ySEVab3hzaWlZS2JoT1dtYzFVSGxhNwpXSFJkK3dsYlE3K241VmRYczhXQ01vUUFtNWZmVktVN1lqdU5Uak9YMFlkenZFMzFtTURsV2ZuVStJVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
API_SERVER_URL=https://504E9A153B3F63259E9F92706.sk1.us-west-2.clusters.wesley.amazonaws.com
/etc/eks/bootstrap.sh hermione --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL

Using familiar workflows with eksctl

As mentioned above, eksctl’s support for launch templates goes so far as to translate your specified configuration into the creation and use of a launch template on your behalf. This extends to specifying a custom AMI as well as the required user data. You can simply specify your AMI and the bootstrap script in your eksctl configuration. This is a great way to start using this new feature, and is transparent to existing eksctl workflows.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ginny
  region: us-west-2
  version: '1.17'
managedNodeGroups:
- name: ginny-mng-custom-ami
  instanceType: t3.small
  desiredCapacity: 2
  labels: {role: worker}
  ami: ami-0030109261aa0205b
  ssh:
  publicKeyName: bastion
  preBootstrapCommands:
  - kubelet --version > /etc/eks/test-preBootstrapCommands
  overrideBootstrapCommand: |
    #!/bin/bash
    set -ex
    /etc/eks/bootstrap.sh ginny --kubelet-extra-args "--node-labels=alpha.eksctl.io/cluster-name=ginny,alpha.eksctl.io/nodegroup-name=ginny-mng-custom-ami,eks.amazonaws.com/nodegroup=ginny-mng-custom-ami,eks.amazonaws.com/nodegroup-image=ami-0030109261aa0205b"

This is a great approach to integrating this new feature with existing usage patterns, and well done to the maintainers. For full details on using eksctl with launch templates and custom AMIs for managed node groups, see the project site.

Working with launch templates for Managed Node Groups

As you can see above, launch template and custom AMI support for EKS managed node groups provide you with a flexible array of options in configuration and customization for your managed nodes. In this section, we’ll provide some details on working with this new feature. The best reference to get the latest guidance on this feature is the Amazon EKS User Guide section on Launch Templates.

When working with launch templates, if a version is not specified EKS will use the default version. The default version can be seen in EC2 API or CLI when describing a launch template, or in the console.

Remember that managed node groups can’t be migrated to using launch templates through updates. To start using launch templates, you’ll create a new node group. Similarly, if a managed node group is already using the Amazon EKS-optimized AMI, you’ll need to create a new node group to start using custom AMIs that you specify.

When using launch templates to create or update your EKS managed node groups, some configuration must be specified in the launch template while other parameters must be specified in the managed node group configuration.

When using a launch template, if any of the following parameters are specified in the node group configuration, your create or update request will fail. Specify these in your launch template:

Instance type
Disk size
Remote access configuration
- EC2 SSH key
- Source Security Groups

Further, if any of the following parameters are specified in your launch template, your create or update request will fail. Specify these in your node group configuration:

Instance profile
Network subnets

Specific to working with custom AMIs, some node group configuration elements are used by EKS to select which Amazon EKS-optimized AMI to launch your instances with. These aren’t needed when an AMI is specified, so when using a custom AMI, do not specify:

AMI Type
Release version
Version

Finally, while we feel a majority of use cases will be met, some parameters are not supported for specification at this time. These include:

Instance market options (EC2 Spot support will be added in a future release)
Shutdown behavior (Amazon EKS must control instance lifecycles)

It’s worth repeating here, the best reference for supported configuration elements, using launch templates, and other EKS features is the Amazon EKS User Guide.

Wrapping up

We hope that this tour of the new managed node customization features has given many of our existing customers a new path to leveraging the benefits of managed node groups in EKS. There are many benefits to using managed nodes, and if you have customization and configuration needs beyond the most basic, we hope that this unblocks your exploration of these benefits.

If you’re a new EKS user, using managed node groups is a great way to take advantage of the service’s ability to not only remove the undifferentiated heavy lifting of building and keeping your Kubernetes clusters running, but also in the provisioning and lifecycle management of your nodes.

As always, we build these new features based on customer guidance and input, and anticipating your needs. To see what’s coming next, to add your story to an existing request, or to make new suggestions for features and enhancements, please visit the AWS Containers Roadmap on GitHub.