Amazon EBS CSI driver is now generally available in Amazon EKS add-ons
To provide workloads with optional persistent storage, Kubernetes implements volume lifecycle operations and supports various types of storage for use with these operations. Currently, storage provider–specific code is kept in the Kubernetes project source code, which is referred to as in-tree. This code is complex to maintain and release, and is tied to the Kubernetes release cycle as part of the in-tree code.
To decouple the lifecycle of storage implementations from the Kubernetes project itself, the Container Storage Interface (CSI) was created. This is a standard for implementing storage-specific solutions for container-based applications, exposing various block and file storage systems to workloads on container orchestration systems, such as Kubernetes. Kubernetes has been moving from its initial in-tree storage plugin system to the CSI model. To prepare for this move, a CSI migration feature was introduced in Kubernetes 1.14.
Amazon Elastic Kubernetes Service (EKS) support for the Amazon Elastic Block Store (EBS) CSI driver was announced in September 2019 and has gained broad adoption since then. This CSI driver enables container orchestrators (such as Kubernetes) to manage the lifecycle of Amazon EBS volumes. When used with Amazon EKS, this driver is installed on a cluster and storage resources can be specified. To facilitate moving from use of the Amazon EBS in-tree plugin to the Amazon EBS CSI driver, a dedicated AWS specific feature, CSIMigrationAWS, was introduced. Upstream Kubernetes v1.23 now enables this feature by default. This requires that the Amazon EBS CSI driver is installed in a Kubernetes cluster.
Amazon EKS add-ons provide installation and management of a curated set of add-ons for Amazon EKS clusters. In December 2021, AWS introduced a preview of an Amazon EBS CSI EKS add-on to prepare for CSIMigrationAWS. This add-on simplifies the installing, managing, and patching of the Amazon EBS CSI driver. Today we are happy to share the general availability of the Amazon EBS CSI EKS add-on. This release contains added functionality, such as volume resizing and snapshotting for Linux, Windows support including snapshotting but without resizing, ARM64 support, and an AWS managed IAM policy.
The Amazon EBS CSI driver can now be installed, managed, and updated directly through the Amazon EKS console, AWS Command Line Interface (CLI), and Amazon EKS API. You can see available add-ons and compatible versions in the Amazon EKS API, select the version of the add-on you want to run on your cluster, and configure key settings such as the IAM role used by the add-on when it runs.
To learn more and get started, see the Amazon EKS add-ons documentation.
In the remainder of this post, we provide some background on CSI and the ongoing CSI Migration project within Kubernetes. We then demonstrate creating an Amazon EBS CSI EKS add-on and using it in the context of a Kubernetes workload.
Container Storage Interface and Kubernetes
Kubernetes currently supports Amazon Elastic Block Store (EBS) with the in-tree kubernetes.io/aws-ebs plugin, which has been deprecated since Kubernetes 1.17. We recommend moving to the Amazon EBS CSI Driver, using the ebs.csi.aws.com provisioner. AWS maintains open-source CSI drivers for Amazon EBS, as well as Amazon Elastic File System (EFS) and FSx for Lustre, for seamless integration with Kubernetes volume operations and their respective storage services. These CSI drivers are maintained separately from the upstream project by implementing the CSI interface, allowing independent maintenance and feature development.
Kubernetes CSI migration
The migration from the in-tree plugins to the new CSI model posed an initial problem of API compatibility because many workloads were already specified using the in-tree storage providers, and moving to the new CSI model required changes to these workloads. To solve this, the CSI migration effort was introduced. This is a Kubernetes feature that translates in-tree APIs to the CSI API, allowing those APIs to be delegated to CSI drivers over time. The CSI migration feature gate CSIMigration has been in beta since Kubernetes 1.17 and is enabled by default. This allows the replacement of all in-tree volume plugins in Kubernetes with solution-specific CSI drivers using the provider-specific migration feature gates such as CSIMigrationAWS. The CSIMigrationAWS feature gate will be enabled by default in upstream Kubernetes and Amazon EKS Kubernetes version 1.23.
In practice, CSI migration means that if your application workloads use the in-tree kubernetes.io/aws-ebs provisioner, they can still do so in their specifications, and the operations are automatically routed to the Amazon EBS CSI driver. This migration feature is best used as a stop gap before migrating workloads to natively specify the CSI driver because new features implemented in CSI won’t be available to the in-tree-style specifications. For example, a PersistentVolume (PV) migrated from the in-tree plugin to the CSI driver can’t specify new features and attributes of the CSI driver solution. Therefore, we recommend defining a new CSI-based StorageClass (SC) and explicitly migrating workloads as described in the “Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes” blog post.
Creating Amazon EKS add-ons
To get started with Amazon EKS add-ons, you need to opt in by creating one or more add-ons in a supported cluster. Because the implementation is tied to a specific Kubernetes feature, the minimum Kubernetes version supported is 1.18. This is due to the use of the Kubernetes 1.18 Server-side Apply feature to implement the update functionality for add-ons. This Kubernetes feature combines configuration field ownership tracking as well as a new configuration merging algorithm within the context of the Kubernetes api-server.
All Amazon EKS add-on operations are supported through the interfaces supported by Amazon EKS. This includes the AWS console, the AWS CLI, and the eksctl utility. You can also use supported infrastructure-as-code tools, including the configuration file path for eksctl, AWS CloudFormation, the AWS provider for Terraform, and the AWS Cloud Development Kit (CDK). This can be done for existing EKS clusters or when you create a new cluster.
Working with the Amazon EBS CSI Driver EKS add-on
We’ll now shift to working with the Amazon EBS CSI driver EKS add-on. You can create and manage add-ons with your choice of interface. In the example workflows shown later, we use eksctl, the AWS CLI, and Terraform. You need an Amazon EKS cluster for this walkthrough. If you don’t already have one provisioned, consider using eksctl to create one.
For instructions on installing the latest release of eksctl, see Installing eksctl in the Amazon EKS User Guide.
The Amazon EBS CSI managed add-on is called aws-ebs-csi-driver. The following command lists all Amazon EKS cluster and platform versions that the add-on is available for:
You can use the
--kubernetes-version option to determine an add-on version for a specific Kubernetes version. For example, the following command returns the versions that are compatible with an Amazon EKS cluster running Kubernetes version 1.21:
Note: The output shows that using the aws-ebs-csi-driver add-on version v1.5.2-eksbuild.1 for EKS version 1.21 requires EKS platform version eks.5 or higher, whereas the v1.4.0-eksbuild.preview version requires EKS platform version eks.3 or higher.
To use the snapshot functionality of the Amazon EBS CSI driver, you must install the external snapshotter before installing the add-on. The external snapshotter is provided by the Kubernetes CSI Special Interest Group (SIG) and has components that must be installed in the following order:
- CustomResourceDefinition (CRD) for volumesnapshotclasses, volumesnapshots, and volumesnapshotcontents
- RBAC (ClusterRole, ClusterRoleBinding, and so on)
- controller deployment
The EBS CSI driver includes the csi-snapshotter sidecar container to interact with the external snapshotter. For more information, see the upstream GitHub repository.
The EBS CSI add-on needs Amazon EBS volume–related IAM permissions. To use the least IAM privileges for the add-on, we use IAM roles for service accounts (IRSA). For more information, see Configuring the Amazon EBS CSI plugin to use IAM roles for service accounts in the Amazon EKS User Guide.
Note: IRSA is used only for the EBS CSI add-on, not for the external snapshotter.
You can download a sample IAM policy document or use the AWS managed IAM policy AmazonEBSCSIDriverPolicy, which we use in this blog post.
We use the following eksctl config YAML to create a new cluster with the add-on using the ARN of the AWS managed IAM policy. Note that withOIDC is set to true to support IRSA. Because the add-on is installed on the data plane, we create a managed node group as well. In our demo cluster, we’re using the default Amazon Linux 2 nodes.
In the following example, we create the cluster in the eu-west-1 region. Set the following environment variables to shorten the upcoming CLI commands:
We can check the add-on for an ACTIVE status with eksctl:
Or we can check the add-on for an ACTIVE status with Amazon EKS API:
eksctl sets our current Kubernetes context to the newly created cluster:
For existing clusters that match the EBS CSI add-on prerequisite of platform version eks.5+, we can create the add-on by using the
eksctl create addon command and specifying the AWS managed IAM policy. The corresponding IRSA setup and ServiceAccount annotation is added automatically.
Note that you may need to enable OIDC if it’s not already done for your cluster.
Then, you can create the add-on.
Many customers are using Terraform to build their infrastructure. The Terraform AWS provider contains the aws_eks_addon resource to create EKS managed add-ons as well. Here is a code snippet:
The Amazon EBS CSI driver runs in the kube-system namespace and consists of two components—a controller that is deployed as a Deployment, and the CSI nodes that are rolled out as DaemonSets in your EKS cluster. The components are shown in the following kubectl command output. Note that even though we have no Windows-capable nodes in the demo cluster, a DaemonSet is installed and ready to support them if added at a later time. For a detailed explanation, see the official EBS CSI documentation.
Each ebs-csi-controller pod contains six sidecar containers, as listed here (The actual pod name is different in your environment):
Note: The csi-snapshotter and csi-resizer sidecars and the respective capabilities weren’t supported as part of the Amazon EBS CSI add-on preview but are introduced in the General Availability (GA) version.
By default, every node is capable of using CSI-based Amazon EBS volumes:
IRSA is used only by the CSI controller component when using the ServiceAccount (SA) ebs-csi-controller-sa. The SA is automatically annotated with the IAM role created by either eksctl or Terraform. The role name is generated and will appear differently in your cluster.
Every Amazon EKS cluster currently comes with an in-tree plugin–based StorageClass (SC):
To use the external EBS CSI driver, we need to create a new StorageClass based upon it:
Note: Because the GA version supports the volume resizing feature, we set the allowVolumeExpansion property to true.
The provisioner column shows that this StorageClass is now based on the Amazon EBS CSI add-on driver:
Note: The in-tree provisioner–based StorageClass gp2 is still available. It’s usable and coexists with the EBS CSI–based gp3 StorageClass. No migration of in-tree to external CSI provisioner occurs when using the Amazon EKS managed add-on Amazon EBS CSI.
Now we create a PersistentVolumeClaim (PVC), which will be used by a pod to show the dynamic provisioning capability of a gp3-based StorageClass:
The PVC is still in pending status because the gp3 StorageClass uses a volumeBindingMode of WaitForFirstConsumer. This attribute makes sure that the PersistentVolume (PV) and Pod will be provisioned in the same AWS availability zone (AZ). We can now deploy a Pod to create and bind an underlying PV.
Once the pod is running (it can take a few seconds), the PVC goes into Bound status and an underlying PV is provisioned:
Keep in mind that the underlying EBS volume of a PV is bound to an AZ. If you want to stop and restart the pod later on, make sure to add nodeSelector or nodeAffinity to the YAML specification to run the pod on a node that is part of the same AZ as the EBS volume.
Additional examples such as resizing, snapshotting, and more can be found in the official Amazon EBS CSI driver GitHub repository.
We delete the pod and the PVC. Doing this deletes the underlying PV as well:
Then we delete the Amazon EBS CSI add-on:
Finally, we delete the ebs-demo-cluster cluster:
This blog post describes how Amazon EKS add-ons can simplify using the Amazon EBS CSI driver to provision and manage volumes for your persistent storage needs in Kubernetes. For more information on using the Amazon EBS CSI driver, see the open-source project and the Amazon EKS User Guide.
As mentioned above, we recommend moving to the Amazon EBS CSI Driver now and specifying the ebs.csi.aws.com provisioner in your storage specifications. For Amazon EKS Kubernetes 1.23, CSI migration will be enabled by default and clusters will need to have the EBS CSI driver installed regardless of specifications. For more information on defining a new CSI-based StorageClass (SC) and explicitly migrating workloads refer to the Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes blog.
Amazon EKS provides you with a managed control plane, options for managing the data plane with managed node groups, and managed cluster add-ons for critical components such as the Amazon VPC Container Network Interface (CNI), CoreDNS, kube-proxy, and now the Amazon EBS CSI driver. We’re planning to release more add-ons to simplify Amazon EKS cluster operations with additional operational software in the coming months.
To check on upcoming features or to suggest features, see the AWS Containers Roadmap on GitHub. If you have questions or suggestions, please leave a comment.