Amazon EBS CSI driver is now generally available in Amazon EKS add-ons

Introduction

To provide workloads with optional persistent storage, Kubernetes implements volume lifecycle operations and supports various types of storage for use with these operations. Currently, storage provider–specific code is kept in the Kubernetes project source code, which is referred to as in-tree. This code is complex to maintain and release, and is tied to the Kubernetes release cycle as part of the in-tree code.

To decouple the lifecycle of storage implementations from the Kubernetes project itself, the Container Storage Interface (CSI) was created. This is a standard for implementing storage-specific solutions for container-based applications, exposing various block and file storage systems to workloads on container orchestration systems, such as Kubernetes. Kubernetes has been moving from its initial in-tree storage plugin system to the CSI model. To prepare for this move, a CSI migration feature was introduced in Kubernetes 1.14.

Amazon Elastic Kubernetes Service (EKS) support for the Amazon Elastic Block Store (EBS) CSI driver was announced in September 2019 and has gained broad adoption since then. This CSI driver enables container orchestrators (such as Kubernetes) to manage the lifecycle of Amazon EBS volumes. When used with Amazon EKS, this driver is installed on a cluster and storage resources can be specified. To facilitate moving from use of the Amazon EBS in-tree plugin to the Amazon EBS CSI driver, a dedicated AWS specific feature, CSIMigrationAWS, was introduced. Upstream Kubernetes v1.23 now enables this feature by default. This requires that the Amazon EBS CSI driver is installed in a Kubernetes cluster.

Amazon EKS add-ons provide installation and management of a curated set of add-ons for Amazon EKS clusters. In December 2021, AWS introduced a preview of an Amazon EBS CSI EKS add-on to prepare for CSIMigrationAWS. This add-on simplifies the installing, managing, and patching of the Amazon EBS CSI driver. Today we are happy to share the general availability of the Amazon EBS CSI EKS add-on. This release contains added functionality, such as volume resizing and snapshotting for Linux, Windows support including snapshotting but without resizing, ARM64 support, and an AWS managed IAM policy.

The Amazon EBS CSI driver can now be installed, managed, and updated directly through the Amazon EKS console, AWS Command Line Interface (CLI), and Amazon EKS API. You can see available add-ons and compatible versions in the Amazon EKS API, select the version of the add-on you want to run on your cluster, and configure key settings such as the IAM role used by the add-on when it runs.

To learn more and get started, see the Amazon EKS add-ons documentation.

In the remainder of this post, we provide some background on CSI and the ongoing CSI Migration project within Kubernetes. We then demonstrate creating an Amazon EBS CSI EKS add-on and using it in the context of a Kubernetes workload.

Container Storage Interface and Kubernetes

Kubernetes currently supports Amazon Elastic Block Store (EBS) with the in-tree kubernetes.io/aws-ebs plugin, which has been deprecated since Kubernetes 1.17. We recommend moving to the Amazon EBS CSI Driver, using the ebs.csi.aws.com provisioner. AWS maintains open-source CSI drivers for Amazon EBS, as well as Amazon Elastic File System (EFS) and FSx for Lustre, for seamless integration with Kubernetes volume operations and their respective storage services. These CSI drivers are maintained separately from the upstream project by implementing the CSI interface, allowing independent maintenance and feature development.

Kubernetes CSI migration

The migration from the in-tree plugins to the new CSI model posed an initial problem of API compatibility because many workloads were already specified using the in-tree storage providers, and moving to the new CSI model required changes to these workloads. To solve this, the CSI migration effort was introduced. This is a Kubernetes feature that translates in-tree APIs to the CSI API, allowing those APIs to be delegated to CSI drivers over time. The CSI migration feature gate CSIMigration has been in beta since Kubernetes 1.17 and is enabled by default. This allows the replacement of all in-tree volume plugins in Kubernetes with solution-specific CSI drivers using the provider-specific migration feature gates such as CSIMigrationAWS. The CSIMigrationAWS feature gate will be enabled by default in upstream Kubernetes and Amazon EKS Kubernetes version 1.23.

In practice, CSI migration means that if your application workloads use the in-tree kubernetes.io/aws-ebs provisioner, they can still do so in their specifications, and the operations are automatically routed to the Amazon EBS CSI driver. This migration feature is best used as a stop gap before migrating workloads to natively specify the CSI driver because new features implemented in CSI won’t be available to the in-tree-style specifications. For example, a PersistentVolume (PV) migrated from the in-tree plugin to the CSI driver can’t specify new features and attributes of the CSI driver solution. Therefore, we recommend defining a new CSI-based StorageClass (SC) and explicitly migrating workloads as described in the “Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes” blog post.

Creating Amazon EKS add-ons

To get started with Amazon EKS add-ons, you need to opt in by creating one or more add-ons in a supported cluster. Because the implementation is tied to a specific Kubernetes feature, the minimum Kubernetes version supported is 1.18. This is due to the use of the Kubernetes 1.18 Server-side Apply feature to implement the update functionality for add-ons. This Kubernetes feature combines configuration field ownership tracking as well as a new configuration merging algorithm within the context of the Kubernetes api-server.

All Amazon EKS add-on operations are supported through the interfaces supported by Amazon EKS. This includes the AWS console, the AWS CLI, and the eksctl utility. You can also use supported infrastructure-as-code tools, including the configuration file path for eksctl, AWS CloudFormation, the AWS provider for Terraform, and the AWS Cloud Development Kit (CDK). This can be done for existing EKS clusters or when you create a new cluster.

Working with the Amazon EBS CSI Driver EKS add-on

We’ll now shift to working with the Amazon EBS CSI driver EKS add-on. You can create and manage add-ons with your choice of interface. In the example workflows shown later, we use eksctl, the AWS CLI, and Terraform. You need an Amazon EKS cluster for this walkthrough. If you don’t already have one provisioned, consider using eksctl to create one.

For instructions on installing the latest release of eksctl, see Installing eksctl in the Amazon EKS User Guide.

The Amazon EBS CSI managed add-on is called aws-ebs-csi-driver. The following command lists all Amazon EKS cluster and platform versions that the add-on is available for:

$ aws eks describe-addon-versions --addon-name aws-ebs-csi-driver

You can use the --kubernetes-version option to determine an add-on version for a specific Kubernetes version. For example, the following command returns the versions that are compatible with an Amazon EKS cluster running Kubernetes version 1.21:

$ aws eks describe-addon-versions --addon-name aws-ebs-csi-driver --kubernetes-version 1.21
{
    "addons": [
        {
            "addonName": "aws-ebs-csi-driver",
            "type": "storage",
            "addonVersions": [
                {
                    "addonVersion": "v1.5.2-eksbuild.1",
                    "architecture": [
                        "amd64",
                        "arm64" 
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.21",
                            "platformVersions": [
                                "eks.5+"
                            ],
                            "defaultVersion": true
                        }
                    ]
                },
                {
                    "addonVersion": "v1.4.0-eksbuild.preview",
                    "architecture": [
                        "amd64"
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.21",
                            "platformVersions": [
                                "eks.3+"
                            ],
                            "defaultVersion": false
                        }
                    ]
                }               
            ]
        }
    ]
}

Note: The output shows that using the aws-ebs-csi-driver add-on version v1.5.2-eksbuild.1 for EKS version 1.21 requires EKS platform version eks.5 or higher, whereas the v1.4.0-eksbuild.preview version requires EKS platform version eks.3 or higher.

To use the snapshot functionality of the Amazon EBS CSI driver, you must install the external snapshotter before installing the add-on. The external snapshotter is provided by the Kubernetes CSI Special Interest Group (SIG) and has components that must be installed in the following order:

CustomResourceDefinition (CRD) for volumesnapshotclasses, volumesnapshots, and volumesnapshotcontents
RBAC (ClusterRole, ClusterRoleBinding, and so on)
controller deployment

The EBS CSI driver includes the csi-snapshotter sidecar container to interact with the external snapshotter. For more information, see the upstream GitHub repository.

The EBS CSI add-on needs Amazon EBS volume–related IAM permissions. To use the least IAM privileges for the add-on, we use IAM roles for service accounts (IRSA). For more information, see Configuring the Amazon EBS CSI plugin to use IAM roles for service accounts in the Amazon EKS User Guide.

Note: IRSA is used only for the EBS CSI add-on, not for the external snapshotter.

You can download a sample IAM policy document or use the AWS managed IAM policy AmazonEBSCSIDriverPolicy, which we use in this blog post.

We use the following eksctl config YAML to create a new cluster with the add-on using the ARN of the AWS managed IAM policy. Note that withOIDC is set to true to support IRSA. Because the add-on is installed on the data plane, we create a managed node group as well. In our demo cluster, we’re using the default Amazon Linux 2 nodes.

In the following example, we create the cluster in the eu-west-1 region. Set the following environment variables to shorten the upcoming CLI commands:

$ export AWS_REGION=eu-west-1; export AWS_DEFAULT_REGION=eu-west-1

$ cat config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ebs-demo-cluster
  region: eu-west-1
  version: "1.21"
 
iam:
  withOIDC: true
 
addons:
- name: aws-ebs-csi-driver
  version: v1.5.2-eksbuild.1
  attachPolicyARNs:
  - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
 
managedNodeGroups:
  - name: managed-ng-1
    instanceType: t2.large
    privateNetworking: true
    minSize: 2
    maxSize: 3

$ eksctl create cluster -f config.yaml

$ eksctl get cluster
2021-12-14 18:00:59 [ℹ]  eksctl version 0.76.0
2021-12-14 18:00:59 [ℹ]  using region eu-west-1
NAME                    REGION          EKSCTL CREATED
ebs-demo-cluster        eu-west-1       True

We can check the add-on for an ACTIVE status with eksctl:

$ eksctl get addon --name aws-ebs-csi-driver --cluster ebs-demo-cluster
…
NAME                    VERSION                 STATUS  ISSUES  IAMROLE
                                UPDATE AVAILABLE
aws-ebs-csi-driver      v1.5.2-eksbuild.1 ACTIVE  0       arn:aws:iam::<accountId>:role/eksctl-ebs-demo-cluster-addon-aws-ebs-csi-dr-Role1-TX3SPPGLTRLZ

Or we can check the add-on for an ACTIVE status with Amazon EKS API:

$ aws eks describe-addon --cluster-name ebs-demo-cluster --addon-name aws-ebs-csi-driver
{
    "addon": {
        "addonName": "aws-ebs-csi-driver",
        "clusterName": "ebs-demo-cluster",
        "status": "ACTIVE",
        "addonVersion": "v1.5.2-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:eu-west-1:<accountId>:addon/ebs-demo-cluster/aws-ebs-csi-driver/18bedd09-c3e8-8a5f-fffa-a6a32fbd108c",
        "createdAt": 1639504382.132,
        "modifiedAt": 1639504449.617,
        "serviceAccountRoleArn": "arn:aws:iam::<accountId>:role/eksctl-ebs-demo-cluster-addon-aws-ebs-csi-dr-Role1-TX3SPPGLTRLZ",
        "tags": {}
    }

eksctl sets our current Kubernetes context to the newly created cluster:

$ kubectl config current-context
juw@ebs-demo-cluster.eu-west-1.eksctl.io

For existing clusters that match the EBS CSI add-on prerequisite of platform version eks.5+, we can create the add-on by using the eksctl create addon command and specifying the AWS managed IAM policy. The corresponding IRSA setup and ServiceAccount annotation is added automatically.

Note that you may need to enable OIDC if it’s not already done for your cluster.

$ eksctl utils associate-iam-oidc-provider --cluster <existing cluster> --approve

Then, you can create the add-on.

$ eksctl create addon --name aws-ebs-csi-driver --version v1.5.2-eksbuild.1 --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --cluster <existing cluster>

Many customers are using Terraform to build their infrastructure. The Terraform AWS provider contains the aws_eks_addon resource to create EKS managed add-ons as well. Here is a code snippet:

$ cat eks-add-on-ebs-csi.tf
...
data "aws_eks_cluster" "eks" {
  name = local.name
}

data "aws_iam_policy" "ebs_csi_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

module "irsa-ebs-csi" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "4.7.0"

  create_role                   = true
  role_name                     = "AmazonEKSTFEBSCSIRole-${local.name}"
  provider_url                  = replace(data.aws_eks_cluster.eks.identity.0.oidc.0.issuer, "https://", "")
  role_policy_arns              = [data.aws_iam_policy.ebs_csi_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
}

resource "aws_eks_addon" "ebs-csi" {
  cluster_name             = local.name
  addon_name               = "aws-ebs-csi-driver"
  addon_version            = "v1.5.2-eksbuild.1"
  service_account_role_arn = module.irsa-ebs-csi.iam_role_arn
  tags = {
    "eks_addon" = "ebs-csi"
    "terraform" = "true"
  }
}

The Amazon EBS CSI driver runs in the kube-system namespace and consists of two components—a controller that is deployed as a Deployment, and the CSI nodes that are rolled out as DaemonSets in your EKS cluster. The components are shown in the following kubectl command output. Note that even though we have no Windows-capable nodes in the demo cluster, a DaemonSet is installed and ready to support them if added at a later time. For a detailed explanation, see the official EBS CSI documentation.

$ kubectl get deploy,ds -l=app.kubernetes.io/name=aws-ebs-csi-driver -n kube-system
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ebs-csi-controller   2/2     2            2           5m30s
 
NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/ebs-csi-node           2         2         2       2            2           kubernetes.io/os=linux   5m31s
daemonset.apps/ebs-csi-node-windows   0         0         0       0            0           kubernetes.io/os=windows 5m31s

$ kubectl get po -n kube-system -l 'app in (ebs-csi-controller,ebs-csi-node)'
NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-67c885fcc7-n6ll6   6/6     Running   0          6m2s
ebs-csi-controller-67c885fcc7-zvcct   6/6     Running   0          6m32s
ebs-csi-node-7b7b9                    3/3     Running   0          6m32s
ebs-csi-node-nszsg                    3/3     Running   0          6m32s

Each ebs-csi-controller pod contains six sidecar containers, as listed here (The actual pod name is different in your environment):

$ kubectl get -n kube-system pod/ebs-csi-controller-67c885fcc7-n6ll6 -o jsonpath='{.spec.containers[*].name}'
ebs-plugin csi-provisioner csi-attacher csi-snapshotter csi-resizer liveness-probe

Note: The csi-snapshotter and csi-resizer sidecars and the respective capabilities weren’t supported as part of the Amazon EBS CSI add-on preview but are introduced in the General Availability (GA) version.

By default, every node is capable of using CSI-based Amazon EBS volumes:

$ kubectl get csinodes
NAME                                            DRIVERS   AGE
ip-192-168-115-162.eu-west-1.compute.internal   1         41m
ip-192-168-137-52.eu-west-1.compute.internal    1         41m

IRSA is used only by the CSI controller component when using the ServiceAccount (SA) ebs-csi-controller-sa. The SA is automatically annotated with the IAM role created by either eksctl or Terraform. The role name is generated and will appear differently in your cluster.

$ kubectl get sa -n kube-system ebs-csi-controller-sa -o yaml | head -5
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<accountId>:role/eksctl-ebs-demo-cluster-addon-aws-ebs-csi-dr-Role1-Q269IPNF9XDB

Every Amazon EKS cluster currently comes with an in-tree plugin–based StorageClass (SC):

$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 62m

To use the external EBS CSI driver, we need to create a new StorageClass based upon it:

$ cat gp3-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3
allowVolumeExpansion: true
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

Note: Because the GA version supports the volume resizing feature, we set the allowVolumeExpansion property to true.

$ kubectl apply -f gp3-sc.yaml
storageclass.storage.k8s.io/gp3 created

The provisioner column shows that this StorageClass is now based on the Amazon EBS CSI add-on driver:

$ kubectl get sc gp3
NAME            PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3             ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   6s

Note: The in-tree provisioner–based StorageClass gp2 is still available. It’s usable and coexists with the EBS CSI–based gp3 StorageClass. No migration of in-tree to external CSI provisioner occurs when using the Amazon EKS managed add-on Amazon EBS CSI.

Now we create a PersistentVolumeClaim (PVC), which will be used by a pod to show the dynamic provisioning capability of a gp3-based StorageClass:

$ cat pvc-csi.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-csi
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 1Gi

$ kubectl apply -f pvc-csi.yaml
persistentvolumeclaim/pvc-csi created

$ kubectl get pvc pvc-csi
NAME      STATUS  VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-csi   Pending                              gp3          6s

The PVC is still in pending status because the gp3 StorageClass uses a volumeBindingMode of WaitForFirstConsumer. This attribute makes sure that the PersistentVolume (PV) and Pod will be provisioned in the same AWS availability zone (AZ). We can now deploy a Pod to create and bind an underlying PV.

$ cat pod-csi.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-gp3
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: pvc-csi

Once the pod is running (it can take a few seconds), the PVC goes into Bound status and an underlying PV is provisioned:

$ kubectl apply -f pod-csi.yaml
pod/app-gp3 created
$ kubectl get pod
NAME    READY STATUS   RESTARTS AGE
app-gp3 1/1   Running  0        45s
$ kubectl get pvc
NAME     STATUS VOLUME                                   CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-csi  Bound  pvc-19b44f3a-b4f8-4376-8089-60e9eada0d24 1Gi      RWO          gp3          11m
$ kubectl get pv pvc-19b44f3a-b4f8-4376-8089-60e9eada0d24
NAME                                     CAPACITY ACCESS MODES RECLAIM POLICY  STATUS  CLAIM            STORAGECLASS REASON AGE
pvc-19b44f3a-b4f8-4376-8089-60e9eada0d24 1Gi      RWO          Delete          Bound   default/pvc-csi  gp3                 52s

Keep in mind that the underlying EBS volume of a PV is bound to an AZ. If you want to stop and restart the pod later on, make sure to add nodeSelector or nodeAffinity to the YAML specification to run the pod on a node that is part of the same AZ as the EBS volume.

Additional examples such as resizing, snapshotting, and more can be found in the official Amazon EBS CSI driver GitHub repository.

Cleanup

We delete the pod and the PVC. Doing this deletes the underlying PV as well:

$ kubectl delete pod app-gp3
pod "app-gp3" deleted
$ kubectl delete pvc pvc-csi
persistentvolumeclaim "pvc-csi" deleted
$ kubectl get pv
No resources found

Then we delete the Amazon EBS CSI add-on:

$ eksctl delete addon --name aws-ebs-csi-driver --cluster ebs-demo-cluster

Finally, we delete the ebs-demo-cluster cluster:

$ eksctl delete cluster ebs-demo-cluster

Conclusion

This blog post describes how Amazon EKS add-ons can simplify using the Amazon EBS CSI driver to provision and manage volumes for your persistent storage needs in Kubernetes. For more information on using the Amazon EBS CSI driver, see the open-source project and the Amazon EKS User Guide.

As mentioned above, we recommend moving to the Amazon EBS CSI Driver now and specifying the ebs.csi.aws.com provisioner in your storage specifications. For Amazon EKS Kubernetes 1.23, CSI migration will be enabled by default and clusters will need to have the EBS CSI driver installed regardless of specifications. For more information on defining a new CSI-based StorageClass (SC) and explicitly migrating workloads refer to the Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes blog.

Amazon EKS provides you with a managed control plane, options for managing the data plane with managed node groups, and managed cluster add-ons for critical components such as the Amazon VPC Container Network Interface (CNI), CoreDNS, kube-proxy, and now the Amazon EBS CSI driver. We’re planning to release more add-ons to simplify Amazon EKS cluster operations with additional operational software in the coming months.

To check on upcoming features or to suggest features, see the AWS Containers Roadmap on GitHub. If you have questions or suggestions, please leave a comment.

Containers