Containers

Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes

Kubernetes (sometimes referred to as K8s) is an open-source container orchestration engine and a fast-growing project hosted by the Cloud Native Computing Foundation (CNCF). K8s has a massive adoption on premises and in the cloud for running stateless and stateful containerized workloads. Stateful workloads require persistent storage. To support on-premises and cloud-provider-related infrastructure like storage and networking, Kubernetes source code originally included so-called “in-tree plugins.” Storage and cloud vendors who wanted to add new storage systems or features or just wanted to fix bugs had to rely on the Kubernetes release cycle. To decouple the life cycle of Kubernetes from vendor-specific implementations, the development of Container Storage Interface (CSI), a standard for exposing arbitrary block and file storage systems to containerized workloads on container orchestration systems like Kubernetes, was initiated. Customers can now benefit from the latest CSI driver without having to wait for new Kubernetes version releases.

AWS launched their managed Kubernetes service Amazon Elastic Kubernetes Service (Amazon EKS) at re:Invent 2017. In September 2019, AWS released support for the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver in Amazon EKS, and in May 2021 AWS announced the general availability of this CSI plugin. The Kubernetes Amazon EBS related in-tree storage plugin—called a provisioner of type “kubernetes.io/aws-ebs”—only supports Amazon EBS types io1, gp2, sc1, and st1, and it does not support volume snapshot related features.

Our customers are now asking when and how to migrate an EKS cluster from the Amazon EBS in-tree plugin to the Amazon EBS CSI driver to make use of additional EBS volume types (like gp3 and io2) and take advantage of new features (like Kubernetes Volume Snapshots).

Container Storage Interface (CSI) migration infrastructure has been in beta feature state since Kubernetes v1.17 and is described in detail in this K8s blog post. It is important to understand that the migration will eventually remove the in-tree plugin from the Kubernetes source code, and all migrated volumes will be controlled by the CSI driver. It does not mean that those migrated PVs will get new features and attributes of the CSI driver. It only supports features that are already supported by in-tree drivers as described here.

This blog post will walk you through a migration scenario and outline the necessary steps in detail.

Prerequisites

You need an EKS cluster with version 1.17 or newer and a corresponding version of kubectl. Make sure you are authorized to install the Amazon EBS CSI-related objects.

Kubernetes uses so-called feature gates to implement the storage migration. The CSIMigration and CSIMigrationAWS feature for Amazon EBS, when enabled, redirects all plugin operations from the existing in-tree plugin to the ebs.csi.aws.com CSI driver. Please note that Amazon EKS has not yet turned on the features CSIMigration and CSIMigrationAWS for Amazon EBS migration. Nevertheless, you can already use the Amazon EBS CSI driver in parallel to the in-tree plugin.

For the sake of the demo, we will create a dynamic PersistentVolume (PV), which we are going to migrate later.

We use dynamic volume provisioning as described in the K8s documentation.

The in-tree storage driver-based default StorageClass (SC) gp2 will be used to create a PersistentVolumeClaim (PVC) :

$ kubectl get sc
NAME                    PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2   (default)     kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  242d

$ cat ebs-gp2-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-gp2-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp2 

$ kubectl apply -f ebs-gp2-claim.yaml
persistentvolumeclaim/ebs-gp2-claim created

The PVC is created in status “pending” because the gp2 StorageClass has a Volume Binding Mode (attribute volumeBindingMode) of WaitForFirstConsumer and there is no pod yet consuming the PVC.

$ kubectl get pvc ebs-gp2-claim
NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ebs-gp2-claim   Pending                                      gp2            45s

So let’s create a pod (our “demo application”) that uses the PVC:

$ cat test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-gp2-in-tree
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-gp2-claim

$ kubectl apply -f test-pod.yaml
pod/app-gp2-in-tree created

After a few seconds the pod is created:

$ kubectl get po app-gp2-in-tree
NAME              READY   STATUS    RESTARTS   AGE
app-gp2-in-tree   1/1     Running   0          16s

This will dynamically provision the underlying PV pvc-646fef81-c677-46f4-8f27-9d394618f236, which is now bound to the PVC “ebs-gp2-claim”

$ kubectl get pvc ebs-gp2-claim
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ebs-gp2-claim   Bound    pvc-646fef81-c677-46f4-8f27-9d394618f236   1Gi        RWO            gp2            5m3s

Let’s quickly check if the volume contains some data:

$ kubectl exec app-gp2-in-tree -- sh -c "cat /data/out.txt"
…
Thu Sep 16 13:56:34 UTC 2021
Thu Sep 16 13:56:39 UTC 2021
Thu Sep 16 13:56:44 UTC 2021

Let’s have a look at the details of the PV:

$ kubectl get pv pvc-646fef81-c677-46f4-8f27-9d394618f236
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
pvc-646fef81-c677-46f4-8f27-9d394618f236   1Gi        RWO            Delete           Bound    default/ebs-gp2-claim   gp2                     2m54s

$ kubectl get pv pvc-646fef81-c677-46f4-8f27-9d394618f236 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: aws-ebs-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
…
  labels:
    topology.kubernetes.io/region: eu-central-1
    topology.kubernetes.io/zone: eu-central-1c
  name: pvc-646fef81-c677-46f4-8f27-9d394618f236
…
spec:
  accessModes:
  - ReadWriteOnce
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://eu-central-1c/vol-03d3cd818a2c2def3
…
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - eu-central-1c
        - key: topology.kubernetes.io/region
          operator: In
          values:
          - eu-central-1
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp2
…

The PV was (as expected) created by the “kubernetes.io/aws-ebs” provisioner as shown in the annotation. The “awsElasticBlockStore.volumeId” attribute within the spec section shows the actual Amazon EBS Volume ID “vol-03d3cd818a2c2def3” together with the AWS Availability Zone (AZ) the EBS volume was created –  eu-central-1c in this case. EBS volumes and EC2 instances are zonal (not Regional) resources. The nodeAffinity section advises the kube-scheduler to provision the pod on a node in the same AZ where the PV is created.

The following command is a short-form to retrieve the Amazon EBS details:

$ kubectl get pv pvc-646fef81-c677-46f4-8f27-9d394618f236 –o jsonpath='{.spec.awsElasticBlockStore.volumeID}'
aws://eu-central-1c/vol-03d3cd818a2c2def3

We want to use this PV for the storage migration scenarios described later. To ensure that the PV will not be deleted when the corresponding PVC, using it, is deleted we are going to patch the “VolumeReclaimPolicy” to “Retain”. Note: This is only possible on the PV level not the SC level!

$ kubectl patch pv pvc-646fef81-c677-46f4-8f27-9d394618f236 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-646fef81-c677-46f4-8f27-9d394618f236 patched
$ kubectl get pv pvc-646fef81-c677-46f4-8f27-9d394618f236
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
pvc-646fef81-c677-46f4-8f27-9d394618f236   1Gi        RWO            Retain           Bound    default/ebs-gp2-claim   gp2                     9m4s

To install the Amazon EBS CSI driver, follow our GitHub documentation. The required high-level steps are:

  • Attach the IAM permissions required for Amazon EBS operations to either the worker node profile or alternatively, following least privilege, use IRSA (IAM roles for service account) to create a properly annotated ServiceAccount (SA)
  • Install the external volume snapshot controller related K8s objects (CRD, RBAC resources, deployment, and validating webhook) using YAML
  • Install Amazon EBS CSI driver using YAML or use corresponding Helm chart (use existing service account if you use IRSA)

Double-check that the Amazon EBS CSI related Kubernetes components are registered to the K8s API server:

$ kubectl api-resources | grep "storage.k8s.io/v1"
volumesnapshotclasses                          snapshot.storage.k8s.io/v1               false        VolumeSnapshotClass
volumesnapshotcontents                         snapshot.storage.k8s.io/v1               false        VolumeSnapshotContent
volumesnapshots                                snapshot.storage.k8s.io/v1               true         VolumeSnapshot
csidrivers                                     storage.k8s.io/v1                        false        CSIDriver
csinodes                                       storage.k8s.io/v1                        false        CSINode
csistoragecapacities                           storage.k8s.io/v1beta1                   true         CSIStorageCapacity
storageclasses                    sc           storage.k8s.io/v1                        false        StorageClass
volumeattachments                              storage.k8s.io/v1                        false        VolumeAttachment

Confirm they are up and running:

$ kubectl get po -n kube-system -l 'app in (ebs-csi-controller,ebs-csi-node,snapshot-controller)'
NAME                                   READY   STATUS    RESTARTS   AGE
ebs-csi-controller-569b794b57-md99s    6/6     Running   0          6d15h
ebs-csi-controller-569b794b57-trkks    6/6     Running   0          6d15h
ebs-csi-node-4fkb8                     3/3     Running   0          6d14h
ebs-csi-node-vc48t                     3/3     Running   0          6d14h
snapshot-controller-6984fdc566-4c49f   1/1     Running   0          6d15h
snapshot-controller-6984fdc566-jlnbn   1/1     Running   0          6d15h
$ kubectl get csidrivers
NAME              ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
ebs.csi.aws.com   true             false            false             <unset>         false               Persistent   7d19h

Migration scenario

First we will discuss the migration scenario at a high level.

We are conducting a physical storage migration by copying the in-tree based PV data using Amazon EBS snapshots as an external snapshot mechanism (note: K8s in-tree Amazon EBS plugin does not support volume snapshots!) and import this data using the CSI  Volume Snapshots feature of the CSI Amazon EBS driver.

Now we will guide you through the migration in detail.

We start by taking a snapshot of the in-tree plugin based PV pvc-646fef81-c677-46f4-8f27-9d394618f236 via the AWS API.

$ kubectl get pv pvc-646fef81-c677-46f4-8f27-9d394618f236 -o jsonpath='{.spec.awsElasticBlockStore.volumeID}'
aws://eu-central-1c/vol-03d3cd818a2c2def3

$ aws ec2 create-snapshot --volume-id vol-03d3cd818a2c2def3 --tag-specifications 'ResourceType=snapshot,Tags=[{Key="ec2:ResourceTag/ebs.csi.aws.com/cluster",Value="true"}]'
{
…
    "SnapshotId": "snap-06fb1faafc1409cc5",
…
    "State": "pending",
    "VolumeId": "vol-03d3cd818a2c2def3",
    "VolumeSize": 1,
…    
}

Wait until snapshot is in state “completed”

$ aws ec2 describe-snapshots --snapshot-ids snap-06fb1faafc1409cc5
{
    "Snapshots": [
        {
…
            "Progress": "100%",
            "SnapshotId": "snap-06fb1faafc1409cc5",
..
            "State": "completed",
            "VolumeId": "vol-03d3cd818a2c2def3",
…
        }
    ]
}

Now create a VolumeSnapshotClass object:

$ cat vsc-ebs-csi.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-csi-aws
driver: ebs.csi.aws.com
deletionPolicy: Delete

$ kubectl apply -f vsc-ebs-csi.yaml
volumesnapshotclass.snapshot.storage.k8s.io/ebs-csi-aws created

$ kubectl get volumesnapshotclass
NAME          DRIVER            DELETIONPOLICY   AGE
ebs-csi-aws   ebs.csi.aws.com   Delete           12s

Next we have to create a VolumeSnapshotContent object that uses the AWS snapshot snap-06fb1faafc1409cc5 and already references a VolumeSnapshot we will create in a next step. This seems odd but is necessary for the bidirectional binding of VolumeSnapshotContent and VolumeSnapshot for preexisting snapshots!

$ cat vsc-csi.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: imported-aws-snapshot-content
spec:
  volumeSnapshotRef:
    kind: VolumeSnapshot
    name: imported-aws-snapshot
    namespace: default
  source:
    snapshotHandle: snap-06fb1faafc1409cc5 # <-- snapshot to import
  driver: ebs.csi.aws.com
  deletionPolicy: Delete
  volumeSnapshotClassName: ebs-csi-aws

$ kubectl apply -f vsc-csi.yaml
volumesnapshotcontent.snapshot.storage.k8s.io/imported-aws-snapshot-content created

$ kubectl get volumesnapshotcontent imported-aws-snapshot-content
NAME                            READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT          VOLUMESNAPSHOTNAMESPACE   AGE
imported-aws-snapshot-content   true         1073741824    Delete           ebs.csi.aws.com   ebs-csi-aws           imported-aws-snapshot   default                   12s

Now we need to create the VolumeSnapshot that references the VolumeSnapshotContent object:

$ cat vs-csi.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: imported-aws-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: ebs-csi-aws
  source:
    volumeSnapshotContentName: imported-aws-snapshot-content

$ kubectl apply -f  vs-csi.yaml
volumesnapshot.snapshot.storage.k8s.io/imported-aws-snapshot created
$ kubectl get volumesnapshot imported-aws-snapshot
NAME                    READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT           RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                 CREATIONTIME   AGE
imported-aws-snapshot   true                     imported-aws-snapshot-content   1Gi           ebs-csi-aws     imported-aws-snapshot-content   33m            60s

During this migration we like to benefit from the new Amazon EBS gp3 storage class. In order to do so we have to create a CSI-based gp3 storage class! Because we want this SC to be the default one we remove the annotation from the Amazon EKS default SC gp2 first:

$ kubectl annotate sc gp2 storageclass.kubernetes.io/is-default-class-
storageclass.storage.k8s.io/gp2 annotated

$ kubectl get sc gp2
NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  243d

$ cat gp3-def-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

$ kubectl apply -f gp3-def-sc.yaml
storageclass.storage.k8s.io/gp3 created

$ kubectl get sc gp3
NAME            PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3 (default)   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   6s

The VolumeSnapshot we created earlier can be used to create a PersistentVolumeClaim. As the storage class we use the new gp3 CSI based SC.

$ cat pvc-vs-csi.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: imported-aws-snapshot-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 1Gi
  dataSource:
    name: imported-aws-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

$ kubectl apply -f pvc-vs-csi.yaml
persistentvolumeclaim/imported-aws-snapshot-pvc created

$ kubectl get pvc imported-aws-snapshot-pvc
NAME                        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
imported-aws-snapshot-pvc   Pending                                      gp3            54s

Note that the PVC is still in pending status because the gp3 SC uses a volumeBindingMode of WaitForFirstConsumer. So we have to create an application (pod) again to create an underlying PV. For demo purposes, we just mount the PVC without writing new data and use “kubectl exec” to have a look at the data of the snapshot:

$ cat test-pod-snap.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-imported-snapshot-csi
spec:
  containers:
  - name: app
    image: centos
    args:
    - sleep
    - "10000"
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: imported-aws-snapshot-pvc

$ kubectl apply -f test-pod-snap.yaml
pod/app-imported-snapshot-csi created

A PV pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20 was automatically created and the pod is running with access to the migrated data:

$ kubectl get pvc imported-aws-snapshot-pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
imported-aws-snapshot-pvc   Bound    pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20   1Gi        RWO            gp3            11m

$ kubectl get po app-imported-snapshot-csi
NAME                        READY   STATUS    RESTARTS   AGE
app-imported-snapshot-csi   1/1     Running   0          85s

$ kubectl exec app-imported-snapshot-csi -- sh -c "cat /data/out.txt" | more
Thu Sep 16 13:56:04 UTC 2021
Thu Sep 16 13:56:09 UTC 2021
Thu Sep 16 13:56:14 UTC 2021
…

The PVC, as expected, uses CSI based gp3 SC:

$ kubectl get pv pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS   REASON   AGE
pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20   1Gi        RWO            Delete           Bound    default/imported-aws-snapshot-pvc   gp3                     3m34s

$ kubectl get pv pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
  creationTimestamp: "2021-09-17T09:57:15Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/ebs-csi-aws-com
  name: pvc-25d2d19d-6ede-47d2-bd2e-32d45832ec20
…
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: imported-aws-snapshot-pvc
    namespace: default
…
  csi:
    driver: ebs.csi.aws.com
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1630589410219-8081-ebs.csi.aws.com
    volumeHandle: vol-036ef87c533d529de
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.ebs.csi.aws.com/zone
          operator: In
          values:
          - eu-central-1c
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp3
  volumeMode: Filesystem
status:
  phase: Bound

Cleanup

 To avoid unnecessary costs, clean up your environment after performing the demo migration.


$ kubectl delete pod app-imported-snapshot-csi

$ kubectl delete pvc imported-aws-snapshot-pvc

$ kubectl delete volumesnapshotcontent imported-aws-snapshot-content

$ kubectl delete volumesnapshot imported-aws-snapshot

$ aws ec2 delete-snapshot --snapshot-ids <snap-id>

$ kubectl delete pv <pvc-id>

For future use, you can leave the CSI driver in your EKS cluster.

Conclusion

Amazon EKS provides customers with a managed control plane, options for managing the data plane (managed node groups), and managed cluster add-ons for critical components like AWS VPC CNI, CoreDNS, and kube-proxy. Once all features related to Amazon EBS CSI migration are finalized, AWS will take care of the heavy lifting of implementing all the bits and pieces on the managed control plane and data plane for you!

This blog post described how you can even start today migrating your workloads to PersistentVolumes, which supports all the new capabilities and features of the Amazon EBS CSI driver.

We hope this post helps with your Kubernetes projects. If you have questions or suggestions, please leave a comment.

TAGS: ,
Jens-Uwe Walther

Jens-Uwe Walther

Jens-Uwe Walther is a Senior Specialist Technical Account Manager for Containers at Amazon Web Services with 28 years of professional experience in IT based in Jena, Germany. He is passionate about Kubernetes and helping customers building container solutions on Amazon EKS. In his free time, he loves playing guitar, taking photographs and traveling with his wife into the wilderness.