AWS Storage Blog
Simplifying Amazon EBS volume migration and modification on Kubernetes using the EBS CSI Driver
Enterprises running critical applications in containers may require access to a persistent storage layer that extends beyond the lifetime of a container instance. A block storage solution such as Amazon Elastic Block Store (Amazon EBS) is a good fit due to its high performance, low latency, and persistence which ensures that data can be re-attached to new containers. In particular, AWS customers deploying and running containerized applications using Kubernetes (K8s) orchestration or a container orchestration service on AWS such as Amazon Elastic Kubernetes Service (Amazon EKS) choose the SSD-based general-purpose EBS volumes as their preferred storage solution. Using EBS customers can achieve high performance at low cost for a wide variety of applications, such as virtual desktops, databases, as well as development and testing environments.
Kubernetes customers use the EBS Container Storage Interface (CSI) driver to provision and manage their EBS volumes. In August 2022, AWS enabled CSI migration (CSIMigration) feature by default in EKS 1.23, making CSI driver the default storage driver for EKS customers using EBS and replacing the Kubernetes “in-tree” storage driver that exists in the Kubernetes project source code. Using the CSI driver, customers can now use all available EBS volume types and additional feature set for their storage needs, previously not available via the in-tree driver. Until now, K8s customers modify their volume performance and types directly using AWS Console or APIs. However, when using the K8s control plane, users build workarounds, sometimes recreating the volumes, to migrate between EBS volume types and change performance attributes. The volume type migration often requires a downtime of applications, slowing down enterprise customers’ adoption of newer generation of EBS volumes.
Customers like SAP want to use the EBS CSI driver to be the one stop shop for all their storage management needs running on EKS or other K8s managements, without having to use AWS specific APIs. We are working with the K8s community to make volume modification available for all storage providers using standard Kubernetes interfaces. Given that EKS 1.22 is End of Support (EOS) on June 4, 2023 and EKS 1.23 has CSIMigration enabled by default, customers increasingly depend on the CSI driver, we felt that it was important to provide a solution to AWS customers now, before a standardized solution becomes available. To simplify operations, we added support for new ModifyVolume capabilities within the CSI driver. In addition to the existing ability to increase volume size, customers can now also change the volume type and adjust the performance (IOPS and throughput) of their EBS volumes, by modifying the annotations within their Persistent Volume Claims (PVC). Changes made through annotations apply dynamically to the Amazon EBS volumes without the need to detach them.
In this blog post, we cover how the new ModifyVolume capability works with the EBS CSI driver. Then, we walk through an example of seamlessly migrating to EBS gp3 volumes, a newer generation of general-purpose volumes that provides 20% lower price per GB than the gp2 volumes, and changing the performance characteristics, using Kubernetes native API via the EBS CSI driver. This enables customers to update volume properties directly using the CSI driver, using Kubernetes APIs without downtime of the applications, while the changes take effect.
Solution overview
In general quality-of-service related volume parameters throughput and input output per second (IOPS), as well as volume types, are parameters of the K8s StorageClass as described here. Amazon EBS CSI driver uses the following parameters as part of the StorageClass specification definition. Here is an example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
csi.storage.k8s.io/fstype: xfs
type: gp3
iops: 5000
throughput: 250
encrypted: "true"
These parameters of a StorageClass are immutable, such that neither PersistentVolume (PV) nor PVC are allowed to change them. Only the size of a PVC can be modified via spec.requests.resources.storage. That means to modify the K8s storage properties you have to do a storage migration by creating another StorageClass and migrating the PVC/PV which requires application downtime. This is a cumbersome and error-prone task especially given the fact that cloud storage providers usually allow online modification of volume parameters.
The EBS CSI driver contains two main components – the controller which is implemented as a K8s Deployment with two replicas and a node component which is a DaemonSet. The controller pods are then comprised of several sidecars. The new ModifyVolume feature is implemented as an additional sidecar, called “volumemodifier”. With volumemodifier sidecar, there is no need to change the IAM permissions for the controller component because it already utilizes the AWS EC2 ModifyVolume API call for PersistentVolumeClaim (PVC) resize which is a standard K8s feature.
Prerequisites
The new feature is available starting with EBS CSI version v1.19.0, which is used by Helm chart 2.19.0 or EKS managed add-on v1.19.0-eksbuild.2, but the feature will not be enabled by default.
Self-managed configuration using Helm
You have to opt-in by either modifying the Helm chart “values.yaml” file as follows:
# modify EBS CSI Helm chart
$ cat values.yaml
...
controller:
...
volumeModificationFeature:
enabled: true
...
# apply changes according to your Helm chart and repository names, for example
$ helm upgrade --install aws-ebs-csi-driver --namespace kube-system \
aws-ebs-csi-driver/aws-ebs-csi-driver -f values.yamlEKS managed add-on
Use EKS API CreateAddon, DescribeAddonConfiguration and UpdateAddon to modify the add-on configuration.
Note: The following output use the “jq” utility to enable a better visualization of the EBS CSI driver configuration schema. Jq has to be installed separately.
Here you can see sample outputs from modifying an existing EBS CSI add-on in a demo cluster called “tf-git-eks-demo-ipv4”.
# list all configured EKS add-ons
$ aws eks list-addons --cluster-name tf-git-eks-demo-ipv4
{
"addons": [
"kube-proxy",
"vpc-cni",
"aws-ebs-csi-driver"
]
}
# describe the currently installed EBS CSI driver add-on including addon version
$ aws eks describe-addon --cluster-name tf-git-eks-demo-ipv4 --addon-name aws-ebs-csi-driver
{
"addon": {
"addonName": "aws-ebs-csi-driver",
"clusterName": "tf-git-eks-demo-ipv4",
"status": "ACTIVE",
"addonVersion": "v1.19.0-eksbuild.1",
…
}
# describe the available EBS CSI driver add-on versions for EKS 1.27
$ aws eks describe-addon-versions --addon-name aws-ebs-csi-driver --kubernetes-version 1.27
{
"addons": [
{
"addonName": "aws-ebs-csi-driver",
"type": "storage",
"addonVersions": [
{
"addonVersion": "v1.19.0-eksbuild.2",
"architecture": [
"amd64",
"arm64"
],
"compatibilities": [
{
"clusterVersion": "1.27",
"platformVersions": [
"*"
],
"defaultVersion": false
}
],
"requiresConfiguration": false
},
…
}
# describe the EBS CSI driver add-on configuration and parse the configurationSchema attribute
$ aws eks describe-addon-configuration --addon-name aws-ebs-csi-driver --addon-version v1.19.0-eksbuild.2 --query configurationSchema --output text | jq
{
…
"additionalProperties": false,
"description": "Configurable parameters of the AWS EBS CSI Driver",
..
"volumeModificationFeature": {
"additionalProperties": false,
"properties": {
"enabled": {
"default": false,
"description": "Enable modification of volume type, iops, etc via volume-modifier-for-k8s sidecar",
"type": "boolean"
}
},
"type": "object"
…
}
# create a customized configuration file to enable the "volumeModificationFeature" feature
$ cat aws-ebs-csi-addon-config.json
{
"controller": {
"volumeModificationFeature": {
"enabled": true,
},
}
}
# update the EBS CSI driver add-on
$ aws eks update-addon --cluster-name tf-git-eks-demo-ipv4 --addon-name aws-ebs-csi-driver --addon-version v1.19.0-eksbuild.2 --configuration-values 'file://aws-ebs-csi-addon-config.json'
{
"update": {
…
"status": "InProgress",
"type": "AddonUpdate",
"params": [
{
"type": "AddonVersion",
"value": "v1.19.0-eksbuild.2"
},
{
"type": "ConfigurationValues",
"value": "{\n \"controller\": {\n \"volumeModificationFeature\": {\n \"enabled\": true,\n },\n }\n}"
}
],
}Here you can see example outputs of creating the EBS CSI add-on with the ModifyVolume feature enabled in a demo cluster called “tf-git-eks-demo-ipv4”:
# create the EBS CSI driver add-on
$ aws eks create-addon --cluster-name tf-git-eks-demo-ipv4 --addon-name aws-ebs-csi-driver --addon-version v1.19.0-eksbuild.2 --configuration-values 'file://aws-ebs-csi-addon-config.json'
{
"addonName": "aws-ebs-csi-driver",
"clusterName": " tf-git-eks-demo-ipv4 ",
"status": "CREATING",
"addonVersion": "v1.19.0-eksbuild.2",
…
}
}Depending on your environment the controller now contains 6 (without usage of external-snapshotter sidecar) or 7 sidecar containers:
$ kubectl get pod -n kube-system -l=app=ebs-csi-controller
NAME READY STATUS RESTARTS AGE
ebs-csi-controller-794d6dfc54-l5qwq 7/7 Running 0 20m
ebs-csi-controller-794d6dfc54-q2wmk 7/7 Running 0 20m
$ kubectl get pod -n kube-system ebs-csi-controller-794d6dfc54-l5qwq -o jsonpath='{.spec.containers[*].name}{"\n"}'
ebs-plugin csi-provisioner csi-attacher csi-snapshotter volumemodifier csi-resizer liveness-probeWalkthrough
The ModifyVolume feature is implemented by annotating the PVC with 3 possible annotations, two of them change the so-called quality-of-service parameters volume throughput and volume IOPS, one modifies the EBS volume type.
Here is an example of possible annotations:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-claim
annotations:
"ebs.csi.aws.com/volumeType": "gp3"
"ebs.csi.aws.com/iops": "5000"
"ebs.csi.aws.com/throughput": "250"
spec:
accessModes:
- ReadWriteOnce
storageClassName: ebs-sc
resources:
requests:
storage: 4Gi
The annotations will be reconciled with the AWS EC2 API for new and existing PVC i.e. one can now change the EBS volume type of an existing PVC.
In this section, we walk through the following three real-word examples using the ModifyVolume feature in EBS CSI driver:
- Volume type change
- IOPS modification
- Multiple modifications in one step
Note: Be aware that there is a limitation of one volume change per six hours according to EBS documentation. Either use different EBS volumes or apply one change with multiple annotations to an EBS volume. Otherwise, you will get an error message similar to the following example output:
$ kubectl describe pvc -n prometheus prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning VolumeModificationFailed 17s volume-modifier-for-k8s-ebs.csi.aws.com rpc error: code = Internal desc = Could not modify volume "vol-0a57fe49357e9effc": unable to modify AWS volume "vol-0a57fe49357e9effc": VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
status code: 400, request id: 37938e79-6eca-42c4-afdb-28bed041adaf
Volume type change
- In the first example we are going to migrate an in-tree provisioned gp2 based EBS volume to gp3. The PVC was initially created by in-tree provisioner based StorageClass gp2 and migrated to EBS CSI driver and is of EBS volume type gp2 which can be seen in the following outputs:
$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 8d $ kubectl get pvc redis-data-redis-master-0 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE redis-data-redis-master-0 Bound pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e 8Gi RWO gp2 5d22h # get the underlying PV $ kubectl get pv pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e 8Gi RWO Delete Bound default/redis-data-redis-master-0 gp2 5d22h # get details about the PVC $ kubectl get pvc redis-data-redis-master-0 -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com volume.kubernetes.io/selected-node: ip-<redacted>.eu-west-1.compute.internal volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com ... finalizers: - kubernetes.io/pvc-protection labels: app.kubernetes.io/component: master app.kubernetes.io/instance: redis app.kubernetes.io/name: redis name: redis-data-redis-master-0 namespace: default ... spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: gp2 volumeMode: Filesystem volumeName: pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e status: accessModes: - ReadWriteOnce capacity: storage: 8Gi phase: Bound # get details about the underlying PV $ kubectl get pv pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/migrated-to: ebs.csi.aws.com pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs ... finalizers: - kubernetes.io/pv-protection - external-attacher/ebs-csi-aws-com labels: topology.kubernetes.io/region: eu-west-1 topology.kubernetes.io/zone: eu-west-1b name: pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e ... spec: accessModes: - ReadWriteOnce awsElasticBlockStore: fsType: ext4 volumeID: vol-041a6c12bcc9e4cfc capacity: storage: 8Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: redis-data-redis-master-0 namespace: default ... nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - eu-west-1b - key: topology.kubernetes.io/region operator: In values: - eu-west-1 persistentVolumeReclaimPolicy: Delete storageClassName: gp2 volumeMode: Filesystem status: phase: Bound # get details about the corresponding AWS EBS object $ aws ec2 describe-volumes --volume-ids vol-041a6c12bcc9e4cfc { "Volumes": [ { "Attachments": [ { "AttachTime": "2023-05-08T07:56:33+00:00", "Device": "/dev/xvdaa", "InstanceId": "i-<redacted>", "State": "attached", "VolumeId": "vol-041a6c12bcc9e4cfc", "DeleteOnTermination": false } ], "AvailabilityZone": "eu-west-1b", "CreateTime": "2023-05-03T08:39:41.629000+00:00", "Encrypted": false, "Size": 8, "SnapshotId": "", "State": "in-use", "VolumeId": "vol-041a6c12bcc9e4cfc", "Iops": 100, ... "VolumeType": "gp2", "MultiAttachEnabled": false } ] } - Here we imperatively change the PVC by adding the annotation for volume type modification ebs.csi.aws.com/volumeType=”gp3″:
$ kubectl annotate pvc redis-data-redis-master-0 ebs.csi.aws.com/volumeType="gp3" persistentvolumeclaim/redis-data-redis-master-0 annotate $ kubectl get pvc redis-data-redis-master-0 -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: ebs.csi.aws.com/volumeType: gp3 pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com ... - Using the kubectl describe call for the PVC we are able to easily see the modification in the “Events” section of the output:
$ kubectl describe pvc redis-data-redis-master-0 .. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal VolumeModificationStarted 21s volume-modifier-for-k8s-ebs.csi.aws.com External modifier is modifying volume pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e Normal VolumeModificationSuccessful 10s volume-modifier-for-k8s-ebs.csi.aws.com External modifier has successfully modified volume pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e - The modification will be reconciled with the AWS EC2 (EBS) API and is visible with the following AWS CLI command:
$ aws ec2 describe-volumes --volume-ids vol-041a6c12bcc9e4cfc { "Volumes": [ { ... "VolumeId": "vol-041a6c12bcc9e4cfc", "Iops": 3000, ... "VolumeType": "gp3", "MultiAttachEnabled": false, "Throughput": 125 } ] } - For more information on how to monitor this progress, see the documentation on monitoring the progress of volume modifications.
Note: The underlying PV will be annotated automatically with the same annotations applied to the PVC:# get details about the underlying PV $ kubectl get pv pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: ebs.csi.aws.com/volumeType: gp3 pv.kubernetes.io/migrated-to: ebs.csi.aws.com pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs …
IOPS modification
- In the second example we are modifying the IOPS of an EBS CSI based volume:
$ kubectl get pvc -n prometheus prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com volume.kubernetes.io/selected-node: ip-<redacted>.eu-west-1.compute.internal volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com ... finalizers: - kubernetes.io/pvc-protection ... name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 namespace: prometheus ... spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: gp3 volumeMode: Filesystem volumeName: pvc-8a920f77-0187-45ba-bbff-cc4fb749480b status: accessModes: - ReadWriteOnce capacity: storage: 50Gi phase: Bound $ kubectl get pv pvc-8a920f77-0187-45ba-bbff-cc4fb749480b -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com ... finalizers: - kubernetes.io/pv-protection - external-attacher/ebs-csi-aws-com name: pvc-8a920f77-0187-45ba-bbff-cc4fb749480b ... spec: accessModes: - ReadWriteOnce capacity: storage: 50Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 namespace: prometheus ... csi: driver: ebs.csi.aws.com fsType: ext4 volumeAttributes: storage.kubernetes.io/csiProvisionerIdentity: 1683103179760-8081-ebs.csi.aws.com volumeHandle: vol-0a57fe49357e9effc nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: topology.ebs.csi.aws.com/zone operator: In values: - eu-west-1a persistentVolumeReclaimPolicy: Delete storageClassName: gp3 volumeMode: Filesystem status: phase: Bound $ aws ec2 describe-volumes --volume-ids vol-0a57fe49357e9effc { "Volumes": [ { "Attachments": [ { "AttachTime": "2023-05-09T07:06:52+00:00", "Device": "/dev/xvdaa", "InstanceId": "i-<redacted>", "State": "attached", "VolumeId": "vol-0a57fe49357e9effc", "DeleteOnTermination": false } ], "AvailabilityZone": "eu-west-1a", "CreateTime": "2023-05-03T08:39:43.260000+00:00", "Encrypted": true, "KmsKeyId": "arn:aws:kms:eu-west-1:<redacted>:key/<redacted>", "Size": 50, "SnapshotId": "", "State": "in-use", "VolumeId": "vol-0a57fe49357e9effc", "Iops": 3000, ... "VolumeType": "gp3", "MultiAttachEnabled": false, "Throughput": 125 } ] } - This time we apply the annotation for volume IOPS modification ebs.csi.aws.com/iops=3500:
$ kubectl annotate pvc -n prometheus prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 ebs.csi.aws.com/iops=3500 persistentvolumeclaim/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 annotate $ kubectl get pvc -n prometheus prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: ebs.csi.aws.com/iops: "3500" pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com ...Note: K8s annotations can be managed declaratively in K8s YAML manifests as well allowing these changes to be applied via Infrastructure as Code (IaC) or GitOps tools like Terraform, ArgoCD or Flux.
- Again, using the kubectl describe call for the PVC we are able to easily see the modification in the Events section of the output:
$ kubectl describe pvc -n prometheus prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 .. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal VolumeModificationStarted 16s volume-modifier-for-k8s-ebs.csi.aws.com External modifier is modifying volume pvc-8a920f77-0187-45ba-bbff-cc4fb749480b Normal VolumeModificationSuccessful 10s volume-modifier-for-k8s-ebs.csi.aws.com External modifier has successfully modified volume pvc-8a920f77-0187-45ba-bbff-cc4fb749480b - The AWS CLI describe-voume call shows the expected change:
$ aws ec2 describe-volumes --volume-ids vol-0a57fe49357e9effc { "Volumes": [ ... "AvailabilityZone": "eu-west-1a", "CreateTime": "2023-05-03T08:39:43.260000+00:00", "Encrypted": true, "KmsKeyId": "arn:aws:kms:eu-west-1:<redacted>:key/<redacted>", "Size": 50, "SnapshotId": "", "State": "in-use", "VolumeId": "vol-0a57fe49357e9effc", "Iops": 3500, ... "MultiAttachEnabled": false, "Throughput": 125 } ] }
Multiple modifications in one step
- In the last example we are modifying PVC “redis-data-redis-master-0” again. This time we are going to modify the volume type and IOPS :
$ kubectl get pvc redis-data-redis-master-0 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE redis-data-redis-master-0 Bound pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e 8Gi RWO gp2 13d $ kubectl describe pvc redis-data-redis-master-0 Name: redis-data-redis-master-0 Namespace: default StorageClass: gp2 Status: Bound Volume: pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e Labels: app.kubernetes.io/component=master app.kubernetes.io/instance=redis app.kubernetes.io/name=redis Annotations: ebs.csi.aws.com/volumeType: gp3 pv.kubernetes.io/bind-completed: yes ... - Because this PVC already contains an annotation “ebs.csi.aws.com/volumeType“ we have to use the “–overwrite” flag. In addition, we have to apply both annotations at once to call the AWS EC2 ModifyVolume API with both changes to avoid running into “maximum modification rate per volume limit of 6 hours”.
$ kubectl annotate pvc redis-data-redis-master-0 ebs.csi.aws.com/volumeType="io2" ebs.csi.aws.com/iops=4000 --overwrite persistentvolumeclaim/redis-data-redis-master-0 annotate $ k describe pvc redis-data-redis-master-0 Name: redis-data-redis-master-0 Namespace: default StorageClass: gp2 Status: Bound Volume: pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e Labels: app.kubernetes.io/component=master app.kubernetes.io/instance=redis app.kubernetes.io/name=redis Annotations: ebs.csi.aws.com/iops: 4000 ebs.csi.aws.com/volumeType: io2 pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com volume.kubernetes.io/selected-node: ip-<redacted>.eu-west-1.compute.internal volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com Finalizers: [kubernetes.io/pvc-protection] Capacity: 8Gi Access Modes: RWO VolumeMode: Filesystem Used By: redis-master-0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal VolumeModificationStarted 5s volume-modifier-for-k8s-ebs.csi.aws.com External modifier is modifying volume pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e Normal VolumeModificationSuccessful 2s volume-modifier-for-k8s-ebs.csi.aws.com External modifier has successfully modified volume pvc-c06e7ecd-c4c8-46d1-801a-471c2541ad6e $ aws ec2 describe-volumes --volume-ids vol-041a6c12bcc9e4cfc { "Volumes": [ { ... "Size": 8, "SnapshotId": "", "State": "in-use", "VolumeId": "vol-041a6c12bcc9e4cfc", "Iops": 4000, ... "VolumeType": "io2", "MultiAttachEnabled": false } ] }Note: Be aware that for io2 based EBS volumes there is a maximum ratio of 500:1 for IOPS to size.
Recommendation
Even if ModifyVolume feature is now available, a good practice is to create an additional EBS CSI based storage class for gp3 EBS volume type and annotate it as the default StorageClass:
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 8d
gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 8d
$ kubectl get sc gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
...
name: gp2
...
fsType: ext4
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
$ kubectl get sc gp3 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
...
name: gp3
...
parameters:
encrypted: "true"
kmsKeyId: arn:aws:kms:<redacted:key/<redacted>
type: gp3
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
# note the trailing "-" to remove the "default" annotation
$ kubectl annotate sc gp2 storageclass.kubernetes.io/is-default-class-
storageclass.storage.k8s.io/gp2 annotate
# annotate the gp3 storageclass as the default one
$ kubectl annotate sc gp3 storageclass.kubernetes.io/is-default-class=true
storageclass.storage.k8s.io/gp3 annotate
$ kubectl get sc gp3 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
...
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 8d
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 8d
Conclusion
In this blog post, we’ve explored a new solution that simplifies volume migration and modifications using the EBS CSI Driver. As the default storage driver for EKS, our mission is to make the CSI driver as user-friendly as possible. The new addition of the ModifyVolume capability makes it simpler for customers migrating from gp2 to gp3 volumes and adjusting performance metrics like IOPS and throughput, using only native K8s APIs and configurations. This development makes it easier for them to right size their storage performance and cost.
To get started, follow the EBS CSI user guide and ModifyVolume example on GitHub. To learn more, you can visit the Amazon EBS product page, Amazon EKS product page, and the open source project on Github.
Thank you for reading this post. If you have any comments or questions, then don’t hesitate to leave them in the comments section.