AWS Storage Blog
Simplifying Amazon EBS volume migration and modification on Kubernetes using the EBS CSI Driver
Enterprises running critical applications in containers may require access to a persistent storage layer that extends beyond the lifetime of a container instance. A block storage solution such as Amazon Elastic Block Store (Amazon EBS) is a good fit due to its high performance, low latency, and persistence which ensures that data can be re-attached to new containers. In particular, AWS customers deploying and running containerized applications using Kubernetes (K8s) orchestration or a container orchestration service on AWS such as Amazon Elastic Kubernetes Service (Amazon EKS) choose the SSD-based general-purpose EBS volumes as their preferred storage solution. Using EBS customers can achieve high performance at low cost for a wide variety of applications, such as virtual desktops, databases, as well as development and testing environments.
Kubernetes customers use the EBS Container Storage Interface (CSI) driver to provision and manage their EBS volumes. In August 2022, AWS enabled CSI migration (CSIMigration) feature by default in EKS 1.23, making CSI driver the default storage driver for EKS customers using EBS and replacing the Kubernetes “in-tree” storage driver that exists in the Kubernetes project source code. Using the CSI driver, customers can now use all available EBS volume types and additional feature set for their storage needs, previously not available via the in-tree driver. Until now, K8s customers modify their volume performance and types directly using AWS Console or APIs. However, when using the K8s control plane, users build workarounds, sometimes recreating the volumes, to migrate between EBS volume types and change performance attributes. The volume type migration often requires a downtime of applications, slowing down enterprise customers’ adoption of newer generation of EBS volumes.
Customers like SAP want to use the EBS CSI driver to be the one stop shop for all their storage management needs running on EKS or other K8s managements, without having to use AWS specific APIs. We are working with the K8s community to make volume modification available for all storage providers using standard Kubernetes interfaces. Given that EKS 1.22 is End of Support (EOS) on June 4, 2023 and EKS 1.23 has CSIMigration enabled by default, customers increasingly depend on the CSI driver, we felt that it was important to provide a solution to AWS customers now, before a standardized solution becomes available. To simplify operations, we added support for new ModifyVolume capabilities within the CSI driver. In addition to the existing ability to increase volume size, customers can now also change the volume type and adjust the performance (IOPS and throughput) of their EBS volumes, by modifying the annotations within their Persistent Volume Claims (PVC). Changes made through annotations apply dynamically to the Amazon EBS volumes without the need to detach them.
In this blog post, we cover how the new ModifyVolume capability works with the EBS CSI driver. Then, we walk through an example of seamlessly migrating to EBS gp3 volumes, a newer generation of general-purpose volumes that provides 20% lower price per GB than the gp2 volumes, and changing the performance characteristics, using Kubernetes native API via the EBS CSI driver. This enables customers to update volume properties directly using the CSI driver, using Kubernetes APIs without downtime of the applications, while the changes take effect.
Solution overview
In general quality-of-service related volume parameters throughput and input output per second (IOPS), as well as volume types, are parameters of the K8s StorageClass as described here. Amazon EBS CSI driver uses the following parameters as part of the StorageClass specification definition. Here is an example:
These parameters of a StorageClass are immutable, such that neither PersistentVolume (PV) nor PVC are allowed to change them. Only the size of a PVC can be modified via spec.requests.resources.storage.
That means to modify the K8s storage properties you have to do a storage migration by creating another StorageClass and migrating the PVC/PV which requires application downtime. This is a cumbersome and error-prone task especially given the fact that cloud storage providers usually allow online modification of volume parameters.
The EBS CSI driver contains two main components – the controller which is implemented as a K8s Deployment with two replicas and a node component which is a DaemonSet. The controller pods are then comprised of several sidecars. The new ModifyVolume feature is implemented as an additional sidecar, called “volumemodifier”. With volumemodifier sidecar, there is no need to change the IAM permissions for the controller component because it already utilizes the AWS EC2 ModifyVolume API call for PersistentVolumeClaim (PVC) resize which is a standard K8s feature.
Prerequisites
The new feature is available starting with EBS CSI version v1.19.0, which is used by Helm chart 2.19.0 or EKS managed add-on v1.19.0-eksbuild.2, but the feature will not be enabled by default.
Self-managed configuration using Helm
You have to opt-in by either modifying the Helm chart “values.yaml” file as follows:
EKS managed add-on
Use EKS API CreateAddon, DescribeAddonConfiguration and UpdateAddon to modify the add-on configuration.
Note: The following output use the “jq” utility to enable a better visualization of the EBS CSI driver configuration schema. Jq has to be installed separately.
Here you can see sample outputs from modifying an existing EBS CSI add-on in a demo cluster called “tf-git-eks-demo-ipv4”.
Here you can see example outputs of creating the EBS CSI add-on with the ModifyVolume feature enabled in a demo cluster called “tf-git-eks-demo-ipv4”:
Depending on your environment the controller now contains 6 (without usage of external-snapshotter sidecar) or 7 sidecar containers:
Walkthrough
The ModifyVolume feature is implemented by annotating the PVC with 3 possible annotations, two of them change the so-called quality-of-service parameters volume throughput and volume IOPS, one modifies the EBS volume type.
Here is an example of possible annotations:
The annotations will be reconciled with the AWS EC2 API for new and existing PVC i.e. one can now change the EBS volume type of an existing PVC.
In this section, we walk through the following three real-word examples using the ModifyVolume feature in EBS CSI driver:
- Volume type change
- IOPS modification
- Multiple modifications in one step
Note: Be aware that there is a limitation of one volume change per six hours according to EBS documentation. Either use different EBS volumes or apply one change with multiple annotations to an EBS volume. Otherwise, you will get an error message similar to the following example output:
Volume type change
- In the first example we are going to migrate an in-tree provisioned gp2 based EBS volume to gp3. The PVC was initially created by in-tree provisioner based StorageClass gp2 and migrated to EBS CSI driver and is of EBS volume type gp2 which can be seen in the following outputs:
- Here we imperatively change the PVC by adding the annotation for volume type modification ebs.csi.aws.com/volumeType=”gp3″:
- Using the kubectl describe call for the PVC we are able to easily see the modification in the “Events” section of the output:
- The modification will be reconciled with the AWS EC2 (EBS) API and is visible with the following AWS CLI command:
- For more information on how to monitor this progress, see the documentation on monitoring the progress of volume modifications.
Note: The underlying PV will be annotated automatically with the same annotations applied to the PVC:
IOPS modification
- In the second example we are modifying the IOPS of an EBS CSI based volume:
- This time we apply the annotation for volume IOPS modification ebs.csi.aws.com/iops=3500:
Note: K8s annotations can be managed declaratively in K8s YAML manifests as well allowing these changes to be applied via Infrastructure as Code (IaC) or GitOps tools like Terraform, ArgoCD or Flux.
- Again, using the kubectl describe call for the PVC we are able to easily see the modification in the Events section of the output:
- The AWS CLI describe-voume call shows the expected change:
Multiple modifications in one step
- In the last example we are modifying PVC “redis-data-redis-master-0” again. This time we are going to modify the volume type and IOPS :
- Because this PVC already contains an annotation “ebs.csi.aws.com/volumeType“ we have to use the “–overwrite” flag. In addition, we have to apply both annotations at once to call the AWS EC2 ModifyVolume API with both changes to avoid running into “maximum modification rate per volume limit of 6 hours”.
Note: Be aware that for io2 based EBS volumes there is a maximum ratio of 500:1 for IOPS to size.
Recommendation
Even if ModifyVolume feature is now available, a good practice is to create an additional EBS CSI based storage class for gp3 EBS volume type and annotate it as the default StorageClass:
Conclusion
In this blog post, we’ve explored a new solution that simplifies volume migration and modifications using the EBS CSI Driver. As the default storage driver for EKS, our mission is to make the CSI driver as user-friendly as possible. The new addition of the ModifyVolume capability makes it simpler for customers migrating from gp2 to gp3 volumes and adjusting performance metrics like IOPS and throughput, using only native K8s APIs and configurations. This development makes it easier for them to right size their storage performance and cost.
To get started, follow the EBS CSI user guide and ModifyVolume example on GitHub. To learn more, you can visit the Amazon EBS product page, Amazon EKS product page, and the open source project on Github.
Thank you for reading this post. If you have any comments or questions, then don’t hesitate to leave them in the comments section.