EKS support for the EBS CSI driver

Today, we are announcing EKS support for the EBS Container Storage Interface driver, an initiative to create unified storage interfaces between container orchestrators such as Kubernetes and storage vendors like AWS.

A History of Storage in Kubernetes

As originally conceived, containers were a great fit for stateless applications. However, there was no provision for persistent storage, without which stateful workloads were not possible. On the other hand, many applications weren’t designed with containerization in mind, so to support migrating all types of applications to the cloud, container orchestrators built out support for storage, making stateful apps possible.

Kubernetes first introduced support for stateful workloads with in-tree volume plugins, meaning that the plugin code was part of the core Kubernetes code and shipped with the Kubernetes binaries. This model proved challenging as vendors wanting to add support for their storage systems to Kubernetes — or even just fix a bug in an existing volume plugin — were forced to align with the Kubernetes release process. This problem led to the development of Container Storage Interface (CSI), a standard for exposing arbitrary block and file storage storage systems to containerized workloads on container orchestration systems like Kubernetes. CSI support was introduced as alpha in Kubernetes v1.9, moved to beta in Kubernetes v1.10, and went GA in Kubernetes v1.13. The CSI specification enables the container orchestrator and the storage provider to evolve independently in a modular way. By using a CSI driver, you benefit from the decoupling between the Kubernetes upstream release cycle and the CSI driver release cycle. Users can upgrade to the latest driver without waiting for new Kubernetes version releases.

Amazon Elastic Block Store (EBS)

Amazon EBS is a cloud block storage service that provides direct access from an EC2 instance to a dedicated storage volume. Support for EBS initially launched as an in-tree volume plugin in Kubernetes. When the CSI specification was published, we started developing a compatible driver for Amazon EBS. We made the EBS CSI Driver available as open source on GitHub as part of kubernetes-sigs. The driver has been available to use for self-managed Kubernetes installations on AWS since Kubernetes version 1.12 (driver version 0.2 using CSI 0.3 spec) and version 1.13 (driver version 0.3 using CSI 1.0 spec). However, prior to version 1.14, certain features such as CSINodeInfo and CSIBlockVolume were Kubernetes alpha features and thus not supported by Amazon EKS. With EKS support for Kubernetes version 1.14, users can now install the EBS CSI driver (v0.4.0 or greater) to their EKS 1.14 clusters.

The EBS CSI Driver in Action

The EBS CSI driver implements basic operations required by the CSI specification to provision, attach, and mount a volume into a Pod, and the reverse operations to remove it. The existing in-tree workflow of provisioning storage for a pod is unchanged, and all you need to do is to define a new storage class that uses the CSI driver as a provisioner. The following is an example storage class manifest file:

kind: StorageClass
apiVersion: storage.k8s.io/v1 
metadata:
  name: csi-sc 
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

The CSI driver supports all the storage class parameters supported by the current in-tree EBS volume driver. The parameters include type, csi.storage.k8s.io/fsType, iopsPerGB, encrypted and kmsKeyId.  Note that zone/zones is not included because it is deprecated in kubernetes v1.12 in favor of allowedTopologies. If you are building a multi-zone application that requires provisioning volumes in different availability zones for a Pod to access, volume scheduling can be turned on by setting volumeBindingMode to WaitForFirstConsumer.

There are four steps to using the EBS CSI driver for your Kubernetes cluster:

1. Grant proper permission to worker nodes
2. Install driver: kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
3. Create a storage class with a sample manifest file like the above one.
4. Create a persistent volume claim or persistent volume and consume the volume using the same workflow you have been using with the in-tree volume plugin.

For more detailed installation instructions, see the EKS documentation. For more details on the EBS CSI driver, please refer to the README on the aws-ebs-csi-driver GitHub page.

Limitations

To connect to an EBS volume, an application running in a pod on EKS must be on a node in the same availability zone as the EBS volume. Currently, an EBS volume can only be attached to a single Kubernetes node at a time.

Migrating to the EBS CSI Driver

In Kubernetes 1.14, CSIMigrationAWS launched as a Kubernetes alpha feature and added functionality that enables shims and translation logic to route volume operations from the EBS in-tree plugin to the EBS CSI plugin. Given its alpha status, this feature isn’t yet available for production usage in EKS. We are making upstream contributions to Kubernetes in order to move the feature to beta status in a future release. In the meantime, the existing in-tree EBS plugin is still supported.

Future Work

Upcoming work on the EBS CSI plugin will focus on several topics including stabilizing alpha features (for example volume resizing and volume snapshot), getting the driver ready for migration from the in-tree EBS plugin, adding performance tests, and more. If you are interested in more details, our milestones can be found on GitHub. We’re looking forward to hearing your feedback, and any contributions are welcome!

And thanks to Fábio Bertinatto for his help in developing this driver.