Using EBS Snapshots for persistent storage with your EKS cluster

Originally, containers were a great fit for stateless applications. However, for many use cases there is a need for persistent storage, without which stateful workloads are not possible. Kubernetes first introduced support for stateful workloads with in-tree volume plugins, meaning that the plugin code was part of the core Kubernetes code and shipped with the Kubernetes binaries. That proved quite challenging for vendors who wanted to add Kubernetes support to their storage platform, as these features needed to wait for the standard Kubernetes release cycle.

That led to the development of Container Storage Interface (CSI), a standard for exposing arbitrary block and file storage systems to containerised workloads on container orchestration systems like Kubernetes.

Amazon Elastic Block Stock (Amazon EBS) is a cloud block storage service that provides direct access from an EC2 instance to a dedicated storage volume.

Support for EBS initially launched as mentioned above as an in-tree volume plugin in Kubernetes.

When the CSI specification was published, we started developing a compatible driver for Amazon EBS. We made the EBS CSI driver available and open source on GitHub. You can find the list of features supported from the EBS CSI driver here.

In this post, we are going to focus on a specific EBS CSI driver feature: Kubernetes Volume Snapshots. Kubernetes Volume Snapshots lets you create a copy of your EBS volume at a specific point in time. You can use this copy to bring a volume back to a prior state or to provision a new volume.

From version 1.17 and newer, you can provision and attach EBS volume snapshots to your pods with the following components:

• VolumeSnapshotClass
• VolumeSnapshot

Let’s examine all the steps to start using Volume Snapshots.

A prerequisite to complete this tutorial is to have a working Amazon EKS cluster running Kubernetes version 1.17 or newer. If you need instructions on how to launch an EKS cluster up refer to the Amazon EKS documentation.

Please also note that at the moment of writing the VolumeSnapshot feature is in the beta phase. It is currently graduating GA therefore there might be minor changes in the API definition. Please refer this Kubernetes GA blog for further information.

Step 1: deploy the Amazon EBS CSI driver to your Amazon EKS cluster

As mentioned above, the Amazon EBS Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of Amazon EBS volumes for persistent volumes.

Please follow the steps in this link to configure IAM for the CSI driver.

The process of installing the CSI driver includes creating an IAM policy that you will attach to the worker node’s role. The policy document can be found in this GitHub link.

Install the CSI driver by executing the command below:

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/alpha/?ref=master"

Step 2: create a Storage Class and a Persistent Volume Claim

Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem in Kubernetes provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or has been dynamically provisioned using Storage Classes.

To create a Storage Class, you need execute the command below:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/dynamic-provisioning/manifests/storageclass.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/dynamic-provisioning/manifests/claim.yaml

Execute the command below:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/dynamic-provisioning/manifests/pod.yaml

Execute the kubectl get pods command to verify that your pod is running. You should see something similar to the below:

After you verify that your pod is running, execute the command below and you will be able to see what is being written on the persistent volume.

kubectl exec -it app cat /data/out.txt | tail -n 3

You should see something like the below:

In addition, a new 4GB EBS volume should have been created automatically. You can verify this by navigating to EC2 and then the Volumes tab in the AWS Management Console.

Step 3: install the Volume Snapshot Custom Resource Definitions (CRDs) and Volume Snapshot controller

 Many storage systems like Amazon Elastic Block Store provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).

In order to use the Kubernetes Volume Snapshot feature, you must ensure the following components have been deployed on your Kubernetes cluster:

• Volume Snapshot CRDs
• Volume Snapshot controller

A Custom Resource Definition (CRD) is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

The Volume snapshot controller watches Kubernetes Volume Snapshot CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint.

To install the Kubernetes Volume Snapshot CRDs, execute the following commands:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

To install the Volume Snapshot controller, execute the following commands:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Step 4: create a Volume Snapshot Class and Snapshot

We are now going to create a Volume Snapshot Class and then create a Snapshot from the EBS Volume that we created in step 2.

Create a YAML file for the Volume Snapshot Class and name it snapshot-class.yaml. Copy the below YAML definition from below into the file you created:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: test-snapclass
driver: ebs.csi.aws.com
deletionPolicy: Delete

Execute the command below to create the Volume Snapshot Class:

kubectl apply -f snapshot-class.yaml

Let’s now create the EBS Snapshot.

Create a YAML file for the Volume Snapshot and name it volume-snapshot.yaml. Copy the below YAML definition from below into the file you created.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: test-snapshot
spec:
  volumeSnapshotClassName: test-snapclass
  source:
    persistentVolumeClaimName: ebs-claim

Execute the command below to create the Volume Snapshot:

kubectl apply -f volume-snapshot.yaml

You can verify the creation of a new EBS Snapshot by navigating to the EC2 and then Volumes tab in the AWS Management console. You should see something like the following:

You can also verify the Volume Snapshot creation by executing the following command:

kubectl get volumesnapshot

You should see something similar to the below:

Step 5: Create a persistent volume by restoring the Volume Snapshot

First, you need to create a Persistent Volume Claim (PVC) using the Volume Snapshot. Note that the claim this time is 20 GB in size.

Create a YAML file and Copy the YAML definition below in the volume-from-snap.yaml file.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-restore
  namespace: default
spec:
  storageClassName: ebs-sc
  dataSource:
    name: test-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Execute the following command to create the PVC in your EKS cluster.

kubectl apply -f volume-from-snap.yaml

The PVC will be in the pending state until a pod claims it. Execute the following command to verify the pending state:

kubectl get pvc

The command should return something similar to the following:

Now let’s create a pod using the pvc-restore PVC for persistent storage.

Create a YAML file called task-pv-pod.yaml. Copy the YAML definition below in the task-pv-pod.yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: pvc-restore
      persistentVolumeClaim:
        claimName: pvc-restore
  containers:
    - name: task-pv-container
      image: centos
      command: ["/bin/sh"]
      args: ["-c", "sleep 100000"]
      volumeMounts:
        - mountPath: "/data"
          name: pvc-restore

Execute the following command to create the pod:

kubectl apply -f task-pv-pod.yaml

A new 20GB EBS volume should have been created automatically from the snapshot you created in previous steps. You can verify this by navigating to EC2 and then Volumes tab in the AWS Management Console.

Verify that the persistent volume attached to your pod has the data from the snapshot by executing the command below:

kubectl exec -it task-pv-pod -- cat /data/out.txt | tail -n 3

The command should return something similar to the following:

Conclusion

Snapshot operations is seen as a critical function for stateful workloads. By providing the means to trigger snapshot operations within the Kubernetes API, admins can now handle snapshot use cases without having to go around the Kubernetes API.

In this tutorial, we managed to successfully leverage the EBS CSI driver to add persistent storage to your pods by leveraging EBS snapshots and EBS volumes.

We hope this post helps with your Kubernetes projects. If you have questions or suggestions, please leave a comment.