Containers

Using Amazon EBS snapshots for persistent storage with your Amazon EKS cluster by leveraging add-ons

Originally, containers were a great fit for stateless applications. However, for many use cases there is a need for persistent storage, without which stateful workloads aren’t possible. Kubernetes first introduced support for stateful workloads with in-tree volume plugins, meaning that the plugin code was part of the core Kubernetes code and shipped with the Kubernetes binaries. That proved quite challenging for vendors who wanted to add Kubernetes support to their storage platform, as these features needed to wait for the standard Kubernetes release cycle.

That led to the development of Container Storage Interface (CSI), which is a standard for exposing arbitrary block and file storage systems to containerized workloads on container orchestration systems like Kubernetes.

Amazon Elastic Block Stock (Amazon EBS) is a cloud block storage service that provides direct access from an Amazon EC2 instance to a dedicated storage volume. Amazon EBS Snapshots provide a simple and secure data protection solution designed to protect your block storage data, such as Amazon EBS volumes.

Support for Amazon EBS initially launched, as mentioned above, as an in-tree volume plugin in Kubernetes.

When the CSI specification was published, we started developing a compatible driver for Amazon EBS. In 2018, we released the EBS CSI driver on GitHub. Since its initial release, it’s gone through various releases and you can find the latest release and its features here.

Moreover, the Amazon EBS CSI driver is now generally available in Amazon EKS add-ons. An Amazon EKS Managed Add-on is software that provides supporting operational capabilities to Kubernetes applications, but isn’t specific to the application. For more details on Amazon EKS Managed Add-ons and the Amazon EBS CSI driver refer to this post.

Solution overview

In this post, we focus on a specific Amazon EBS CSI driver feature: Kubernetes Volume Snapshots. Kubernetes Volume Snapshots lets you create a copy of your Amazon EBS volume at a specific point in time. You can use this copy to bring a volume back to a prior state or to provision a new volume.

From Kubernetes version 1.17 and newer, you can provision and attach Amazon EBS volume snapshots to your pods with the following components:

Prerequisites

  1. A prerequisite to complete this tutorial is to have an existing Amazon EKS cluster running Kubernetes version 1.20 or newer. If you need instructions on how to launch an Amazon EKS cluster, see the Amazon EKS documentation.
  2. To use the snapshot functionality of the Amazon EBS CSI driver, you must install the external snapshotter before the installation of the Amazon EKS Managed Add-on. The external snapshotter components must be installed in the following order:

A Custom Resource Definition (CRD) is an extension of the Kubernetes application programming interface (API) that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, which makes Kubernetes more modular.

The volume snapshot controller watches Kubernetes Volume Snapshot CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint. For more information, see CSI Snapshotter on GitHub.

  1. The process of installing the CSI driver includes creating an Identity and Access Management (IAM) policy and a role that to use as part of the creation of a service account. AWS provides a managed policy for this called AmazonEBSCSIDriverPolicy. Details on how to create an IAM role, service account, and attach the required AWS managed policy can be found here.
  2. Once the IAM Role and Service Accounts have been created, you can continue the installation Amazon EKS add-on support for the Amazon EBS CSI following the documentation.

To verify that everything has been installed correctly run the following command:

eksctl get addon —name aws-ebs-csi-driver —cluster my-cluster

An example output is similar to the following:

NAME                  VERSION              STATUS ISSUES 
aws-ebs-csi-driver    v1.5.1-eksbuild.1    ACTIVE 0 

Walkthrough

Create a Storage Class and a Persistent Volume Claim

Managing storage is a distinct problem from managing compute instances. The Persistent Volume (PV), PersistentVolume, subsystem in Kubernetes provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim. A PV is a piece of storage in the cluster that has been provisioned by an administrator or has been dynamically provisioned using Storage Classes.

  1. Create the StorageClass and VolumeSnapshotClass:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/classes/snapshotclass.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/classes/storageclass.yaml
  1. Deploy the demo pod on your cluster along with the The demo pod is used just as an example and is writing the current date and time in five-second intervals to a file called out.txt.
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/app/claim.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/app/pod.yaml

Run the kubectl get pods command to verify that your pod is running. You should see something similar to the following diagram:

Diagram 1: kubectl get pods result

After you verify that your pod is running, run the command below and you can see what is being written on the persistent volume.

kubectl exec -it app cat /data/out.txt | tail -n 3

You should see something like the following diagram:

Test: Create a test Volume Snapshot Class and Snapshot

3. Create a VolumeSnapshot referencing the PersistentVolumeClaim name:

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ebs-volume-snapshot
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: ebs-claim
EOF

Verify the creation of a new Amazon EBS Snapshot by navigating to the Amazon EC2 and then Volumes tab in the AWS Management console. You should see something to what is presented in the following diagram:

You can also verify the Volume Snapshot creation by running the following command:

kubectl get volumesnapshot

You should see something similar to what is presented in the following diagram:

Validate:

To test this, we delete the application we created in the Setup phase:

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/app/pod.yaml

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/app/claim.yaml

We restore a volume from the snapshot with a PersistentVolumeClaim referencing the VolumeSnapshot in its dataSource:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-snapshot-restored-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: ebs-volume-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF

We can now go ahead and re-create the application:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/snapshot-restore/pod.yaml

A new Amazon EBS volume should have been created automatically from the snapshot you created in previous steps. You can verify this by navigating to Amazon EC2 and then Volumes tab in the AWS Management Console, as presented in the following diagram.

Verify that the persistent volume attached to your pod has the data from the snapshot by running the command below:

kubectl exec app -- cat /data/out.txt | head -n 10

The command should return information similar to that presented in the following diagram:

The Amazon EBS volume created from the snapshot can have a different size from the original volume as shown in the screenshot below:

If you would like to review the examples that we have outlined throughout this post, please also refer to this Github link.

Conclusion

Snapshot operations is seen as a critical function for stateful workloads. By providing the means to run snapshot operations within the Kubernetes API, administrators can now handle snapshot use cases without having to go around the Kubernetes API. In this post, we showed you how to successfully leverage the Amazon EBS CSI driver to add persistent storage to your pods by leveraging Amazon EBS snapshots and EBS volumes. We hope this post helps with your Kubernetes projects.