Containers
Adding Storage using OpenEBS on EKS Anywhere
Overview
Amazon EKS Anywhere (EKS Anywhere) is an opinionated and automated deployment of the Amazon EKS Distro that enables users to create and operate Kubernetes clusters on user-managed infrastructure. EKS Anywhere does not include a Container Storage Interface (CSI) driver for persistence. In this post, we setup OpenEBS to provide persistence using the disks available in the Kubernetes cluster.
There are platform specific CSI drivers available when deploying EKS Anywhere on vSphere, Apache Cloud Stack, and Nutanix. These drivers are supported by the teams behind those platforms. In a bare metal deployment, we must consider the Open Source and partner supported CSI drivers, and there are many. In this post we deploy OpenEBS as one such solution. Note that OpenEBS is not related to Amazon Elastic Block Store (Amazon EBS).
OpenEBS is an Open Source project in the Cloud Native Computing Foundation portfolio. It is designed to use the storage available on Kubernetes nodes available as replicated container attached storage to applications within the cluster. We are deploying OpenEBS with Jiva to our cluster to provide storage for a sample application. OpenEBS deploys “HostPath” PersistentVolumes that act as persistent replicas for our Jiva volume, which are attached directly to the container through a PersistentVolumeClaim. OpenEBS supports several “Data Engines”. We are using Jiva here as it is the most flexible. You may want to consider Mayastor if you are focused on NVMe disks.
Note that as of this writing Bottlerocket does not support iSCSI. Many Kubernetes CSI drivers that work on Bare Metal have iSCSI as a prerequisite. For this reason, we are using Ubuntu 20.04 as our cluster operating system.
Prerequisites
For this example we have a previously deployed an EKS Anywhere (0.18.7) Kubernetes (1.28) cluster on bare metal using Ubuntu (20.04). The example cluster is built as three stacked (meaning ETCD and Kubernetes API services running on the same node) nodes.
Walkthrough
In this post we walk you through the following steps.
- Install Open-iSCSI on our nodes
- Install OpenEBS 3.4 on our cluster through Helm
- Set our default StorageClass
- Test
- Dive deeper
- Cleaning up
Install Open-iSCSI
We need to set up Open-iSCSI on each worker node in our data plane. Since our data plane is already established, we SSH into each worker node and add that. For a production environment we would customize an image-builder pipeline to add the open-iSCSI package. EKS Anywhere provides documentation on customizing the image-builder images.
SSH into each worker node and install open-iSCSI:
Install OpenEBS
That is the only SSH related task. From here on out we are working with Helm charts and Kubernetes manifests. The first of these is the OpenEBS Helm chart. We use Helm chart values to enable the Jiva container attached storage data engine.
To verify everything is installed and running, we run kubectl get pods -n openebs, which should show a similar output as follows, and the random characters at the ends are different.
$ kubectl get pods -n openebs
NAME READY STATUS RESTARTS AGE
openebs-jiva-csi-controller-0 5/5 Running 0 37m
openebs-jiva-csi-node-8gh25 3/3 Running 0 37m
openebs-jiva-csi-node-gpfwk 3/3 Running 0 37m
openebs-jiva-csi-node-srxm9 3/3 Running 0 37m
openebs-jiva-operator-69bd68fccc-6kjss 1/1 Running 0 37m
openebs-localpv-provisioner-56d6489bbc-2jd62 1/1 Running 0 37m
openebs-ndm-9ftqw 1/1 Running 0 37m
openebs-ndm-j2jpt 1/1 Running 0 37m
openebs-ndm-kn4zb 1/1 Running 0 37m
openebs-ndm-operator-5d7944c94d-wtlkf 1/1 Running 0 37m
pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-ctrl-65798b7jhdgh 2/2 Running 0 28m
pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-0 1/1 Running 0 28m
pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-1 1/1 Running 0 28m
pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-2 1/1 Running 0 28m
Our next verification step is to check all the StorageClasses through kubectl get sc and we should see the following.
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-device openebs.io/local Delete WaitForFirstConsumer false 12m
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 12m
openebs-jiva-csi-default jiva.csi.openebs.io Delete Immediate true 12m
OpenEBS has deployed three StorageClasses in total. The openebs-device and openebs-hostpath StorageClasses are used to create volumes on the nodes in our data plane. In our example cluster, we have three worker nodes in our data plane, and the default number of replicas Jiva creates is three. When we create a PersistentVolumeClaim to request storage, we use the openebs-jiva-csi-default StorageClass. When we use the openebs-jiva-csi-default StorageClass, Jiva creates three openebs-hostpath based PersistentVolumes as replicas, and a volume attached to our container that acts like a proxy reading and writing to the replica volumes that were created.
Now that we have our openebs-jiva-csi-default StorageClass, we can use it to create dynamic persistent volumes by referencing it as our StorageClass. Let’s look at a sample application.
Set the default StorageClass
We can also mark our openebs-jiva-csi-default Storage Class as the default StorageClass for Kubernetes to use if a PersistentVolumeClaim is created without specifying the storageClassName to use. The following command patches in is-default-class label.
kubectl patch storageclass openebs-jiva-csi-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
If we run kubectl get sc openebs-jiva-csi-default, then we see that openebs-jiva-csi-default is now our default StorageClass.
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-jiva-csi-default (default) jiva.csi.openebs.io Delete Immediate true 5m31s
Testing
To test that our persistence layer is working, we deploy a PersistentVolumeClaim to request storage from Jiva. We test and verify that everything is working by deploying a BusyBox container that doesn’t really do anything. We use that container to create a file and test that it still exists after we delete the container.
First, create busybox.yaml with the following contents.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: example-jiva-csi-pvc
spec:
storageClassName: openebs-jiva-csi-default
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Mi
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: busybox
labels:
app: busybox
spec:
replicas: 1
strategy:
type: RollingUpdate
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- resources:
limits:
cpu: 0.5
name: busybox
image: busybox
command: ['sh', '-c', 'sleep 3600']
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /var/output
name: demo-vol1
volumes:
- name: demo-vol1
persistentVolumeClaim:
claimName: example-jiva-csi-pvc
This file defines our PersistentVolumeClaim, and it asks for the storage to be provisioned with the openebs-jiva-csi-default storage class. Additionally, we define a Kubernetes Deployment of busybox with a volume called demo-vol1 that uses our PersistentVolumeClaim as backing. Then, that volume is mounted into our container at /var/output.
Now let’s deploy our PersistentVolumeClaim and our busybox Deployment.
$ kubectl apply -f busybox.yaml
persistentvolumeclaim/example-jiva-csi-pvc created
deployment.apps/busybox created
If we execute kubectl get pods, then we see our new busybox container come up shortly.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox-86b8c7df5f-59r4m 1/1 Running 0 41m
Once we are in the “1/1 Running” state we can test creating a file in our persistence. We connect to that running pod and create a file.
$ kubectl exec -i -t busybox-[random] -- /bin/sh
Now that we are inside the container we can see that our /var/output directory is empty.
$ ls -l /var/output
total 16
drwx------ 2 root root 16384 Apr 28 17:15 lost+found
The lost+found directory is default and we can ignore it. The fact that it exists means things are working.
Let’s create a file quickly to see that things are truly working:
$ echo world > /var/output/hello
$ ls -l /var/output
total 20
-rw-r--r-- 1 root root 6 Apr 28 18:06 hello
drwx------ 2 root root 16384 Apr 28 17:15 lost+found
Now that our hello file exists, let’s delete this container and test that we get re-attached to our PersistentVolume as expected. First we need to exit that exec session.
$ exit
Now we’re back out or normal prompt, we have exited the Busybox container.
Let’s delete the pod we used previously, and wait for Kubernetes to start a new one. Since our Deployment spec says that there should always be one replica, Kubernetes makes sure of that by starting a new one.
$ kubectl delete pod busybox-[random]
Let’s wait for Kubernetes to start a new one. Once it is at the 1/1 Running state we can resume.
$ kubectl get pods --watch
NAME READY STATUS RESTARTS AGE
busybox-86b8c7df5f-4r679 0/1 ContainerCreating 0 54s
busybox-86b8c7df5f-4r679 1/1 Running 0 54s
Now we connect to our new pod and make sure we find the hello file that we expect.
$ kubectl exec -i -t busybox-[random] -- cat /var/output/hello
world
Success! We’ve proven that we can create the file in the persistent layer outside of our pod and have it restored in a new pod. You can stop here if you’d like, or follow along into a deep dive of how this all works.
Dive deeper
With our Busybox Deployment and PersistentVolumeClaim still deployed we can see some of what OpenEBS and Jiva are doing for us. Let’s look at our PersistentVolumeClaim first.
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
example-jiva-csi-pvc Bound pvc-9523fdf1-4d87-4341-a623-404b9cc4142b 4Gi RWO openebs-jiva-csi-default 75m
The example-jiva-csi-pvc PersistentVolumeClaim we created is Bound to the pvc-9523fdf1-4d87-4341-a623-404b9cc4142b Volume that was created for us. Let’s look at the volumes on our cluster now.
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-275ed8d5-fca9-4b04-bbfc-ba4de0c0fb8b 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-2 openebs-hostpath 78m
pvc-60bca1e6-4548-4bd2-a704-265a4842b52a 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-0 openebs-hostpath 78m
pvc-9523fdf1-4d87-4341-a623-404b9cc4142b 4Gi RWO Delete Bound default/example-jiva-csi-pvc openebs-jiva-csi-default 78m
pvc-ff76dcc5-74ca-4c33-b7ff-c35b63f55769 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-1 openebs-hostpath 78m
What’s this? And why are there four PersistentVolumes created when we only requested one? This is what OpenEBS is doing for us behind the scenes. If we look at the list of CLAIMs in the output, then we can see our default/example-jiva-csi-pvc claim we expected to find. However, we also see three other claims that all look the same, except the claims end with jiva-rep-0, jiva-rep-1, and jiva-rep-2.
This is because OpenEBS and Jiva create three replicas of any volume we request by default. This can be changed by passing —set jiva.defaultPolicy.replicas=n when installing the helm chart. The PersistentVolume mounted to our container is like a proxy. When we write to our volume, the writes are actually replicated to these three volume replicas. If we describe our replica-0 volume, then we see that it’s actually a local volume on the disk.
$ kubectl describe pv pvc-60bca1e6-4548-4bd2-a704-265a4842b52a
Name: pvc-60bca1e6-4548-4bd2-a704-265a4842b52a
Labels: openebs.io/cas-type=local-hostpath
Annotations: pv.kubernetes.io/provisioned-by: openebs.io/local
Finalizers: [kubernetes.io/pv-protection]
StorageClass: openebs-hostpath
Status: Bound
Claim: openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-0
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 4Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [eksa-cp01]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /var/openebs/local/pvc-60bca1e6-4548-4bd2-a704-265a4842b52a
Looking into the details of the PersistentVolume we can see that it is a LocalVolume type, and that it’s actual location on disk for this node is /var/openebs/local/pvc-60bca1e6-4548-4bd2-a704-265a4842b52a. However, we don’t have to worry about this, as OpenEBS manages all of this for us. The /var/openebs path is default, but this is configurable through the helm chart values.
Cleaning up
Now all that’s left to do is clean up. We can tell Kubernetes to delete our PersistentVolumeClaim and Deployment that we applied earlier.
$ kubectl delete -f busybox.yaml
persistentvolumeclaim "example-jiva-csi-pvc" deleted
deployment.apps "busybox" deleted
After a moment those resources are deleted and we can see that all four PersistentVolumes that OpenEBS was managing for us have been removed.
$ kubectl get pv
No resources found
Conclusion
In this post we have seen how OpenEBS can leverage the existing storage on nodes in an EKS Anywhere cluster to provide persistence support for the workloads we deploy. To learn more about EKS Anywhere, you can visit the the EKS Anywhere page and test it out for yourself.