Run containerized applications efficiently using Amazon FSx for NetApp ONTAP and Amazon EKS

Kubernetes is a scalable system that offers rapid and easy containerized application deployments for both stateless and long-running, stateful applications. Many Kubernetes applications require a storage system that integrates with the Kubernetes Container Storage Interface (CSI) to create file and block volumes, scale storage, take snapshots, and create clones.

Amazon FSx for NetApp ONTAP supports NetApp’s FlexClone technology, allowing you to clone Kubernetes stateful application data into new volumes and maximize the storage efficiencies of your file system. Cloned data reuses data from the source volume and only requires additional storage when data is written or changed. Because no data copying is required, FlexClone creation is near-instant and has almost no performance impact on your file system. This is important for use cases such as upgrading an application, upgrading Kubernetes clusters, or creating a cloned environment for test purposes.

In this blog, we demonstrate how to use Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon FSx for NetApp ONTAP along with the NetApp Trident CSI driver to deploy a MySQL stateful application. We will also cover FSx for ONTAP rapid cloning and space efficiency capabilities. This blog walks through a typical scenario with a production workload inside an EKS cluster using iSCSI storage, with a spinoff dev/test replica workload deployment. While MySQL is used in this blog as an example, the same principles can be applied to a broad range of stateful, application workloads.

Prerequisites

To create a similar deployment in your own environment, below are the resource prerequisites we use in this solution:

AWS Command Line Interface (CLI) – to interact with the AWS API
Hashicorp Terraform – to provision the sample infrastructure
Kubectl – for interacting with the EKS cluster
Helm – for Trident controller installation
jq – for popular json output manipulation
Amazon FSx for NetApp ONTAP demo sample from Github

Walkthrough

1. Provision Amazon EKS and Amazon FSx for NetApp ONTAP resources using Terraform

EKS Cluster version 1.21
Trident EKS Node preparation: Word node preparation for iSCSI storage outlined in the deploying and configuring Trident EKS with Amazon FSx for NetApp ONTAP user guide.
FSx for ONTAP file system and storage virtual machine (SVM): You can read more about manually creating these resources in the Amazon FSx for NetApp ONTAP getting started guide.

Note: the customized user data to install the iSCSI prerequisite drivers is available in the “terraform folder.” We chose to work with iSCSI block storage for our MySQL database in this solution.

2. Install and configure Trident CSI driver

Install the Kubernetes Snapshot Custom Resources (CRD) and Snapshot Controller Learn more about the snapshot requirements for your cluster in the “How to Deploy Volume Snapshots” Kubernetes blog.

- - Install Trident CSI on EKS
  - Configure Trident CSI backend to connect to the FSx file system

3. Deploy sample MySQL application and clone using Trident CSI driver

- - Create a MySQL stateful application
  - Create a MySQL database and data
  - Create a Snapshot of your MySQL data
  - Clone your MySQL application and data

Setting up your file system and Amazon EKS

In this section of the blog, we create and configure an Amazon FSx for NetApp ONTAP file system with Amazon EKS and the Trident CSI driver for persistent storage.

We can now run all the commands from the machine that we have installed in the prerequisites section.

Create an Amazon EKS cluster and Amazon FSx for NetApp ONTAP file system using Terraform

Clone the sample repository from GitHub and create all relevant resources using the Terraform code in that repository:

$ git clone https://github.com/aws-samples/amazon-eks-fsx-for-netapp-ontap
$ cd amazon-eks-fsx-for-netapp-ontap/terraform
$ terraform init
$ terraform apply -auto-approve

This process can take 10–15 minutes to complete. When finished, the output of the command should look like this:

cluster_endpoint = "https://<GENERATED_ID>.eu-west-1.eks.amazonaws.com"
cluster_id = "fsx-eks-<RANDOM_STRING>"
cluster_name = "fsx-eks-<RANDOM_STRING>"
cluster_security_group_id = "sg-1234567890"
fsx-management-ip = "FSX_MANAGEMENT_IP=198.19.255.37"
fsx-password = "FSX_PASSWORD=<GENERATED_PASSWORD>"
fsx-svm-id = "FSX_SVM_ID=<svm-id123456789>"
fsx-svm-name = "FSX_SVM_NAME=ekssvm"
oidc_provider_arn = "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/<FINGERPRINT>"
region = "eu-west-1"
zz_non_root_volumes_env = "NON_ROOT_VOLUMES=$(aws fsx describe-volumes  --filters Name=storage-virtual-machine-id,Values=<svm-id123456789> | jq -r '.Volumes[] | select(.OntapConfiguration.StorageVirtualMachineRoot==false) | .VolumeId')"
zz_update_kubeconfig_command = "aws eks update-kubeconfig --name fsx-eks-<RANDOM_STRING> --region eu-west-1"

Next, copy and run the AWS CLI command from the update_kubeconfig_command output above and check we’re able to reach the cluster by running kubectl get nodes:

kubectl get nodes 

NAME                                       STATUS   ROLES    AGE   VERSION 
ip-10-0-1-106.eu-west-1.compute.internal   Ready    <none>   1d   v1.21.5-eks- version 
ip-10-0-2-219.eu-west-1.compute.internal   Ready    <none>   1d   v1.21.5-eks-version

Next, install the Kubernetes Snapshot CRDs and Snapshot Controller:

cd..
git clone https://github.com/kubernetes-csi/external-snapshotter 
cd external-snapshotter/ 
kubectl kustomize client/config/crd | kubectl create -f - 
kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f - 
kubectl kustomize deploy/kubernetes/csi-snapshotter | kubectl create -f - 
cd ..

Install Trident CSI on Amazon EKS

Next, download the Trident package and decompress it using tar:

$ wget https://github.com/NetApp/trident/releases/download/v21.10.0/trident-installer-21.10.0.tar.gz 

$ tar -xf trident-installer-21.10.0.tar.gz

Then use helm to deploy the package:

$ helm install trident -n trident --create-namespace  trident-installer/helm/trident-operator-21.10.0.tgz

Next use, kubectl to ensure Trident is up and running.

$ kubectl get pods -n trident 

NAME                                READY   STATUS    RESTARTS   AGE 
trident-csi-678674db66-dzbzw        6/6     Running   0          2m59s 
trident-csi-6vp99                   2/2     Running   0          2m58s 
trident-csi-llng4                   2/2     Running   0          2m58s 
trident-csi-xs2hn                   2/2     Running   0          2m58s 
trident-operator-65b7c4d58b-jh4qf   1/1     Running   0          3m8s

Configure Trident CSI backend to FSx for NetApp ONTAP

Since we are using iSCSI storage for our MySQL database, we set the storage driver name in the Trident backend configuration to ontap-san. You can read more about the different driver types in the Astra Trident documentation. To configure the Trident backend to FSx for ONTAP, we use the file system’s management IP address and fsxadmin credentials.

1. Retrieve your file system’s management IP address by navigating to the Amazon FSx console and selecting either your file system or an SVM belonging to the file system. You can find the management IP address of your SVM on the Summary page for your SVM and find the IP address for your file system in the Network & Security tab of your file system.

2. Create the back-end configuration for the trident driver. Fill in the management IP address, the password, and the SVM name we retrieved from the previous step (Terraform outputs):

$ mkdir temp
$ FSX_MANAGEMENT_IP=<IP_ADDRESS> FSX_PASSWORD=<SVM_PASSWORD> FSX_SVM_NAME=<SVM_NAME/svmeks>    envsubst < manifests/backend-tbc-ontap-san.tmpl > temp /backend-tbc-ontap-san.yaml

$ kubectl create -n trident -f temp/backend-tbc-ontap-san.yaml
 
$ kubectl get tridentbackendconfig -n trident
 
NAME                    BACKEND NAME            BACKEND UUID                           PHASE   STATUS
backend-fsx-ontap-san   backend-fsx-ontap-san   c4701179-6158-4ca9-a78b-1d69ebdb1e5c   Bound   Success

3. Create a Kubernetes Storage Class that will use the backend you’ve just created. Use the existing file of storageclass-fsx-block.yaml in the manifests directory in the repository you’ve cloned in the first step and apply it to the cluster. Note that the backend-type name (ontap-san in our example) should match the storageDriverName field in the TridentBackendConfig you’ve created in the previous step.

4. Create the StorageClass using kubectl:

$ kubectl create -f manifests/storageclass-fsx-block.yaml 
storageclass.storage.k8s.io/fsx-basic-block created
 
$ kubectl get storageclass 
NAME              PROVISIONER            RECLAIM POLICY  VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE 
fsx-basic-block   csi.trident.netapp.io  Delete          Immediate              true                   17h 
gp2 (default)     kubernetes.io/aws-ebs  Delete          WaitForFirstConsumer   false                  21

Create a stateful application

Now that we’ve finished installing the Trident CSI driver and connecting it to our FSx for ONTAP file system, we can create virtually any stateful application using either block or file system volumes on the backend. For this blog, we use MySQL as an example application workload.

Create the MySQL stateful application in Kubernetes

1. Create a Kubernetes PersistantVolumeClaim for the MySQL database. We use the manifest pvc-fsx-block.yaml.

Note: we’re using the fsx-basic-block as the value for the storageClassName field.

$ kubectl create -f manifests/pvc-fsx-block.yaml 
persistentvolumeclaim/mysql-volume created
 
$ kubectl get pvc 
NAME           STATUS   VOLUME                                    CAPACITY   ACCESS MODES   STORAGECLASS      AGE 
mysql-volume   Bound    pvc-26319553-f29b-4616-b2bb-c700c8416a6b   50Gi       RWO            fsx-basic-block   7s

Next, create the MySQL deployment that uses the persistent volume claim created in the previous step. Refer to the claimName field in the deployment manifest.

$ kubectl create -f manifests/mysql.yaml 
secret/mysql-secret created 
deployment.apps/mysql-fsx created 

$ kubectl get pods 
NAME                         READY   STATUS    RESTARTS   AGE 
mysql-fsx-6c4d9f6fcb-mzm82   1/1     Running       0      15d

Populate the MySQL database with data

Use the following commands to connect to our MySQL application by using the password defined in the deployment manifest in the manifests folder that you’ve just created (in our case, “Netapp1!”) to create a database, table, and some data.

kubectl exec -it $(kubectl get pod -l "app=mysql-fsx"-namespace=default -o 
jsonpath='{.items[0].metadata.name}')-- mysql -u root -p 

Enter password: 
… 
mysql> create database fsxdatabase; 
Query OK, 1 row affected (0.01 sec)

mysql> use fsxdatabase;
Database changed 
mysql> create table fsx (filesystem varchar(20), capacity varchar(20), region varchar(20));
Query OK, 0 rows affected (0.04 sec)

mysql> insert into fsx (`filesystem`, `capacity`, `region`) values ('netapp01','1024GB', 'us-east-1'),('netapp02', 
'10240GB', 'us-east-2'),('eks001', '2048GB', 'us-west-1'),('eks002', '1024GB', 'us-west-2'),('netapp03', '1024GB', 'us-east-1'),('netapp04', '1024GB', 'us-west-1'); 
Query OK, 6 rows affected (0.03 sec) 
Records: 6  Duplicates: 0  Warnings: 0

Next, verify that the data has been populated:

mysql> select * from fsx;
+------------+----------+-----------+
| filesystem | capacity | region    |
+------------+----------+-----------+
| netapp01   | 1024GB   | us-east-1 |
| netapp02   | 10240GB  | us-east-2 |
| eks001     | 2048GB   | us-west-1 |
| eks002     | 1024GB   | us-west-2 |
| netapp03   | 1024GB   | us-east-1 |
| netapp04   | 1024GB   | us-west-1 |
+------------+----------+-----------+
6 rows in set (0.00 sec)

Create a snapshot of MySQL data

Now, create a Kubernetes VolumeSnapshotClass so you can snapshot the persistent volume claim (PVC) that is being used for the MySQL deployment. The manifest is under the ‘manifests’ folder, in a file called volume-snapshot-class.yaml.

$ kubectl create -f manifests/volume-snapshot-class.yaml 
volumesnapshotclass.snapshot.storage.k8s.io/fsx-snapclass created

Next, create a snapshot of the existing PVC by creating VolumeSnapshot to take a point-in-time copy of your MySQL data. This will create an FSx efficient snapshot that takes almost no space in the filesystem backend. We use the file volume-snapshot.yaml under the manifest folder, and apply it:

$ kubectl create -f manifests/volume-snapshot.yaml 
volumesnapshot.snapshot.storage.k8s.io/mysql-volume-snap-01 created 


$ kubectl get volumesnapshot
NAME                   READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS
SNAPSHOTCONTENT                                    CREATIONTIME   AGE
mysql-volume-snap-01   true         mysql-volume                           50Gi          fsx-snapclass   snapcontent-bce1f186-7786-4f4a-
9f3a-e8bf90b7c126   13s            14s

Clone MySQL data to a new storage persistent volume

You are now ready to create a clone of the MySQL deployment and data by creating a new deployment and a new PVC based on the data of the VolumeSnapshot just created. This step creates a new FlexClone volume in FSx for ONTAP. As mentioned above, on initial creation, a FlexClone takes almost no space; only a pointer table gets created to the shared data blocks of the volume it is being cloned from.

Create a Kubernetes PVC based on our MySQL snapshot and use the manifest named volume-snapshot.yaml under the manifest folder:

$ kubectl create -f manifests/pvc-from-snapshot.yaml
persistentvolumeclaim/mysql-volume-clone created

$ kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
mysql-volume         Bound    pvc-a3f98de0-06fe-4036-9a22-0d6bd697781a   50Gi       RWO            fsx-basic-block   40m
mysql-volume-clone   Bound    pvc-9784d513-8d45-4996-abe3-7372cd879151   50Gi       RWO            fsx-basic-block   36s

Create a new MySQL application with cloned data

1. Create a new database deployment based on the cloned data and use the file named mysqlclone.yaml under the manifests folder.

2. Create the MySQL deployment using kubectl.

$ kubectl create -f manifests/mysql-clone.yaml 
deployment.apps/mysql-fsx-clone created 

$ k get pods 
NAME                              READY   STATUS    RESTARTS   AGE 
mysql-fsx-6c4d9f6fcb-pglgp        1/1     Running   0          38m 
mysql-fsx-clone-f98f9cc4d-xk7nz   1/1     Running   0          15s

3. Verify that the data was cloned to the new MySQL deployment.

$  kubectl exec -it $(kubectl get pod -l "app=mysql-fsx-clone" --namespace=default -o jsonpath='{.items[0].metadata.name}') -- mysql -u root -p

 
Enter password:
…
mysql> use fsxdatabase;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
mysql> select * from fsx;
+------------+----------+-----------+
| filesystem | capacity | region    |
+------------+----------+-----------+
| netapp01   | 1024GB   | us-east-1 |
| netapp02   | 10240GB  | us-east-2 |
| eks001     | 2048GB   | us-west-1 |
| eks002     | 1024GB   | us-west-2 |
| netapp03   | 1024GB   | us-east-1 |
| netapp04   | 1024GB   | us-west-1 |
+------------+----------+-----------+
6 rows in set (0.00 sec)

Cleaning up

In the root folder, run the following commands to delete any Kubernetes object you’ve created.

kubectl delete -f manifests/mysql-clone.yaml 
kubectl delete -f manifests/mysql.yaml 
kubectl delete pvc mysql-volume mysql-volume-clone 
kubectl delete -f manifests/volume-snapshot.yaml

Next, delete the SVM volumes from the FSx using the AWS CLI.

For your convenience, you can copy the following generated command from the Terraform output field called zz_non_root_volumes_env (see terraform outputs in the “Creating Amazon EKS cluster and FSx for NetApp ONTAP using Terraform” section), or replace the <YOUR_SVM_ID> part with the SVM id created.

NON_ROOT_VOLUMES=$(aws fsx describe-volumes  --filters Name=storage-virtual-machine- 
id,Values=<YOUR_SVM_ID>  | jq -r '.Volumes[] | 
select(.OntapConfiguration.StorageVirtualMachineRoot==false) | .VolumeId') 

for vol in $NON_ROOT_VOLUMES; do aws fsx delete-volume --volume-id $vol; done

Wait until the non-root volumes are deleted from the SVM found in the AWS Management Console.

Wait until the non-root volumes are deleted from the SVM found in the AWS Management Console

Finally, we delete all resources created using Terraform.

cd terraform 
terraform destroy # reply yes to the prompt

Conclusion

In this post, we demonstrated how FSx for ONTAP enables you to run stateful applications on top of Amazon EKS efficiently, while improving data replication speed and simplified cloning operations of Kubernetes persistent volumes. MySQL was used as an example in this blog, but you can see how EKS with FSx for ONTAP can support many different types of containerized applications that require rapid cloning for scaling or dev/test deployments. Kubernetes has additional storage features that are supported by Trident CSI driver for Amazon FSx for NetApp ONTAP, such as volume expansion, volume encryption, topology-aware storage, and more. For more information, please refer to the NetApp Trident documentation.

Thanks for reading this blog post. If you have any comments or questions, feel free to leave them in the comments section.