Using Amazon FSx for Windows File Server as persistent storage on Amazon EKS

Introduction

As the adoption of container environments grows, there is an increasing need to cross-share data between monolith applications and cloud native applications and microservices. As a standard, the Container Storage Interface (CSI) allows exposing storage systems/backends to containerized workloads as persistent storage.

In this blog, we will discuss and create persistent storage for cross-data sharing between Windows-based monolithic applications using Amazon FSx as a shared file system and cloud native microservices applications deployed in an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. By the end of this post, you will have an understanding of how to use a third-party CSI driver to share data across Windows applications and Amazon EKS clusters.

How to achieve this

Currently, the only way to mount an Amazon FSx file systems storage volume to Amazon EKS clusters with Linux nodes is via hostPath. However, this poses potential security concerns. To address this issue, we are going to use SMB CSI Driver, which provides a secure alternative option using Container Storage Interface (CSI) interface. It also allows Amazon EKS clusters to manage the Amazon FSx file systems shares through the CSI. We recommend using the latest version of the CSI driver to manage the Windows File Server as persistent storage on Amazon EKS cluster.

Getting started

We are going to walk you through a step-by-step process on how to deploy the Kubernetes- supported SMB CSI driver csi-driver-smb. This will allow you to use Amazon FSx for Windows File Server as persistent storage for containers running on an Amazon EKS cluster.

Prerequisites and assumptions
To begin, you must have:

Installed and configured AWS CLI. Learn how to install or upgrade your AWS CLI in our documentation.
An existing Amazon EKS cluster with Linux worker nodes. If you don’t currently have a cluster, read our Getting started with Amazon EKS documentation.
Amazon FSx for Windows File Server deployed. Additional information can be found in our Setting up Amazon FSx for Windows File Server documentation.
Installed the latest version of kubectl that aligns to your EKS cluster version.
Amazon FSx for Windows File Server should be accessible within your EKS cluster.

To deploy the SMB CSI driver to an Amazon EKS Linux cluster:

1. Install the CSI driver.

Install Helm on the EKS cluster if not already installed:

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Using Helm, install latest stable version of the driver:

helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v0.6.0

The preceding command will install all the required Kubernetes resources to get the driver running: ServiceAccount, ClusterRole, ClusterRoleBinding, DaemonSet, Deployment, and CSI Driver.

2. To mount the Amazon FSx for Windows File Server volume, create a Kubernetes secret with the credential information.

For file system creation, Amazon FSx works with a managed or self-managed Microsoft Active Directory to provide user authentication and file-/folder-level permissions. By using Kubernetes secrets, you can pass the Active Directory credentials to access the file system of the Amazon FSx. This provides proper permissions to manage the Amazon FSx file system as a persistent storage to the Amazon EKS cluster.

kubectl create secret generic smbcreds --from-literal=username=<user> --from-literal=password=XXXXXXX

Check to see if the secrets have been created.

$ kubectl describe secrets smbcreds
Name: smbcreds
Namespace: default
Labels: <none>
Annotations: <none>

Type: Opaque

Data
====
password: 8 bytes
username: 5 bytes

The preceding output shows the Active Directory credentials used to access the file system for FSx for Windows File Server. You can create as many secrets as the number of users within the Active Directory and mount the different volumes of the Amazon FSx file system.

3. Download the persistent volume manifest.

curl -o pv-smb.yaml https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/pv-smb.yaml

4. Edit the pv-smb.yaml file. Change source in volumeAttributes to have your Amazon FSx server IP address, as shown in the following code:

$ cat pv-smb.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-smb
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
mountOptions:
- dir_mode=0777
- file_mode=0777
- vers=3.0
csi:
driver: smb.csi.k8s.io
readOnly: false
volumeHandle: unique-volumeid # make sure it's a unique id in the cluster
volumeAttributes:
source: "//<FSx FQDN>/share" #Fsx FQDN
nodeStageSecretRef:
name: smbcreds
namespace: default

5. Create the persistent volume (PV) from the YAML file in step 3.

$ kubectl apply -f pv-smb.yaml

6. Download the persistent volume claim (PVC) manifest.

$ curl -o pvc-smb.yaml https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/pvc-smb-static.yaml

7. Create the persistent volume claim from the YAML file in step 6.

$ kubectl apply -f pvc-smb.yaml

8. Confirm that the PV/PVC are created and are Bound.

$kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-smb 100Gi RWX Retain Bound default/pvc-smb 3h41m

$kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-smb Bound pv-smb 100Gi RWX 3h36m

9. Next, deploy a simple application that will create and write log data to a file located on the Amazon FSx file system mounted as PV/PVC SMB. Then share using the following configuration file.

$ cat pv-smb-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
name: pod-smb-reader-writer
spec:
volumes:
- name: smb-pv-storage
persistentVolumeClaim:
claimName: pvc-smb
containers:
- image: busybox
name: smb-writer
command: ["/bin/sh","-c","while true; do echo $(date) Logging data >> /output/output.log; sleep 5; done"]
volumeMounts:
- name: smb-pv-storage
mountPath: /output
- image: busybox
name: smb-reader
command: ['/bin/sh', '-c', 'tail -f /input/output.log']
volumeMounts:
- name: smb-pv-storage
mountPath: /input

10. Deploy the sample POD from the manifest file in step 7.

$ kubectl apply -f pv-smb-pod.yaml 
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-smb-reader-writer 2/2 Running 0 111s

11. Verify that the deployed simple file reader/writer application has the Amazon FSx SMB share specified in the PersistentVolume mounted to the pod pod-smb-reader-writer in step 8.

$ kubectl get pods --all-namespaces -o=json | jq -c \
> '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName:.spec.volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'

{"name":"pod-smb-reader-writer","namespace":"default","claimName":"pvc-smb"}

$ kubectl exec -it pod-smb-reader-writer -c smb-reader /bin/sh
/ #
/ # cd input//input # ls -lhrtl
total 48K 
-rwxrwxrwx 1 root root 0 Jul 2 10:17 Testfile.txt
-rwxrwxrwx 1 root root 48.0K Jul 2 10:18 output.log

12. Display the log file that is being appended to within the SMB share by the smb-reader container executing the following command:

$ kubectl logs pod-smb-reader-writer -c smb-reader
Fri Jul 2 05:58:41 UTC 2021 Logging data
Fri Jul 2 08:43:18 UTC 2021 Logging data
Fri Jul 2 08:43:23 UTC 2021 Logging data
Fri Jul 2 08:43:28 UTC 2021 Logging data
Fri Jul 2 08:43:33 UTC 2021 Logging data
Fri Jul 2 08:43:38 UTC 2021 Logging data
Fri Jul 2 08:43:43 UTC 2021 Logging data
Fri Jul 2 08:43:48 UTC 2021 Logging data

Note: As confirmed from the preceding steps, logs are being written to the SMB share from within EKS cluster pods.

13. Confirm logs written to Amazon FSx by directly accessing the share from outside of EKS cluster as shown in the following screenshot on a Windows File Server having SMB share mounted.

Conclusion

In this post, we have outlined how to configure your Amazon EKS Linux cluster pods to mount a volume through an SMB share hosted in Amazon FSx for Windows File Server. We then used a third-party CSI SMB driver to deploy a sample application to validate the cross-access of the file share. The CSI driver is a free, open-source Kubernetes CSI driver and with no additional cost on the environment. The related costs incurred will be for Amazon EKS cluster and Amazon FSx for Windows File Server in use.

This approach of data persistence covered above is suitable for use cases with workloads built on .NET core or other native Windows apps using file share on Amazon FSx and requires data cross-sharing with cloud native apps/microservices built on EKS platform environment.

This demonstrates the capabilities of extending the use of persistent storage hosted on CIFS/SMB shares with Amazon FSx using Container Storage Interface (CSI) to access Windows shares storage to drive new workloads orchestration and expanding application footprints on the Amazon EKS platform.

Containers

Using Amazon FSx for Windows File Server as persistent storage on Amazon EKS

Resources

Follow