How to use AWS Secrets & Configuration Provider with your Kubernetes Secrets Store CSI driver
April 29, 2021: We’ve updated the order of the commands in Step 1.
April 23, 2021: We’ve updated the commands in Steps 1 and 5 and in the “Additional Features” section.
AWS Secrets Manager now enables you to securely retrieve secrets from AWS Secrets Manager for use in your Amazon Elastic Kubernetes Service (Amazon EKS) Kubernetes pods. With the launch of AWS Secrets and Config Provider (ASCP), you now have an easy-to-use plugin for the industry-standard Kubernetes Secrets Store and Container Storage Interface (CSI) driver, used for providing secrets to applications that operate on Amazon EKS. You can now use ASCP to provide compatibility for legacy Kubernetes workloads that fetched secrets through the file system or etcd. Previously, you had to store your secrets as plaintext in configuration files, or use encryption with Kubernetes etcd to securely see and access secrets through the filesystem. You also had to write custom code to rotate secrets, resulting in a maintenance challenge. To create secure access to your pods, you had to split up clusters to control access, increasing the operational load.
Now, with ASCP, you can securely store and manage your secrets in Secrets Manager, and retrieve them through your applications that are running on Kubernetes, without the need to write custom code. You also have the added benefit of using AWS Identity and Access Management (IAM) and resource policies on your secret to limit and restrict access to specific Kubernetes pods inside a cluster. This tightly controls which secrets are accessible by which pods. If you have enabled the rotation reconciler feature of the Secret Store CSI driver, ASCP will work with it to retrieve the latest secret from the secret provider. After ASCP is installed and enabled, it helps ensure that your applications always receive the most current version of the secret when the pod starts, enabling you to benefit from the lifecycle management capabilities of Secrets Manager. So, you not only gain the benefit of a natively-integrated secrets management solution, but also the ability to provide configurations in a single provider.
In this post, I will show you how to set up AWS Secrets & Configuration Provider (ASCP) to work with the Secrets Store CSI driver on your Kubernetes clusters. The Secrets Store CSI driver allows Kubernetes to mount secrets stored in external secrets stores into the pods as volumes. After the volumes are attached, the data is mounted into the container’s file system. In this example, the external secret store is Secrets Manager.
This solution includes the following steps, which will be described in more detail in the following sections:
- Restrict access to your pods using IAM roles for service accounts
- Install the Kubernetes secrets store CSI driver
- Install the AWS Secrets & Configuration Provider
- Create and deploy the SecretProviderClass custom resource
- Configure and deploy the Pods to mount the volumes based on the configured secrets
- Load secrets and configurations from the volumes mounted to the container
This solution has the following prerequisites:
- An AWS account.
- An IAM policy with permissions to retrieve a secret from Secrets Manager.
- Your secret stored in Secrets Manager
- An existing EKS Cluster
- A user that can modify your Kubernetes cluster
- AWS CLI and kubectl installed
- Helm and eksctl installed
Deploying the solution
Step 1: Restrict access to your pods using IAM roles for service accounts
You will use IAM roles for service accounts (IRSA) to limit secret access to your pods. By setting this up, the provider will retrieve the pod identity and exchange this identity for an IAM role. ASCP will then assume the IAM role of the pod and only retrieve secrets from Secrets Manager that the pod is authorized to access. This prevents the container from accessing secrets that are intended for another container that belongs to another pod.
Run the following command to turn on Open ID Connect (OIDC). Remember to replace <REGION> and <CLUSTERNAME> with your own values.
To create your service account role, run the following command to associate the policy (from the Prerequisites section) with your service account. Replace <NAMESPACE>, <CLUSTERNAME>, <IAM_policy_ARN>, <SERVICE_ACCOUNT_NAME> with your own values.
Step 2: Install the Kubernetes secrets store CSI driver
From your terminal where you have kubectl installed, run the following helm commands to install the CSI driver.
Next, you need to determine whether you want to turn on automated rotation for the driver using the rotation reconciler feature, or whether you do not need to periodically pull updated secrets.
If you do not need to periodically pull updated secrets, initialize the driver with the following command:
Note: If running an older version of the driver, the flag --set grpcSupportedProviders=”aws” might be required.
If you want to turn on automated rotation for the driver using the rotation reconciler feature which is currently in alpha, use the command as follows (you can adjust the rotation interval as you desire to find an appropriate balance between API call cost consideration and rotation frequency):
To validate that the installer is running as expected, run the following command:
The output should show the following Secrets Store CSI driver pods and custom resource definitions (CRDs) deployed:
Step 3: Install the AWS Secrets & Configuration Provider
The CSI driver allows you to mount your secrets in your EKS Kubernetes pods. To retrieve them from Secrets Manager so the CSI driver can mount them, you need to install the AWS Secrets & Configuration Provider (ASCP). You do this by running the following command in your terminal, which will pull down the installer file without the need to clone the entire repo.
Step 4: Create and deploy the SecretProviderClass custom resource
In order to use the Secrets Store CSI driver, you have to create a SecretProviderClass custom resource. This provides driver configurations and provider-specific parameters to the CSI driver itself. The SecretProviderClass resource should have at least the following components:
To use ASCP, you create the SecretProviderClass to provide a few more details of how you are going to retrieve secrets from Secrets Manager. The SecretProviderClass MUST be in the same namespace as the pod referencing it. The following is an example SecretProviderClass configuration:
Step 5: Configure and deploy the pods to mount the volumes based on the configured secrets
Update your deployment YAML to use the secrets-store.csi.k8s.io driver, and reference the SecretProviderClass resource created previously. This should be saved on your local desktop.
The following is an example of how to configure a pod to mount a volume based on the SecretProviderClass to retrieve secrets from Secrets Manager. In this example, I used NGINX. But for your secret, the mount point and SecretProviderClass configuration will be in the pod deployment specification file.
On pod start and restart, the CSI driver will call the provider binary to retrieve the secret and configurations from Secrets Manager and Parameter Store, respectively. After successfully retrieving this information, the CSI driver will mount them to the container’s file system. You can validate that the volume is mounted properly after a restart by running the following command:
You should get the following response:
Step 6: Load secrets and configurations from the volumes mounted to the container.
Both secrets and configurations will be fetched at pod initialization during the mount operation. This can add a small amount of latency when using the native Kubernetes secrets, but it is similar to the experience of retrieving secrets through a custom or third-party tool. After initialization, your pod will not be impacted. ASCP, along with the rotation reconciler component, will update the values in the mount path and in the Kubernetes secret. The workload pods will watch the file system to track changes and automatically pick-up new credentials. In the case of environmental variables, you will need restart your pods.
The CSI driver can also sync your secrets with Kubernetes secrets. To do this, use the optional secretObjects field to define the desired state of your synced Kubernetes secret objects. The volume mount is required for the sync. The following is an example SecretProviderClass custom resource that will sync a secret from AWS Secrets Manager to a Kubernetes secret:
Supported Kubernetes secret types are the following:
You also have the option of using a deployment YAML to set environment variables in your deployment to reference new Kubernetes secrets. The following is an example deployment YAML that creates an environment variable from a synced Kubernetes secret:
In this post, I walked you through how to set up and configure the new AWS Secrets & Configuration Provider (ASCP) to work with Amazon EKS and any other Kubernetes clusters you are running. By using ASCP, you are able to provide more protection of your secrets with encryption and auto-rotation features. This allows you to focus more on developing your applications rather than on fine-tuning their security configurations.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.