AWS Security Blog

How to use AWS Secrets & Configuration Provider with your Kubernetes Secrets Store CSI driver

April 29, 2021: We’ve updated the order of the commands in Step 1.

April 23, 2021: We’ve updated the commands in Steps 1 and 5 and in the “Additional Features” section.


AWS Secrets Manager now enables you to securely retrieve secrets from AWS Secrets Manager for use in your Amazon Elastic Kubernetes Service (Amazon EKS) Kubernetes pods. With the launch of AWS Secrets and Config Provider (ASCP), you now have an easy-to-use plugin for the industry-standard Kubernetes Secrets Store and Container Storage Interface (CSI) driver, used for providing secrets to applications that operate on Amazon EKS. You can now use ASCP to provide compatibility for legacy Kubernetes workloads that fetched secrets through the file system or etcd. Previously, you had to store your secrets as plaintext in configuration files, or use encryption with Kubernetes etcd to securely see and access secrets through the filesystem. You also had to write custom code to rotate secrets, resulting in a maintenance challenge. To create secure access to your pods, you had to split up clusters to control access, increasing the operational load.

Now, with ASCP, you can securely store and manage your secrets in Secrets Manager, and retrieve them through your applications that are running on Kubernetes, without the need to write custom code. You also have the added benefit of using AWS Identity and Access Management (IAM) and resource policies on your secret to limit and restrict access to specific Kubernetes pods inside a cluster. This tightly controls which secrets are accessible by which pods. If you have enabled the rotation reconciler feature of the Secret Store CSI driver, ASCP will work with it to retrieve the latest secret from the secret provider. After ASCP is installed and enabled, it helps ensure that your applications always receive the most current version of the secret when the pod starts, enabling you to benefit from the lifecycle management capabilities of Secrets Manager. So, you not only gain the benefit of a natively-integrated secrets management solution, but also the ability to provide configurations in a single provider.

Overview

In this post, I will show you how to set up AWS Secrets & Configuration Provider (ASCP) to work with the Secrets Store CSI driver on your Kubernetes clusters. The Secrets Store CSI driver allows Kubernetes to mount secrets stored in external secrets stores into the pods as volumes. After the volumes are attached, the data is mounted into the container’s file system. In this example, the external secret store is Secrets Manager.
 

Figure 1: Overview

Figure 1: Overview

This solution includes the following steps, which will be described in more detail in the following sections:

  1. Restrict access to your pods using IAM roles for service accounts
  2. Install the Kubernetes secrets store CSI driver
  3. Install the AWS Secrets & Configuration Provider
  4. Create and deploy the SecretProviderClass custom resource
  5. Configure and deploy the Pods to mount the volumes based on the configured secrets
  6. Load secrets and configurations from the volumes mounted to the container

Prerequisites

This solution has the following prerequisites:

Deploying the solution

Step 1: Restrict access to your pods using IAM roles for service accounts

You will use IAM roles for service accounts (IRSA) to limit secret access to your pods. By setting this up, the provider will retrieve the pod identity and exchange this identity for an IAM role. ASCP will then assume the IAM role of the pod and only retrieve secrets from Secrets Manager that the pod is authorized to access. This prevents the container from accessing secrets that are intended for another container that belongs to another pod.

Run the following command to turn on Open ID Connect (OIDC). Remember to replace <REGION> and <CLUSTERNAME> with your own values.

eksctl utils associate-iam-oidc-provider --region=<REGION> --cluster=<CLUSTERNAME> --approve #Only run this once

To create your service account role, run the following command to associate the policy (from the Prerequisites section) with your service account. Replace <NAMESPACE>, <CLUSTERNAME>, <IAM_policy_ARN>, <SERVICE_ACCOUNT_NAME> with your own values.

eksctl create iamserviceaccount --name <SERVICE_ACCOUNT_NAME> --namespace <NAMESPACE> --cluster <CLUSTERNAME> --attach-policy-arn <IAM_policy_ARN> --approve --override-existing-serviceaccounts

Step 2: Install the Kubernetes secrets store CSI driver

From your terminal where you have kubectl installed, run the following helm commands to install the CSI driver.

Next, you need to determine whether you want to turn on automated rotation for the driver using the rotation reconciler feature, or whether you do not need to periodically pull updated secrets.

If you do not need to periodically pull updated secrets, initialize the driver with the following command:

helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver 

Note: If running an older version of the driver, the flag --set grpcSupportedProviders=”aws” might be required.

If you want to turn on automated rotation for the driver using the rotation reconciler feature which is currently in alpha, use the command as follows (you can adjust the rotation interval as you desire to find an appropriate balance between API call cost consideration and rotation frequency):

helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set enableSecretRotation=true --set rotationPollInterval=3600s

To validate that the installer is running as expected, run the following command:

kubectl get po --namespace=kube-system 

The output should show the following Secrets Store CSI driver pods and custom resource definitions (CRDs) deployed:

csi-secrets-store-qp9r8         3/3     Running   0          4m
csi-secrets-store-zrjt2         3/3     Running   0          4m
 
kubectl get crd
NAME                                               
secretproviderclasses.secrets-store.csi.x-k8s.io
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io

Step 3: Install the AWS Secrets & Configuration Provider

The CSI driver allows you to mount your secrets in your EKS Kubernetes pods. To retrieve them from Secrets Manager so the CSI driver can mount them, you need to install the AWS Secrets & Configuration Provider (ASCP). You do this by running the following command in your terminal, which will pull down the installer file without the need to clone the entire repo.

Step 4: Create and deploy the SecretProviderClass custom resource

In order to use the Secrets Store CSI driver, you have to create a SecretProviderClass custom resource. This provides driver configurations and provider-specific parameters to the CSI driver itself. The SecretProviderClass resource should have at least the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws                               
  parameters:                                 # provider-specific parameters

To use ASCP, you create the SecretProviderClass to provide a few more details of how you are going to retrieve secrets from Secrets Manager. The SecretProviderClass MUST be in the same namespace as the pod referencing it. The following is an example SecretProviderClass configuration:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    objects:  |
      - objectName: "MySecret2"
        objectType: "secretsmanager"

Step 5: Configure and deploy the pods to mount the volumes based on the configured secrets

Update your deployment YAML to use the secrets-store.csi.k8s.io driver, and reference the SecretProviderClass resource created previously. This should be saved on your local desktop.

The following is an example of how to configure a pod to mount a volume based on the SecretProviderClass to retrieve secrets from Secrets Manager. In this example, I used NGINX. But for your secret, the mount point and SecretProviderClass configuration will be in the pod deployment specification file.

kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: aws-node
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: mysecret2
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: mysecret2
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"

On pod start and restart, the CSI driver will call the provider binary to retrieve the secret and configurations from Secrets Manager and Parameter Store, respectively. After successfully retrieving this information, the CSI driver will mount them to the container’s file system. You can validate that the volume is mounted properly after a restart by running the following command:

kubectl exec -it nginx-secrets-store-inline -- ls /mnt/secrets-store/

You should get the following response:

MySecret2

Step 6: Load secrets and configurations from the volumes mounted to the container.

Both secrets and configurations will be fetched at pod initialization during the mount operation. This can add a small amount of latency when using the native Kubernetes secrets, but it is similar to the experience of retrieving secrets through a custom or third-party tool. After initialization, your pod will not be impacted. ASCP, along with the rotation reconciler component, will update the values in the mount path and in the Kubernetes secret. The workload pods will watch the file system to track changes and automatically pick-up new credentials. In the case of environmental variables, you will need restart your pods.

Additional features

The CSI driver can also sync your secrets with Kubernetes secrets. To do this, use the optional secretObjects field to define the desired state of your synced Kubernetes secret objects. The volume mount is required for the sync. The following is an example SecretProviderClass custom resource that will sync a secret from AWS Secrets Manager to a Kubernetes secret:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  secretObjects:       # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - data:
  - key: username      # data field to populate
    objectName: ACSPSecrets   # name of the mounted content to sync. this could be the object name or the object alias 
  secretName: ACSPEKSSecret     # name of the Kubernetes Secret object
  type: Opaque         # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls

  parameters:          # provider-specific parameters
    objects: |
      array:
        - |
          objectName: "arn:aws:ssm:us-east-1:[ACCOUNT]:parameter/MyConfigValue"
          objectVersion: "1"# [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: "arn:aws:secretsmanager:us-east-1:[ACCOUNT]:secret:MySecret-00AABB"
          objectVersion: "00112233AABB00112233445566778899"
        - |
          objectName: "MyConfigValue2"
          objectType: "ssmparameter"# object types: secretsmanager for secrets and ssmparameter for
configuration values
          objectVersion: "1"
        - |
          objectName: "MySecret2"
          objectType: "secretsmanager"
          objectVersion: "00112233AABB00112233445566778899"
        - |
          objectName: "MySecret3"
          objectType: "secretsmanager"
          objectVersionLabel: "AWSCURRENT"# [OPTIONAL] object version stage, default to
latest if empty

Supported Kubernetes secret types are the following:

  • Opaque
  • Kubernetes.io/basic-auth
  • bootstrap.kubernetes.io/token
  • Kubernetes.io/dockerconfigjson
  • Kubernetes.io/dockercfg
  • Kubernetes.io/ssh-auth
  • Kubernetes.io/service-account-token
  • Kubernetes.io/tls

You also have the option of using a deployment YAML to set environment variables in your deployment to reference new Kubernetes secrets. The following is an example deployment YAML that creates an environment variable from a synced Kubernetes secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: ASCPEKSSecret
              key: username
        volumeMounts:
        - name: mysecret2
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: mysecret2
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"

Conclusion

In this post, I walked you through how to set up and configure the new AWS Secrets & Configuration Provider (ASCP) to work with Amazon EKS and any other Kubernetes clusters you are running. By using ASCP, you are able to provide more protection of your secrets with encryption and auto-rotation features. This allows you to focus more on developing your applications rather than on fine-tuning their security configurations.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Secrets Manager forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Tracy Pierce

Tracy is a Senior Consultant, Security Specialty, for Remote Consulting Services. She enjoys the peculiar culture of Amazon and uses that to ensure every day is exciting for her fellow engineers and customers alike. Customer Obsession is her highest priority both internal and external. She has her AS in Computer Security and Forensics from SCTD, SSCP certification, AWS Developer Associate certification, AWS Solutions Architect Associates certificate, and AWS Security Specialist certification. Outside of work, she enjoys time with friends, her fiance’, her Great Dane, and three cats. She also reads (a lot), builds Legos, and loves glitter.