Containers

Leverage AWS secrets stores from EKS Fargate with External Secrets Operator

Secrets management is a challenging but critical aspect of running secure and dynamic containerized applications at scale. To support this need to securely distribute secrets to running applications, Kubernetes provides native functionality to manage secrets in the form of Kubernetes Secrets. However, many customers choose to centralize the management of secrets outside of their Kubernetes clusters by using external secret stores such as AWS Secrets Manager to improve the security, management, and auditability of their secret usage.

Consuming secrets from external secret stores often requires modifications to your application code to support API-based calls to the external store to retrieve a secret at application run time. When running applications on Amazon EKS, you can use Kubernetes’s flexibility to expose secrets directly to pods without requiring any application code changes. One example of how to accomplish this is by using the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver. ASCP uses the Secrets Store CSI driver to expose secrets from AWS Secrets Manager to your pods as a mounted storage volume.

Customers using Amazon EKS to orchestrate their applications often will use AWS Fargate as their compute layer to reduce the management complexity of operating their containerized workloads. For customers who have EKS clusters with AWS Fargate nodes, a different method of consuming external secrets will be required since ASCP with the Secrets Store CSI is deployed to your EKS cluster as a daemonset. As of today, daemonsets are not supported on Fargate.

One such option is the open-source External Secrets Operator (External Secrets) project. External Secrets manages your secrets in a different manner from the Secrets Store CSI driver. Instead of mounting secrets as volumes, External Secrets reads secrets from your external secret store and automatically stores the values as native Kubernetes Secrets in the Kubernetes control plane. External Secrets is installed as a deployment on your cluster and works with Amazon EKS Fargate-only clusters as well as those with Amazon EC2-based nodes.

In this post, I walk through using the External Secrets Operator on an EKS Fargate cluster to consume secrets stored in AWS Secrets Manager. This same method could apply to secrets stored in an AWS Systems Manager Parameter Store but will not be covered directly in this post.

Solution overview

External Secrets will be deployed to your EKS cluster as a standard Kubernetes deployment, allowing Fargate to be used as your container compute provider. Once configured, External Secrets will sync defined secrets from AWS Secrets Manager back to your EKS cluster. The secrets will be automatically synced on a recurring basis to capture any updates or changes, such as periodic credential rotations.

The following diagram outlines how the sync process will occur.

  1. External Secrets will make API calls to AWS Secrets Manager on a recurring basis to copy specified secrets values.
  2. External Secrets will take the copied values and create native Kubernetes Secrets based on the external secret values.
  3. Kubernetes Secrets will be available for consumption by specified applications running on your EKS cluster. Kubernetes RBAC will define what applications can consume the secrets.
  4. Pods will consume the secrets as either volume mounts or environment variables as defined in the pod specifications.

Solution architecture overview comprising Amazon EKS, AWS Fargate, AWS Secrets Manager. External Secrets Operator is used to sync secrets from AWS Secrets Manager to EKS Kubernetes Secrets to be consumed by applications running on Fargate podsFigure 1. Solution overview

Walkthrough

This walkthrough will guide you through the steps of using the External Secrets Operator to sync a secret from AWS Secrets Manager to your EKS Fargate cluster, then consuming that secret in an application pod running on Fargate.

The steps to be outlined in the following sections include:

  • Deploying External Secrets Operator on a EKS Fargate cluster
  • Configuring External Secrets Operator resources
  • Consuming synced Kubernetes Secret in an application pod

Prerequisites

For this walkthrough, you should have the following prerequisites:

Step 1. Create an EKS Fargate profile

With Amazon EKS, Fargate profiles allow administrators to specify which pods in your cluster should be designated to run on Fargate. Profile selectors declare namespaces and optionally labels that will be evaluated against to determine if a pod should be scheduled on Fargate. Fargate profiles can be created via numerous CLI and Infrastructure as Code (IaC) methods. The following example will showcase the eksctl method. To create your Fargate profile using the AWS console, see creating a Fargate profile in the Amazon EKS user guide.

External Secrets will, by default, deploy its resources in a namespace called “external-secrets.” To ensure our External Secrets pods run on Fargate, we will specify this namespace in our Fargate profile selector.

From the terminal where you have eksctl installed, run the following command to create your Fargate profile. Replace “CLUSTERNAME” with the name of your EKS cluster.

eksctl create fargateprofile \
    --cluster <CLUSTERNAME> \
    --name externalsecrets \
    --namespace external-secrets

Once the profile creation is complete, run the following command to verify.

eksctl get fargateprofile --cluster <CLUSTERNAME> -o yaml

The expected output should look similar to the following with your “externalsecrets” profile listed.

- name: externalsecrets
  podExecutionRoleARN: arn:aws:iam::<ACCOUNTNUM>:role/eksctl-blogdemo-cluster-FargatePodExecutionRole-1GY5JZHKR1993
  selectors:
  - namespace: external-secrets
  status: ACTIVE
  subnets:
  - subnet-072cad201ce783be1
  - subnet-086d0835e05be3d89

Step 2. Deploy External Secrets Operator

The External Secrets Operator provides Helm Charts for ease of deployment. The Helm Charts can be found in the project’s Github repository. The following commands will perform a default installation and are further outlined in the External Secrets Getting Started guide.

From the terminal where you have Helm installed, run the following commands.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
   external-secrets/external-secrets \
   -n external-secrets \
   --create-namespace \
   --set installCRDs=true \
   --set webhook.port=9443

Once the deployment is complete, run the following command to verify.

kubectl get pods -n external-secrets

The expected output should look like the following with the External Secrets pod running.

NAMESPACE          NAME                                                READY   STATUS    RESTARTS   AGE
external-secrets   external-secrets-8cdbf85cd-k4hvs                    1/1     Running   0          87s
external-secrets   external-secrets-cert-controller-655b7b7d45-bxh24   1/1     Running   0          87s
external-secrets   external-secrets-webhook-75db54d748-85l8p           1/1     Running   0          87s

Step 3. Set up IAM roles for service accounts

IAM roles for service accounts (IRSA) is a feature of EKS that allows you to map AWS IAM roles to Kubernetes Service Accounts. This feature provides a strategy for managing AWS credentials for your applications running on EKS without having to directly manage static credentials. Reference this AWS blog post, Diving into IAM Roles for Service Accounts, for more information on IRSA.

For our use case with External Secrets, we will be using IRSA to provide AWS credentials to the External Secrets pods for fine-grained access control to our AWS Secrets Manager secret.

The first step in enabling IRSA is to create an IAM OIDC provider for your cluster, if one is not already created. You can follow the steps outlined in create an IAM OIDC provider for your cluster in the Amazon EKS user guide to create an IAM OIDC provider for your cluster or use the following eksctl command. Once the OIDC provider is associated with your EKS cluster, we can create an IRSA service account for External Secrets to use.

From the terminal where you have eksctl installed, run the following command.

eksctl utils associate-iam-oidc-provider --cluster=<CLUSTERNAME> --approve

From the terminal where you have eksctl installed, run the following commands. IAMPOLICYARN will be replaced with the Amazon Resource Name (ARN) of the IAM policy that has permissions to access your AWS Secrets Manager secret.

eksctl create iamserviceaccount \
    --name blogdemosa \
    --namespace default \
    --cluster <CLUSTERNAME> \
    --role-name "blogdemosa" \
    --attach-policy-arn <IAMPOLICYARN> \
    --approve \
    --override-existing-serviceaccounts

The IAM policy associated with the IRSA service account will be used to provide granular access to the secrets stored in AWS Secrets Manager. Limiting the scope of your IAM policy will be important for following security best practices. The following is an example IAM policy used for this blog post. This example policy limits actions to describing and retrieving the single specified secret.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:<REGION>:<ACCOUNTNUM>:secret:prod/blogdemo/mysql1-Ft6k18"
        }
    ]
}

To verify successful creation of the Kubernetes Service Account, run the following kubectl command.

kubectl get sa

You should see the specified service account name listed.

NAME         SECRETS   AGE
blogdemosa   1         2m49s
default      1         18h

You can further inspect the service account by running the following kubectl command.

kubectl describe sa blogdemosa

The output will provide extended service account details. You should see the EKS annotation that references the IAM role created by the eksctl command.

Name:                blogdemosa
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNTNUM>:role/blogdemosa
Image pull secrets:  <none>
Mountable secrets:   blogdemosa-token-8d46d
Tokens:              blogdemosa-token-8d46d
Events:              <none>

With our IRSA service account created, we can now configure External Secrets resources.

Step 4. Configuring External Secrets to sync AWS Secrets Manager secrets

External Secrets provides custom resources (CRDs) for configuring the required operator functionality to sync external secrets to your cluster.

SecretStore is used to define the external secrets store and the authentication mechanisms to access the declared store. ExternalSecret defines what data to fetch from the secret store defined in the SecretStore resource. Reference the External Secret resource model documentation for additional details.

For this example, we are going to create a SecretStore object that references our existing AWS Secrets Manager store. We will specify the IRSA-based service account we previously created to define the AWS credentials that will be used to access the secret store.

Use the following object definition YAML to create the resource using kubectl.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: blogdemo
spec:
  provider:
    aws:
      service: SecretsManager
      region: <REGION>
      auth:
        jwt:
          serviceAccountRef:
            name: blogdemosa

From the terminal where you have kubectl installed and the YAML file created, run the following commands.

kubectl apply -f secretstore.yaml

You can validate your SecretStore was created by running the following kubectl command.

kubectl get secretstore

You should see the created SecretStore referenced with STATUS of Valid.

NAME       AGE   STATUS
blogdemo   82s   Valid

We will now create our ExternalSecret resource, specifying the secret we want to access and referencing the previously created SecretStore object. You will specify your existing AWS Secrets Manager secret name and keys where highlighted. For details on creating an AWS Secrets Manager secret, see Create and manage secrets with AWS Secrets Manager in the AWS Secrets Manager user guide. For additional External Secrets configuration options, see the AWS Secrets Manager provider in the External Secrets documentation.

Use the following object definition YAML to create the resource using kubectl. Replace the highlight areas with your specific Secrets Manager values

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: blogdemo
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: blogdemo
    kind: SecretStore
  target:
    name: blogdemosecret
    creationPolicy: Owner
  data:
  - secretKey: blogdemo-mysql-username
    remoteRef:
      key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name
      property: username #AWS Secrets Manager secret key
  - secretKey: blogdemo-mysql-password
    remoteRef:
      key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name
      property: password #AWS Secrets Manager secret key

Here is the example secret used for this blog as it is stored in Secrets Manager. Only the username and password are being referenced in the previous ExternalSecret definition.

Example of a secret value stored in AWS Secrets Manager being represented as a set of key/value pairs.

Figure 2. AWS Secrets Manager Secret value example

From the terminal where you have kubectl installed and the YAML file created, run the following commands.

kubectl apply -f externalsecret.yaml

Once you have created the ExternalSecret resource, you will be able to view the newly created Kubernetes Secret that is being synced with the Secrets Manager store. Run the following kubectl command.

kubectl describe secret blogdemosecret

The output will describe the secret as in the following:

Name:         blogdemosecret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: a12b24f34338b71e8d118c6a04107b0e

Type:  Opaque

Data
====
blogdemo-mysql-password:  8 bytes
blogdemo-mysql-username:  8 bytes

You now have a synced Kubernetes Secret that can be used within your pod specification for consumption by your applications.

One thing to consider with Kubernetes Secrets is that by default, they are stored in etcd unencrypted in base64 encoded form. With EKS, you are able to leverage AWS Key Management Service (AWS KMS) keys to provide envelope encryption of Kubernetes secrets stored in EKS. This feature, along with the fact that we operate the etcd volumes encrypted at disk-level using AWS-managed encryption keys, provides a defense in-depth strategy for protection of your Kubernetes secrets stored in etcd. Reference this AWS blog post for more information on EKS envelope encryption of Kubernetes Secrets.

Step 4. Consuming secret in pod

Now that External Secrets has synced your AWS Secrets Manager secret to a Kubernetes Secret, you can consume this secret by referencing it in your Pod specification. Refer to Using a Secret in the Kubernetes documentation for additional information on different secret consumption options available.

Use the following pod spec as an example for using your secret via environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    env:
      - name: BLOG_SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: blogdemosecret
            key: blogdemo-mysql-username
            optional: false 
      - name: BLOG_SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: blogdemosecret
            key: blogdemo-mysql-password
            optional: false
  restartPolicy: Always

From the terminal where you have kubectl installed and the YAML file created, run the following commands.

kubectl apply -f podexample.yaml

Once the pod has been deployed, you can directly access the running container’s shell using the following command.

kubectl exec --stdin --tty busybox -- /bin/sh

From there, you can test your environment variables to view your secrets.

/ # echo $BLOG_SECRET_USERNAME
blogdemo
/ # echo $BLOG_SECRET_PASSWORD
Test1234

By exposing your secrets as environment variables, you are able to consume these secrets in your application as required.

Cleaning up

To avoid incurring continued charges, delete any created resources from your cluster, especially those that are running as Fargate pods. Make sure you run the following commands from the same machine used to originally deploy the resources.

kubectl delete -f podexample.yaml
kubectl delete -f externalsecret.yaml
kubectl delete -f secretstore.yaml
eksctl delete iamserviceaccount \
    --name blogdemosa \
    --namespace default \
    --cluster <CLUSTERNAME>
helm uninstall external-secrets -n external-secrets
eksctl delete fargateprofile --cluster <CLUSTERNAME> --name externalsecrets

Also delete any IAM or Secrets Manager resources you created specifically for this walkthrough.

Conclusion

This blog post showcased how customers using EKS Fargate can consume secrets stored in AWS external secrets managers using the External Secrets Operator without requiring any code changes to your applications. This same solution can also apply to EC2 and hybrid EC2/Fargate clusters.

For more information on the External Secrets Operator, see the External Secrets Operator documentation or the External Secrets Github.