AWS Secrets Manager controller POC: an EKS operator for automatic rotation of secrets

In an earlier blog post, we showed you how to mount a secret from AWS Secrets Manager using mutating webhooks. If this secret changes when the pod is in running state, the pod can’t capture the change and continues to use the old secret value. One solution is to terminate the pod and then re-create it. But if the number of Kubernetes applications increase with requirements to specific secrets, this quickly becomes tedious.

Kubernetes is highly configurable and extensible. Users and developers can add more resources in the form of custom resource definitions (CRDs) and use operators as an application-specific controller. In simple terms, an operator watches the Kubernetes API and uses custom resources to manage applications and their components.

In this blog post, we share another proof of concept (PoC). It’s built using an operator with its own CRD in an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. The operator automates the pod rotation to enable the secret rotation so that applications inside the pods always have access to latest secret value.

Solution overview

The following Kubernetes constructs are used in this PoC:

• A custom resource definition is an extension of the Kubernetes API.
• An operator is a software extension to Kubernetes that uses custom resource definitions to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.
• Labels are key-value pairs that are attached to objects, such as pods. Labels play an important role in Kubernetes. They are used for selectors for services, deployments, ReplicaSets, nodes, and so on.

Whenever a deployment or pod is created, a CRD mapping label is created with the deployment or pod name and the associated secret ARN.

The following diagram shows the solution flow:

1. AWS Secrets Manager streams secret change events to Amazon CloudWatch Events.
2. CloudWatch Events pushes these events to the attached target, an Amazon Simple Queue Service (Amazon SQS) queue.
3. The Kubernetes operator constantly reads the messages from the SQS queue for any new events.
4. When new events are detected, the operator checks the CRD mapping and provides information about the deployment tied to that secret.
5. The operator restarts the deployment’s pod to reflect the changed secret value.

Prerequisites

• Install Kubebuilder.
• Because the controller components are written in Go, install Go on your system.
• Install Kustomize.
• Your AWS account must have permissions to create resources such as an SQS queue and EventBridge rule.

Deploying the operator and CRD

Follow these steps to deploy the EKS operator for automatic secrets rotation.

1. Clone the GitHub repo into the project path:

git clone https://github.com/aws-samples/aws-secret-sidecar-injector.git && cd secretoperator

2. The CloudFormation template in the repo will create required resources (for example, the rule and queue). An EventBridge rule captures the AWS Secret Manager secrets update API calls and publishes them to an SQS queue. The secrets controller reads these events and responds by restarting the deployment.

After the resources are created, we create the CRD resources that map the Kubernetes deployment names to the AWS Secrets Manager secret name.

Run the following make command to complete the setup:

make install

This setup deploys the controller on the cluster using the registry image  amazon/aws-secrets-manager-secret-rotator.

Testing the solution

Now that we have deployed the building blocks, let’s test the operator.

1. Create a sample CRD and deployment in the default namespace. In this example, we’re going to create a CRD that maps the secret eks-controller-test-secret to a deployment named nginx.

make test_operator

When the controller detects any update calls on this secret ID, it scans for  deployments with matching labels and restarts the deployment pods. This is how the pods pick up the new secret value. In this case, we restart the nginx pods with the  environment: operatortest label.

2. To see the controller redeploy the deployment pods, create a PutSecretValue You can change the sqssecretsecret value in the console or by using the following AWS CLI command:

aws secretsmanager put-secret-value --secret-id eks-controller-test-secret --secret-string testsqssec:newsecret

Run the following command to verify from the pod logs that the new secret has been successfully updated:

kubectl logs -n secretoperator-system $(kubectl get po -n secretoperator-system -o=jsonpath='{.items[*].metadata.name}') -c manager -f

Sample output

Secret ID rotated eks-controller-test-secret
Rotating deployment operatortest
DeleteMessageBatchList output: {
  Successful: [{
      Id: "22211be0-aaaa-bbbb-cccc-733a1ff7bd17"
    }]
}
2021-03-08T20:43:25.715Z    DEBUG    controller-runtime.controller    Successfully Reconciled    {"controller": "secretsrotationmapping", "request": "default/secretsrotationmapping-sample"}

Alternatively, you can pass the watch option to list the sample deployment pods and observe the restart.

kubectl get pods --selector environment=operatortest --watch

Sample output

NAME                            READY   STATUS    RESTARTS   AGE
operatortest-8569978555-cl76n   1/1     Running   0          5s
operatortest-54cd7d5795-6r72c   0/1     Pending   0          0s
operatortest-54cd7d5795-6r72c   0/1     Pending   0          0s
operatortest-54cd7d5795-6r72c   0/1     ContainerCreating   0          0s
operatortest-54cd7d5795-6r72c   1/1     Running             0          2s
operatortest-8569978555-cl76n   1/1     Terminating         0          37s
operatortest-8569978555-cl76n   0/1     Terminating         0          38s
operatortest-8569978555-cl76n   0/1     Terminating         0          39s
operatortest-8569978555-cl76n   0/1     Terminating         0          39s

Cleaning up

Run the following command to clean up all the provisioned resources:

make delete

The output should look like this:

echo "Cleaning up the k8s and aws resources..."
Cleaning up the k8s and aws resources...
kubectl delete -f config/samples/deployment.yaml
deployment.apps "operatortest" deleted
kubectl delete -f config/samples/awssecretsoperator_v1_secretsrotationmapping.yaml
secretsrotationmapping.awssecretsoperator.secretoperator "secretsrotationmapping-sample" deleted
kustomize build config/default | kubectl delete -f -
namespace "secretoperator-system" deleted
customresourcedefinition.apiextensions.k8s.io "secretsrotationmappings.awssecretsoperator.secretoperator" deleted
serviceaccount "secretoperator-operator-serviceaccount" deleted
role.rbac.authorization.k8s.io "secretoperator-leader-election-role" deleted
clusterrole.rbac.authorization.k8s.io "secretoperator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "secretoperator-proxy-role" deleted
clusterrole.rbac.authorization.k8s.io "secretoperator-metrics-reader" deleted
rolebinding.rbac.authorization.k8s.io "secretoperator-leader-election-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "secretoperator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "secretoperator-proxy-rolebinding" deleted
service "secretoperator-controller-manager-metrics-service" deleted
deployment.apps "secretoperator-controller-manager" deleted
aws cloudformation delete-stack --stack-name EKS-Secrets-Operator-Stack
aws cloudformation wait stack-delete-complete --stack-name EKS-Secrets-Operator-Stack
make init
cp config/manager/kustomization_bkup.yaml config/manager/kustomization.yaml
cp config/manager/manager_bkup.yaml config/manager/manager.yaml
rm config/manager/*.bak

Caveats

Although this PoC gives you a Kubernetes native way to automate the pod rotation to enable the secret rotation, you should be aware of the following:

• There are costs for storing and retrieving secrets.
• AWS Secrets Manager has limits on the size (64 KB) of secrets and the rate at which they can be retrieved. For example, the limit for GetSecretValue is 5,000 per second.

Before you implement this solution, be sure to review the costs and limits.

Compared to native Kubernetes secrets, the POC is also more complex. For example, you need to install and maintain the controller. You also need to correctly annotate the deployment or daemonset or statefulset that will consume the secrets from AWS Secrets Manager and the CRDs that the controller looks for to rotate the secrets.

The purpose of this POC is to demonstrate the type of integration that can be achieved between AWS Secrets Manager and Kubernetes. It is not meant to be used in production.

Conclusion

We hope you enjoyed learning about securing secrets for your Kubernetes applications. You can find the source code for this solution on GitHub. If you have questions or other feedback,  open an issue on GitHub. PRs are welcome.