Using AWS Secrets Manager Agent with Amazon EKS

AWS Secrets Manager is a service that you can use to manage, retrieve, and rotate database credentials, application credentials, API keys, and other secrets throughout their lifecycles. You can also use Secrets Manager to replace hard-coded credentials in application source code with runtime calls to retrieve credentials dynamically when needed.

Managing secrets in Amazon Elastic Kubernetes Service (Amazon EKS) environments creates three main challenges: dependency on language-specific AWS SDKs, network dependencies from direct API calls, and complex secret rotation across multiple pods.

The AWS Secrets Manager Agent addresses these challenges by providing a language-agnostic HTTP interface that runs locally within your compute environment. In this post, we show you how to deploy the Secrets Manager Agent as a sidecar container in Amazon EKS to retrieve secrets through HTTP calls.

The Secrets Manager Agent is a client-side agent that you can use to standardize consumption of secrets from Secrets Manager across your AWS compute environments. The agent pulls and caches secrets in your compute environment and allows your applications to consume secrets directly from the in-memory cache through a local HTTP endpoint (localhost:2773).

Instead of making network calls to Secrets Manager, you fetch secret values from the local agent, improving application availability while reducing API calls. Because the Secrets Manager Agent is language agnostic, you can use it across different programming languages without requiring AWS SDK dependencies.

The Secrets Manager Agent implements ML-KEM (Machine Learning-based Key Encapsulation Mechanism) key exchange, which provides additional cryptographic protection for secret retrieval operations. This feature is enabled by default and requires no additional configuration.

This solution uses Amazon EKS Pod Identity for secure authentication to AWS services. Pod Identity provides a simplified way to associate AWS Identity and Access Management (IAM) roles with Kubernetes service accounts, avoiding the need for OpenID Connect (OIDC) provider configuration. IAM principals need GetSecretValue and DescribeSecret permissions to retrieve secrets through the agent.

The Secrets Manager Agent offers protection against server-side request forgery (SSRF). When you install the agent, it generates a random SSRF token and stores it in /var/run/awssmatoken. The agent actively blocks requests that don’t include this token in the X-Aws-Parameters-Secrets-Token header.

In this solution, you deploy the Secrets Manager Agent as a sidecar container in an Amazon EKS pod alongside an NGINX application. The sidecar pattern helps make sure that each pod has its own agent instance, providing isolation and fine-grained security boundaries.

This post demonstrates the Secrets Manager Agent sidecar approach, complementing the AWS Secrets and Configuration Provider (ASCP) guidance covered in Announcing ASCP integration with Pod Identity.

Amazon EKS supports multiple patterns for consuming Secrets Manager secrets. The ASCP for the Kubernetes Secrets Store CSI Driver works well when you want secrets mounted as files and prefer Kubernetes-native secret management. Use the Secrets Manager Agent when you need HTTP-based secret access, want to avoid pod restarts during secret rotation, or need granular refresh control via the refreshNow parameter.

Choosing between Secrets Manager Agent and CSI Driver:

Each secret management approach has specific advantages for different use cases. The Secrets Manager Agent works well for applications requiring HTTP-based access and dynamic secret updates, while the ASCP with CSI Driver is ideal for applications that need file-based secret mounting. Consider your application’s specific requirements, operational patterns, and security needs when choosing between these approaches.

To deploy the solution, you build the agent binary, containerize it, and deploy it to Amazon EKS using Kubernetes manifests with Amazon EKS Pod Identity for secure access to Secrets Manager.

Figure 1: Solution workflow

The workflow of the solution is shown in Figure 1 and includes the following steps:

1. The application container sends GET /secretsmanager/get (localhost:2773) to retrieve secret
2. Secrets Manager Agent checks the local cache to determine if the secret is already stored in memory
   • If not cached, authenticate using Pod Identity to establish secure access to AWS Secrets Manager
   • Assume the IAM role to retrieve the secret from AWS Secrets Manager
   • Return the secret to the sidecar container for caching
3. Return the secret to the application container to fulfill the original request

To build the solution in this post, you must have the following:

• An AWS account
• An Amazon EKS cluster (1.32) in the us-west-2 AWS Region
  • Please note, the AWS Secrets Manager Agent supports the versions of Amazon EKS and Kubernetes available since its initial launch, providing universal compatibility for secure secrets management across cluster versions.
• Amazon EKS Pod Identity Agent installed on your cluster
• A Linux-based virtual machine (VM) or and Amazon Elastic Compute Cloud (Amazon EC2) instance to build the agent.
• AWS Command Line Interface (AWS CLI) version 2
• kubectl installed and configured
• Docker installed

In this section, you install the Secrets Manager Agent. With the agent installed, you then create the Pod Identity association, Secrets Manager binary image, push the binary image to Amazon Elastic Container Registry (Amazon ECR), and create a secret in Secrets Manager.

1. Verify the Pod Identity Agent installation:

kubectl get daemonset eks-pod-identity-agent -n kube-system

2. Create the Pod Identity association using the following commands:

aws eks create-pod-identity-association \   
	--cluster-name <cluster> \   
	--namespace default \   
	--service-account secrets-manager-sa \   
	--role-arn arn:aws:iam::<ACCOUNT_ID>:role/eks-secrets-manager-role

3. Create a file named install and add the following content:

#!/bin/bash -e 
PATH=/bin:/usr/bin:/sbin:/usr/sbin # Use a safe path 
AGENTTARGETDIR=/opt/aws/secretsmanageragent 
AGENTSOURCEDIR=/etc/aws_secretsmanager_agent/configuration 
AGENTBIN=aws_secretsmanager_agent 
TOKENGROUP=awssmatokenreader 
AGENTUSER=awssmauser 
TOKENSCRIPT=/etc/aws_secretsmanager_agent/configuration/awssmaseedtoken 
AGENTSCRIPT=awssmastartup 
if [ `id -u` -ne 0 ]; then     
	echo "This script must be run as root" >&2     
	exit 1 
fi 
if [ ! -r ${TOKENSCRIPT} ]; then     
	echo "Can not read ${TOKENSCRIPT}" >&2     
	exit 1 
fi 
if [ ! -r ${AGENTSOURCEDIR}/${AGENTBIN} ]; then     
	echo "Can not read ${AGENTBIN}" >&2     
	exit 1 
fi 
groupadd -f ${TOKENGROUP} 
useradd -r -m -g ${TOKENGROUP} -d ${AGENTTARGETDIR} ${AGENTUSER} || true 
chmod 755 ${AGENTTARGETDIR} 
install -D -T -m 755 
${AGENTSOURCEDIR}/${AGENTBIN} ${AGENTTARGETDIR}/bin/${AGENTBIN} 
chown -R ${AGENTUSER} ${AGENTTARGETDIR} 
exit 0

4. Build the agent binary on a Linux based instance using the following commands:

#!/bin/bash -e
# Here we are building the Secrets Manager Agent Binary for Linux x86_64 architecture
sudo yum -y groupinstall "Development Tools"
sudo yum install -y git
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env
git clone https://github.com/aws/aws-secretsmanager-agent
cd aws-secretsmanager-agent
mv ../install aws_secretsmanager_agent/configuration
cargo build --release --target x86_64-unknown-linux-gnu

5. Create a file named startup.sh for the entry point and add the following content:

#!/bin/bash 
set -e 

echo "Starting AWS Secrets Manager Agent initialization..." 

# Step 1: Run the install script (equivalent to install-agent init container) 
echo "Running agent installation..." 
/etc/aws_secretsmanager_agent/configuration/install 

# Step 2: Initialize the token (equivalent to token-init init container) 
echo "Starting token initialization..." 
chmod +x 
/etc/aws_secretsmanager_agent/configuration/awssmaseedtoken /etc/aws_secretsmanager_agent/configuration/awssmaseedtoken start 

# Step 3: Start the main secrets manager agent 
echo "Starting secrets manager agent..." 
exec 
/etc/aws_secretsmanager_agent/configuration/aws_secretsmanager_agent

6. Create a file named Docker-eks and add the following content:

FROM public.ecr.aws/amazonlinux/amazonlinux:2023 

# Install required dependencies 
RUN yum install -y ca-certificates bash shadow-utils && yum clean all 
RUN mkdir -p /opt/aws/secretsmanageragent /var/run 

# Copy in the agent binary and configuration scripts 
COPY aws_secretsmanager_agent/configuration/ 
/etc/aws_secretsmanager_agent/configuration 
COPY target/x86_64-unknown-linux-
gnu/release/aws_secretsmanager_agent 
/etc/aws_secretsmanager_agent/configuration 

# Make binaries and scripts executable 
RUN chmod -R +x /etc/aws_secretsmanager_agent/configuration 

# Copy and setup startup script 
COPY startup.sh /startup.sh 
RUN chmod +x /startup.sh 

WORKDIR / 
# Use the startup script as entrypoint 
ENTRYPOINT ["/startup.sh"]

7. Build and publish the image using the following commands:

#!/bin/bash -e 

#Create the ECR Repo ( us-west-2 region) 
aws ecr create-repository --repository-name secrets-manager-agent --image-tag-mutability MUTABLE 

#Build the image 
docker build -f Dockerfile-eks -t secrets-manager-agent:eks . 

#Tag the image 
docker tag secrets-manager-agent:eks <ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/secrets-manager-agent:eks 

# Login into ECR 
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com 

#Push the image  
docker push  <ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/secrets-manager-agent:eks         

When successful, your private Amazon ECR repo will display the published image.

With the image successfully published, you’re ready to create the secret.

1. Create a secret in Secrets Manager by using the AWS CLI to enter the following command in a terminal.

aws secretsmanager create-secret --name MySecret --description "My Secret" \       
	--secret-string "{\"user\": \"my_user\", \"password\": \"my-password\"}"

2. You should see an output like the following:

	{     
    "ARN": "arn:aws:secretsmanager:us-west-
   2:XXXXXXXXXXXX:secret:MySecret-LrBlpm",     
   	"Name": "MySecret",     "VersionId": "b5e73e9b-6ec5-4144-a176-3648304b2d60"     
    }

3. Record the secret Amazon Resource Name (ARN) to use in the next section.

The Amazon EKS application needs an IAM role that grants permission to retrieve the secret you just created.

To create the IAM role:
1. Using an editor, create a file named eks_iam_policy.json with the following content:

 {     
     "Version": "2012-10-17",     
     "Statement": [         
         {             
             "Effect": "Allow",             
             "Principal": {                 
                 "Service": "pods.eks.amazonaws.com"             
             },             
             "Action": [                 
                 "sts:AssumeRole",                 
                 "sts:TagSession"             
             ]         
         }     
     ] 
 }

2. Enter the following command in a terminal to create the IAM role:
aws iam create-role --role-name eks-secrets-manager-role\
--assume-role-policy-document file://eks_iam_policy.json
3. Create a file named iam_permission.json with the following content, replacing <SECRET_ARN> with the secret ARN you noted earlier:

{     
    "Version": "2012-10-17",     
    "Statement": [         
        {             
            "Effect": "Allow",             
            "Action": [                 
                "secretsmanager:GetSecretValue",                 
                "secretsmanager:DescribeSecret"             
            ],             
            "Resource": "<SECRET_ARN>"         
        }     
    ] 
}

4. Enter the following command to create a policy:
aws iam create-policy \
--policy-name get-secret-policy \
--policy-document file://iam_permission.json
5. Record the policy ARN to use in the next step.

6. Enter the following command to add this policy to the IAM role, replacing <POLICY_ARN> with the value you just noted:
aws iam attach-role-policy \
--role-name eks-secrets-manager-role \
--policy-arn <POLICY_ARN>

Here is the sample Kubernetes deployment YAML for installing the Secrets Manager Agent as a sidecar container along with an application container. Replace <ACCOUNT_ID> with your AWS account number and run the code to deploy the NGINX application to the Amazon EKS cluster.

# nginx-with-secrets-agent.yaml 
apiVersion: apps/v1 
kind: Deployment 
metadata:   
	name: nginx-with-secrets-simplified   
	labels:     
		app: nginx-with-secrets-simplified 
	spec:   
		replicas: 1   
		selector:     
			matchLabels:       
				app: nginx-with-secrets-simplified   
		template:     
			metadata:       
				labels:         
					app: nginx-with-secrets-simplified     
			spec:       
				serviceAccountName: secrets-manager-sa       
				containers:         
					- 	name: nginx           
						image: nginx:latest           	
						ports:             
							- containerPort: 80           
							volumeMounts:                  
								- 	name: token-volume               
									mountPath: /var/run         
					- 	name: secrets-manager-agent           
						image: <ACCOUNT_ID>.dkr.ecr.us-west-
2.amazonaws.com/secrets-manager-agent:eks           
						env:             
							- 	name: AWS_TOKEN               
								value: "file:///var/run/awssmatoken"           
						volumeMounts:             
							- 	name: token-volume               
								mountPath: /var/run                    

					volumes:         
						- 	name: token-volume           
							emptyDir: {}          
--- 
apiVersion: v1 
kind: Service 
metadata:   
	name: nginx-service 
spec:   
	selector:     
		app: nginx-with-secrets-simplified   
	ports:     
		- 	port: 80       
			targetPort: 80   
	type: ClusterIP 

--- 
apiVersion: v1 
kind: ServiceAccount 
metadata:   
	name: secrets-manager-sa

kubectl apply -f nginx-with-secrets-agent.yaml

If successful, the pod will run with two active containers.

Now you can run the following command to use the local web server to retrieve the agent. kubectl exec into the app container to retrieve the secret with a REST API call from the web server.
kubectl exec -it nginx-with-secrets-c7945f8dc-7hrzr -c nginx -- sh
curl -v -H “X-Aws-Parameters-Secrets-Token: $(cat
/var/run/awssmatoken)”
‘http://localhost:2773/secretsmanager/get?secretId=<SecretID>'

You should see a Success 200 message and the secret value if IAM permissions are configured correctly.

Run the following cleanup script to delete the resources created for the solution:
bash
chmod +x cleanup.sh
./cleanup.sh

When done, you can check the file named cleanup.sh in the repo to verify that the cleanup was successful:

bash 
#!/bin/bash 
set -e 

echo "Cleaning up EKS resources..." 
kubectl delete deployment nginx-with-secrets-simplified --ignore-not-found=true 
kubectl delete service nginx-service --ignore-not-found=true 
kubectl delete serviceaccount secrets-manager-sa --ignore-not-found=true 

echo "Cleaning up Pod Identity association..." 
# Replace with your actual cluster name 
read -p "Enter your CLUSTER_NAME: " CLUSTER_NAME 

if [ -n "$CLUSTER_NAME" ]; then     
	ASSOCIATION_ID=$(aws eks list-pod-identity-associations \       
                     --cluster-name $CLUSTER_NAME \       
                     --query 'associations[?serviceAccount==`secrets-manager-sa`].associationId' \       
                     --output text)          

if [ -n "$ASSOCIATION_ID" ] && [ "$ASSOCIATION_ID" != 
"None" ]; then         
		aws eks delete-pod-identity-association \           
			--cluster-name $CLUSTER_NAME \           
			--association-id $ASSOCIATION_ID || echo "Pod Identity 
association already deleted"         
		echo "Pod Identity association deleted"     
	else         
		eiifcbfhcfglkdirgljchvkildrknntukkidjtldeekk
echo "No Pod Identity association found"     
	fi 
fi 

echo "Cleaning up IAM resources..." 
# Replace with your actual policy ARN from the create-policy 
output 
read -p "Enter your POLICY_ARN: " POLICY_ARN 

if [ -n "$POLICY_ARN" ]; then     
		aws iam detach-role-policy \       
			--role-name eks-secrets-manager-role \       
			--policy-arn $POLICY_ARN || echo "Policy already detached"

		aws iam delete-policy --policy-arn $POLICY_ARN || echo 
"Policy already deleted" 
fi 
aws iam delete-role --role-name eks-secrets-manager-role || echo 
"Role already deleted" 

echo "Cleaning up secret..." 
aws secretsmanager delete-secret --secret-id MySecret || echo 
"Secret already deleted" 

echo "Cleaning up container image..." 
aws ecr delete-repository \   
	--repository-name secrets-manager-agent \   
	--force || echo "Repository already deleted" 

echo "Cleanup complete!"

In this post, we showed you how to deploy the AWS Secrets Manager Agent as a sidecar container in Amazon EKS. This approach provides a language-agnostic way to retrieve secrets through HTTP calls, reducing SDK dependencies while maintaining security through SSRF protection and IAM-based access controls.

The Secrets Manager Agent can be deployed as either a sidecar container or DaemonSet. Use sidecar deployment for isolated secrets and fine-grained security boundaries and use DaemonSet deployment for shared secrets across multiple applications with optimized resource utilization.

This approach complements existing secret management patterns and provides teams with HTTP-based secret access, immediate refresh control, and consistent interfaces across AWS compute environments.

To learn more, visit the AWS Secrets Manager documentation.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.