How to use the AWS Workload Credentials Provider for cross-account secret retrieval and prefetching secrets

If you manage secrets across multiple AWS accounts or need faster secret access for latency-sensitive applications, this post shows you how to meet those requirements using two new features of the AWS Workload Credentials Provider (provider). You will learn how to configure role chaining for cross-account secret retrieval and prefetching of secrets to reduce cold-start latency.

By using role chaining, you can access secrets across AWS accounts through a single provider instance by assuming AWS Identity and Access Management (IAM) roles. Prefetching populates the provider’s in-memory cache with secrets at startup so your application can retrieve secrets without waiting for the first request to trigger a network call at runtime.

AWS Secrets Manager stores and rotates credentials, API keys, and other secrets. The AWS Workload Credentials Provider is a client-side HTTP service that retrieves and caches secrets locally. This reduces latency, improves availability during transient failures, and lowers costs. It supports post-quantum TLS by default, requires no language-specific SDK, and works across Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and AWS Lambda. For more details, see the Workload Credentials Provider documentation and GitHub repository.

The Server-Side Requst Forger (SSRF) token prevents unauthorized processes from accessing the provider’s HTTP endpoint. Only applications that can read the token file can retrieve secrets through the provider.

Any identity that can access the provider’s endpoint and SSRF token can retrieve secrets through role chaining. This means users with compute environment access can retrieve cross-account secrets when role assumption is configured. Scope the target role’s permissions to only the secrets required by following the principle of least privilege.

For prefetching, secrets are loaded into the provider’s in-memory cache at startup. Any process that can reach the provider’s localhost endpoint and provide a valid SSRF token can retrieve prefetched secrets from the cache.

Organizations might store secrets in a dedicated AWS account, or need to share one secret across applications in different accounts. Until now, cross-account retrieval through the provider required attaching resource-based policies directly to each secret. Some customers prefer IAM role assumption. Before this feature, you had to deploy multiple provider instances with different credentials or build custom credential-switching logic. The provider now supports both approaches: resource-based policies and IAM role assumption. While role assumption is especially useful for cross-account scenarios, it also helps within the same account when secrets are protected by different customer-managed KMS keys.

When you include the roleArn query parameter in a request, the provider uses AWS Security Token Service (AWS STS) AssumeRole to obtain temporary credentials for the specified role and retrieves the secret with those credentials. The provider creates and caches a separate client for each role ARN, so subsequent requests to the same role reuse the existing client. Each role client maintains its own independent cache.

Note: The source account runs the Workload Credentials Provider and your application. The target account contains the secret you want to retrieve. A single provider instance in the source account can assume roles in one or more target accounts.

• A Workload Credentials Provider built and installed in your environment (see the README for build instructions)
• AWS credentials configured in your compute environment with permission to call sts:AssumeRole on the target role ARN
  • If you also retrieve secrets from the source account through the provider, the credentials need secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions for those secrets
• A secret in a target AWS account that you want to retrieve
• An IAM role in the target account with a trust policy that allows the provider’s identity to assume it

The provider is written in Rust and compiles to a single executable. The following steps are for an RPM-based system such as Amazon Linux 2023:

1. Install build dependencies:

   sudo yum -y groupinstall "Development Tools"

2. Install Rust:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source "$HOME/.cargo/env"

3. Clone the repository and build the provider (use the latest tag available):

   git clone --branch <git tag> https://github.com/aws/aws-workload-credentials-provider.git
   cd aws-workload-credentials-provider
   cargo build --release

The compiled binary is at target/release/aws-workload-credentials-provider.

After building the provider, install it as a system service on your EC2 instance and configure access to the SSRF token.

1. After configuring your config.toml file (see Configuration options section), run the install script to deploy the provider as a systemd service and generate the SSRF token:

   cd aws_workload_credentials_provider_common/configuration
   sudo ./install --config config.toml

2. Add your application user to the aws-wcp-token group. This grants your application permission to read the SSRF token file, which is required for all secret retrieval requests:

   sudo usermod -aG aws-wcp-token <APP_USER>

To install on Amazon ECS, Amazon EKS, or Lambda, see the installation instructions in the GitHub repository.

1. Check that the provider is running:

   curl -v -H \
       "X-Aws-Parameters-Secrets-Token: $(</var/run/awssmatoken)" \
       'http://localhost:2773/secretsmanager/get?secretId=<YOUR_SECRET_ID>'

2. You’ll receive a JSON response with the secret value. If you see a connection refused error, check that the provider process is running. If you see a 401 or 403 error, verify the SSRF token file is readable and that the provider’s IAM credentials have secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions.

The provider’s base IAM identity requires:

• sts:AssumeRole on the target role ARN

The target role requires:

• secretsmanager:GetSecretValue
• secretsmanager:DescribeSecret

Create an IAM role in the target account with a trust policy that allows the provider’s identity in the source account to assume it. Then attach a policy that grants access to the required secrets.

1. Create an IAM role in the target account with a trust policy that allows the provider’s identity in the source account to assume it.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::111111111111:role/WCProviderRole"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

2. Attach a policy to this role that grants access to the secret:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-1:222222222222:secret:MyDatabaseSecret"
        }
    ]
}

Before the provider can assume the role you created in the target account, grant it permission to call sts:AssumeRole. Attach the following policy to the provider’s IAM role in the source account:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole"
        }
    ]
}

Call the Workload Credentials Provider endpoint with the roleArn parameter. The following curl example shows how to retrieve a secret using a different IAM role:

curl -v -H "X-Aws-Parameters-Secrets-Token: $(</var/run/awssmatoken)" 'http://localhost:2773/secretsmanager/get?secretId=MyDatabaseSecret&roleArn=arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole'

The following Python example shows the same operation:

import requests

def get_secret_cross_account():
    role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole"
    url = f"http://localhost:2773/secretsmanager/get?secretId=MyDatabaseSecret&roleArn={role_arn}"

    with open('/var/run/awssmatoken') as fp:
        token = fp.read()

    headers = {
        "X-Aws-Parameters-Secrets-Token": token.strip()
    }

    response = requests.get(url, headers=headers)

    if response.status_code == 200:
        return response.text
    else:
        raise Exception(f"Status code {response.status_code} - {response.text}")

You can configure the maximum number of simultaneous assumed roles with the max_roles option in the provider’s TOML configuration file. The default is 20, and the range is 1–20.

By default, the Workload Credentials Provider populates its cache lazily—the first request for a secret triggers a network call to Secrets Manager. Prefetching reduces this cold-start latency by loading secrets at startup.

You can configure prefetching by adding a [capabilities.secrets_manager.prefetch] section to the provider’s TOML configuration file. You can specify secrets to prefetch in two ways:

• Explicit secrets – List specific secret IDs or ARNs using [[capabilities.secrets_manager.prefetch.secrets]] entries.
• Tag-based discovery – Discover secrets by tag key using [[capabilities.secrets_manager.prefetch.filter_tags]] entries. The provider calls BatchGetSecretValue with tag key filters to find and cache all matching secrets.

You can use both methods together. Each entry optionally accepts a role_arn field for cross-account prefetching through role chaining.

The following permissions are required on the IAM role that performs the prefetch, depending on whether the secrets are in the source account or a target account.

• secretsmanager:BatchGetSecretValue – Required on the source account role for source-account secrets, or on the target role for cross-account secrets
• secretsmanager:ListSecrets – Required when using tag-based discovery (filter_tags), on whichever role is performing the discovery

You can tune prefetch behavior with the following options in the [capabilities.secrets_manager.prefetch] section of your TOML configuration file:

• cache_buffer_ratio – The maximum fraction of the cache to fill per caching client during prefetch, in the range 0.1–1.0. The default is 0.8. For example, if your cache holds 100 secrets, a ratio of 0.8 prefetches up to 80, leaving room for 20 on-demand secrets to be cached.
• max_jitter_seconds – The maximum random delay in seconds before starting the prefetch task, in the range 0–10. The default is 0 (no jitter). Use this to prevent fleet-wide synchronized API calls when deploying across many instances.

The following configuration prefetches two secrets at startup, one from the source account and one from a different account using role chaining:

[capabilities.secrets_manager.prefetch]
secrets = [
    { secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:MySecret-AbCdEf" },
    { secret_id = "cross-account-secret", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" }
]

The following configuration discovers and caches all secrets tagged with the Environment key, and all secrets tagged with the Team key in a different account:

[capabilities.secrets_manager.prefetch]
filter_tags = [
    { key = "Environment" },
    { key = "Team", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]

The following example shows a complete provider configuration that combines both features:

[logging]
log_level = "info"

[capabilities.secrets_manager]
http_port = 2773
region = "us-east-1"

[capabilities.secrets_manager.cache]
ttl_seconds = 300

[capabilities.secrets_manager.prefetch]
cache_buffer_ratio = 0.6
max_jitter_seconds = 5
secrets = [
    { secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:MySecret-AbCdEf" },
    { secret_id = "arn:aws:secretsmanager:us-east-1:222222222222:secret:CrossAccount-AbCdEf", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]
filter_tags = [
    { key = "Environment" },
    { key = "Team", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]

Start the provider with your configuration file:

./aws-workload-credentials-provider sm start --config config.toml

This post showed you how to use role chaining for cross-account secret retrieval and prefetching to reduce cold-start latency. Role chaining simplifies multi-account architectures—a single provider instance can retrieve secrets across accounts using IAM role assumption. Prefetching reduces cold-start latency by populating the provider’s cache before your application makes its first request. Combined, these features let you run the Workload Credentials Provider across multiple accounts with faster secret access.

• AWS Workload Credentials Provider documentation
• Access AWS Secrets Manager secrets from a different account
• GitHub repository for source code and README
• AWS Secrets Manager documentation

Submit feedback in the comments below, or contact AWS Support with questions.