Migrating from API keys to service account tokens in Grafana dashboards using Terraform

With the release of Grafana 9.4, Amazon Managed Grafana added support for service accounts, which have become the recommended authentication method for applications interacting with Amazon Managed Grafana, replacing the previous API key system.

While API keys are created with a specific role that determines their level of access, service accounts offer a more flexible and maintainable approach. They support multiple tokens, can be enabled or disabled independently, and aren’t tied to individual users, allowing applications to remain authenticated even if a user is deleted. Permissions can be assigned directly to service accounts using role-based access control, simplifying management of long-lived access for non-human entities like applications or scripts.

In this blog post, we walk through how to migrate from API keys to service account tokens when automating Amazon Managed Grafana resource management. We will also show how to securely store tokens using AWS Secrets Manager and automate token rotation with AWS Lambda. All infrastructure is deployed using Terraform, though the pattern can be adapted to your infrastructure-as-code framework of choice.

What are service accounts and tokens?

A service account is designed to authenticate automated tools and systems with Amazon Managed Grafana and is intended for programmatic access. A service account token is a secure credential issued to a service account and can be used to authenticate requests to the Amazon Managed Grafana HTTP API. Multiple tokens can be associated with a single service account, and tokens can be individually revoked or rotated without affecting other services or requiring changes to user accounts.

For a deeper understanding, see the Grafana service account documentation.

Solution overview

In this solution, we show you how to create a service account, reference it in your Terraform stack, and then implement rotation of the token associated with it using Lambda and Secrets Manager as shown in the following diagram:

Architecture diagram illustrating the integration between Terraform, AWS Secrets Manager secret store, and an Amazon Managed Grafana workspace, with secret rotation functionality.

The following are the basic steps to set up the solution.

1. Set up Amazon Managed Grafana with service accounts.
2. Update the secret in Secrets Manager with the token value.
3. Automate resource creation in Amazon Managed Grafana using service account tokens in Terraform.
4. Create a service account and token in your Amazon Managed Grafana workspace.
5. Store the token securely using Secrets Manager.
6. Use Terraform to automate Amazon Managed Grafana resource creation with the token.
7. Automate the rotation of the service account token.

GitHub repo for cloning the code and deploying the Terraform stack.

Prerequisites

Before starting this walkthrough, make sure that you have the following:

• The Terraform CLI (1.2.0+) installed.
• The AWS CLI installed.
• An AWS account with permissions to create resources such as Lambda functions, AWS Identity and Access Management (IAM) roles, Secrets Manager secrets, and Amazon Managed Grafana workspaces.

Solution walkthrough

Use the following steps to set up and configure the solution.

Provision resources using the Terraform stack

The full source code of the solution is in sample-migrate-from-apikeys-grafana and is deployed using Terraform.

1. Clone the repository.

git clone https://github.com/aws-samples/sample-migrate-from-apikeys-grafana.git

2. Initialise a Terraform project.

terraform init

3. Create infrastructure for the secrets and the Amazon Managed Grafana instance.

terraform apply —target=aws_secretsmanager_secret.token —target=aws_grafana_workspace.grafana

Retrieve service account token from the Amazon Managed Grafana workspace

You must have administrative privileges in your Amazon Managed Grafana workspace to perform this step. This applies whether you’re using IAM Identity Center or an external identity provider for authentication.

1. To change a user’s role in AWS IAM Identity Center (console)
   1. Open the Amazon Managed Grafana console.
   2. In the navigation pane, choose Workspaces.
   3. Select the workspace you want to manage.
   4. On the AWS IAM Identity Center, choose the Assigned users tab.
   5. Select the row of the user that you want to modify.
   6. For Action, choose the following:
      • Make admin
   7. Confirm the role change.

2. Select the workspace URL and sign in using your credentials, you should be able to create a service account under the name grafana-sa (or the name of the variable defined in /variables.tf).

3. Assign the Editor role to the service account to allow it to create dashboards and folders. Learn more about service account roles in the Assign roles to a service account in Grafana.
4. After the service account is created, add a service account token to it, again the name should be similar to the one defined in /variables.tf.

Add the token to Secrets Manager and create the rest of the resources

After you complete this step, the access token will be stored in Secrets Manager and will automatically be used in the provider definition during future runs of terraform apply.

1. Copy the service account token.

2. Paste it into the plaintext section of the Secrets Manager secret created in the previous section

3. With the access token stored in Secrets Manager, there is no longer a need to restrict the apply operation to the rotation module using the --target flag. Use the following code to remove the restriction.

   provider "grafana" {
     url  = "https://${aws_grafana_workspace.grafana.endpoint}"
     auth = module.grafana_sa_key_automation.grafana_sa_token
   }

Clean up

To avoid incurring future charges, use the following command to delete unused Amazon Managed Grafana service accounts and Terraform-managed resources run the cli command terraform destroy.

Security notes

To protect the security of your organization, we recommend the following best practices:

• Always follow least privilege principles. Grant the minimum permissions needed to the service account (for example, Editor instead of Admin).
• Make sure that Amazon Simple Queue Service (Amazon SQS) queues, Secrets Manager secrets, and Amazon CloudWatch Logs are encrypted with a customer-managed KMS key if required by your organization.
• Rotate secrets regularly to minimize exposure.

Conclusion

In this post, we demonstrated how to migrate from API keys to Amazon Managed Grafana service account tokens using Terraform, with secure storage in AWS Secrets Manager and optional automated token rotation via AWS Lambda.This modern approach improves security, scalability, and auditing in your automation pipelines.

For more information, see the Amazon Managed Grafana service account documentation.

About the authors