Setting up Amazon Managed Grafana cross-account data source using customer managed IAM roles
Amazon Managed Grafana is a fully managed and secure data visualization service for open source Grafana that enables customers to instantly query, correlate, and visualize operational metrics, logs, and traces for their applications from multiple data sources. Amazon Managed Grafana integrates with multiple Amazon Web Services (AWS) security services, and supports AWS Single Sign-On (AWS SSO) and Security Assertion Markup Language (SAML) 2.0 to meet corporate security and compliance requirements. SAML authentication support lets you use an existing identity provider to offer single sign-on for accessing the Grafana console in your Amazon Managed Grafana workspace, manage access control, search data, and build visualizations. Read “Amazon Managed Grafana supports direct SAML integration with identity providers” to learn about Amazon Managed Grafana support for direct SAML integration.
As more AWS customers adopt a multi-account strategy, they’re using AWS Organizations and AWS Control Tower to build their landing zones. AWS Organizations helps you centrally provision, govern, and manage multi-account environments as you grow and scale AWS resources. AWS Control Tower uses the best practices and recommendations for setting up multi-account environments to secure, isolate, and manage your workloads.
Using long-term credentials as access keys is one approach to achieving cross-account access, but this approach poses security risk to comply with credentials rotation. As part of best practice, using an AWS Identity and Access Management (IAM) customer managed role to achieve cross-account access is a better approach to accessing data from AWS service sources in other AWS accounts. Using IAM customer managed roles creates temporary credentials using AWS Security Token Service (AWS STS).
In this blog post, we explain how to set up Amazon Managed Grafana to retrieve metrics from Amazon Managed Service for Prometheus and Amazon CloudWatch from different AWS accounts running container workloads using customer managed IAM roles.
The following figure illustrates the architecture to set up and retrieve metrics from Amazon Managed Service for Prometheus and Amazon CloudWatch sources from different AWS accounts to Amazon Managed Grafana.
To deploy this solution, you must have the following prerequisites:
- AWS Control Tower deployed in your AWS environment in the management account. If you have not already installed AWS Control Tower, follow the Getting Started with AWS Control Tower documentation, or you can enable AWS Organizations in the AWS Management Console account and enable AWS SSO.
- An AWS account under AWS Control Tower called Workloads Account A provisioned using the AWS Service Catalog Account Factory product AWS Control Tower Account vending process.
- An AWS account under AWS Control Tower called Workloads Account B provisioned using the AWS Service Catalog Account Factory product AWS Control Tower Account vending process.
- An AWS account under AWS Control Tower called Grafana Account provisioned using the AWS Service Catalog Account Factory product AWS Control Tower Account vending process.
- Install the following tools:
Set up AWS account profiles
Because we will be running this in three different accounts, you must make sure that you have profiles with credentials for both AWS accounts configured in
~/.aws/config files. For example:
Set up environment variables and create the Amazon EKS clusters
Use the following commands to set up required environment variables across all three AWS accounts:
Use the following commands to create an Amazon Elastic Kubernetes Service (Amazon EKS) clusters in workload accounts A and B:
Use the following commands to switch contexts across the two Amazon EKS clusters as needed:
Note: As a best practice, create a VPCE endpoint for Amazon Managed Prometheus in VPCs for both of the workload accounts in which you will be deploying Amazon EKS clusters.
Setting up an IAM role and an Amazon Managed Prometheus workspace in the Workload A account
An Amazon Managed Prometheus workspace is the conceptual location where you ingest, store, and query Prometheus metrics that were collected from application workloads, isolated from other Amazon Managed Prometheus workspaces. One or more workspaces may be created in each Region within the same AWS account, and each workspace can be used to ingest metrics from multiple workloads that export metrics in a Prometheus-compatible format.
To create an Amazon Managed Prometheus workspace, use the following AWS CLI command:
Now, let’s create an IAM role
AMGPrometheusDataSourceRole with permissions to access Amazon Managed Prometheus workspace in the Workload A account. This IAM role will be used by Amazon Managed Grafana to query metrics from Amazon Managed Prometheus workspace.
First, we will create the IAM trust policy. Add the following content in a file called
amp_trust_policy.json. This IAM trust policy will ensure that we have permissions to assume
AMGPrometheusDataSourceRole from the Amazon Managed Grafana account.
To create the IAM role, run the following command:
To create the IAM permissions policy, add the following content in a file called
Now we can attach the preceding Amazon Managed Prometheus workspace permissions policy to IAM role
Test the policy attached to the IAM role
To get the
AMGPrometheusDataSourceRole IAM role ARN:
This will be used later in the article.
Running a container workload to push metrics to Amazon Managed Prometheus workspace in Workload A account
AWS Load Balancer Controller setup
Switch the Kubernetes context to the Workload A Amazon EKS cluster. Run the following commands to deploy the AWS Load Balancer Controller into your cluster:
Deploy a demo (Yelb) app
Read “Service connectivity inside and outside the mesh using AWS App Mesh (ECS/Fargate)” to learn about the Yelb demo app. In order to deploy our demo app called Yelb, run the following:
To get the URL of the load balancer for carrying out testing in your browser, use the following command:
Send metrics to Amazon Managed Prometheus in the Workload A account
The following shell script can be used to perform the following actions:
- Creates an IAM role with an IAM policy that has permissions to remote-write into an Amazon Managed Prometheus workspace
- Creates a Kubernetes service account that is annotated with the IAM role
- Creates a trust relationship between the IAM role and the OIDC provider hosted in your Amazon EKS cluster
Amazon Managed Service for Prometheus does not directly scrape operational metrics from containerized workloads in a Kubernetes cluster. It requires users to deploy and manage a standard Prometheus server, or an OpenTelemetry agent—such as the AWS Distro for OpenTelemetry Collector—in their cluster to perform this task.
Run the following commands to deploy the Prometheus server on the Amazon EKS cluster:
Create a file called
amp_ingest_override_values.yaml as show in the following:
Run the following command to install the Prometheus server configuration and configure the remoteWrite endpoint:
Running a container workload and setting up CloudWatch Container Insights in the Workload B account
Use the following command to switch the profile to a Kubernetes cluster in the Workload B account:
Repeat the following steps from the previous section in Workload A account to run the Yelb app in the Workload B account:
- AWS Load Balancer Controller setup (replace
WLDBwhile following the steps for Workload B).
- Deploy a demo app
Setting up an IAM role in a Workload B account
Next, let’s create an IAM role with permissions to access CloudWatch in the Workload B account.
First, we must create the IAM trust policy. Add the following content in a file called
cw_trust_policy.json. This IAM trust policy will ensure we have permissions to assume
AMGCloudWatchDataSourceRole from the Amazon Managed Grafana account.
To create the IAM role, run:
To create the permissions policy, add the following content in a file called
Now attach the preceding permissions policy to the IAM role
Test the policy attached to the IAM role
To get the
AMGCloudWatchDataSourceRole IAM role ARN, which will be used later in this article:
Send metrics via CloudWatch Container Insights to Amazon CloudWatch in Workload B account
Add the necessary policy to the IAM role for your worker nodes
In order for CloudWatch to get the necessary monitoring info, we must install the CloudWatch Agent to our Amazon EKS cluster.
First, we must ensure that the role name our workers use is set in our environment:
We will attach the policy to the nodes IAM role:
Finally, let’s verify that the policy has been attached to the IAM role:
Now we can proceed to the actual install of the CloudWatch Container Insights.
Installing CloudWatch Container Insights
To complete CloudWatch Container Insights set install, you can follow the quickstart instructions in this section. Run the following command:
The command will:
- Create the namespace amazon-cloudwatch.
- Create the necessary security objects for both DaemonSet:
- Deploy CloudWatch agent, which is responsible for sending the metrics to CloudWatch, as a DaemonSet.
- Deploy fluentd (responsible for sending the logs to CloudWatch) as a DaemonSet.
- Deploy ConfigMap configurations for both DaemonSets.
Find additional information and manual install steps in the documentation.
Verify that the DaemonSets have been deployed by running the following command:
The output looks like the following:
That’s it. It’s that simple to install the agent and get it up and running. You can follow the manual steps in the full documentation, but with the quickstart the daemon deployment is simplified.
Verifying CloudWatch Container Insights in the console
To verify that data is being collected in CloudWatch, launch the CloudWatch Containers UI in a browser using the link generated by the following command:
Grafana account setup
Setting up a cross account role for Amazon Managed Grafana in a Grafana account
To query metrics from Amazon Managed Prometheus workspace in Workload A account and access CloudWatch Logs from Workload B and to display in a Grafana dashboard in a Grafana account, we will create a new IAM role in a Grafana account that can assume the existing IAM roles AMGPrometheusDataSourceRole and AMGCloudWatchDataSourceRole.
First, we must create the IAM trust policy in a Grafana account, and add the following content in a file called
We can now proceed to create IAM role
Now, let’s add permissions to this new role to assume IAM roles
AMGCloudWatchDataSourceRole in Workload A and Workload B accounts:
To validate if policy has been attached to the
AWS SSO in the management account
To use Amazon Managed Grafana in a flexible and convenient manner, we chose to leverage AWS SSO for user management. AWS SSO is available once you’ve enabled AWS Organizations manually, or it’s auto-enabled while setting up AWS Control Tower.
Amazon Managed Grafana integrates with AWS SSO to provide identity federation for your workforce. With Amazon Managed Grafana and AWS SSO, you are redirected to an existing company directory to sign in with existing credentials. Then, you are seamlessly signed in to the Amazon Managed Grafana workspace. This ensures that security settings, such as password policies and two-factor authentication, are enforced. Using AWS SSO does not impact your existing IAM configuration.
Create Amazon Managed Grafana workspace in a Grafana account and query metrics from Amazon Managed Prometheus workspace in Workload A, and CloudWatch in Workload B
You can spin up on-demand, autoscaled Grafana workspaces (virtual Grafana servers) that let you create unified dashboards across multiple data sources. Before we can use Amazon Managed Grafana for the following example, we must set it up. In the following, we’re using the AWS Management Console to walk through the required steps and comment on considerations when performing each step.
After selecting Create new workspace in the Amazon Managed Grafana console landing page, name the new workspace and optionally add a description:
In this step, you also must enable AWS SSO for Amazon Managed Grafana, because this is how we manage user authentication to Grafana workspaces. Select Customer managed as the permission type and choose the Amazon Managed Grafana-cust-managed-role created in the previous step:
Select Next in the Service managed permission settings screen without any selection. To create the Amazon Managed Grafana workspace in the next screen, choose Create workspace without any selections.
By default, the SSO user has Viewer permissions. Because we will be adding new data sources and creating a dashboard in Amazon Managed Grafana, you would want to update the user type as Admin.
Under Authentication, select Configure users and user groups, select the SSO user you want to use to log in to Grafana, and choose Make admin button:
Query metrics in Grafana account from Amazon Managed Prometheus workspace in Workload A account
Once you’re logged into the Amazon Managed Grafana console, add the Amazon Managed Prometheus datasource by selecting Configuration, Data sources.
Choose Add data source, Prometheus:
Configure Prometheus data source
For Name, let’s add AMPDataSource (or a name you choose).
For URL, add the Amazon Managed Prometheus workspace
remote_write URL from Workload A without the
api/v1/remote_write on the end.
Enable SigV4 auth.
For SigV4Auth Auth Details:
- For Authentication Provider, choose Workspace IAM Role from the dropdown.
- For Assume Role ARN, use ARN of
AMGPrometheusDataSourceRolefrom environment variable
- For Default Region, choose the region where you created the Amazon Managed Prometheus workspace.
Select Save & test. You should be shown that the data source is working.
The Amazon Managed Prometheus data source is authenticated through a SigV4 protocol. Grafana (7.3.5 and higher) has the AWS SigV4 proxy built-in as a plugin, which makes this possible.
To query metrics, choose Explore and enter the query
apiserver_current_inflight_requests. You will get a screen similar to the following, which shows that we are able to query metrics successfully from the Amazon EKS cluster through the Amazon Managed Prometheus workspace:
You can also import existing dashboard by selecting the plus sign (+) and selecting Import.
In the Import screen, type 3119 in Import via grafana.com and choose Import.
Select the AMPDataSource from the dropdown at the bottom and select Import.
Once the import completes, the Grafana dashboard with show metrics from the Amazon EKS cluster through Amazon Managed Prometheus data source:
Query metrics in the Grafana account from Amazon CloudWatch in Workload B account
Configure CloudWatch data source
Select AWS services, which will present a screen showing the available AWS data sources:
Select CloudWatch from the Service dropdown and select the AWS Region where the Yelb application is deployed:
Select Go to settings:
Configure CloudWatch data source
- For Name, use CloudWatchDataSource (or a name you choose).
- For Assume Role ARN, add the ARN of
AMGCloudWatchDataSourceRolefrom environment variable
- Choose the Region where you created CloudWatch.
- Add ContainerInsights/yelb for Namespace of Customer Metrics.
- Select Save & test button, which should show Data source working.
To use CloudWatchDataSource, you can use Explore. The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor. You select which API you want to query with using the query mode switch on top of the editor.
- Ensure QueryMode is selected as CloudWatch Metrics.
- Select ContainerInsights as Namespace.
- Choose metrics from the Metric Name dropdown.
- For Dimensions, select Namespace = yelb.
- Disable the Match exact option.
- Select Run Query.
When viewing CloudWatch Metrics in Amazon Managed Grafana, you will be shown a screen similar to the following:
Use the following commands to clean up the Workload B cluster:
Clean up Workload A:
In the Grafana account, navigate to the Amazon Managed Grafana service and delete the Amazon Managed Grafana workspace called
In this blog post, we demonstrated how to set up Amazon Managed Grafana to retrieve metrics from Amazon Managed Service for Prometheus and CloudWatch Container Insights from different AWS accounts running container workloads using customer managed IAM roles.
With Amazon Managed Grafana, you can use Grafana without the operational management of maintaining infrastructure resources, and let AWS take care of the undifferentiated heavy lifting.