Amazon Managed Grafana announces support for Grafana version 10.4

We are excited to announce support for Grafana version 10.4 in Amazon Managed Grafana. You can now run Grafana version 10.4 on a secure, scalable, highly available, and fully managed service. This update gives you access to key features introduced in the open-source Grafana versions 9.5 to 10.4, including Correlations, Subfolders, Transformations redesign, and new visualization panels. It also introduces new APIs for Grafana Service Accounts and Service Account Tokens, which replace the traditional Grafana API keys. When compared to the legacy Grafana API keys, the Service Accounts Tokens approach offers better security and improved automation abilities by integrating with native AWS APIs.

In this blog post, we will take a closer look at the new Amazon Managed Grafana v10.4 features to understand the changes.

Walkthrough

Follow us as we introduce the new capabilities in Amazon Managed Grafana version 10.4, including Correlations for seamlessly linking metrics and logs, Subfolders for better organization of dashboards, and new visualization panels like Trend, Datagrid, and XY Chart. We’ll also explore enhancements to navigation, Service Accounts and APIs for automation, and the Grafana Alerting preview.

We will start with creating a new workspace with version 10.4, deploy an example to get metrics into Amazon Managed Service for Prometheus and logs into Amazon CloudWatch. This will allow us to explore key new features.

We will leverage Amazon Elastic Kubernetes Service (Amazon EKS) cluster metrics from the AWS Observability Accelerator and highlight relevant portions of it. The AWS Observability Accelerator provides opinionated infrastructure as code to set up observability for AWS environments using managed services like Amazon Managed Service for Prometheus, Amazon Managed Grafana, and AWS Distro for OpenTelemetry, with curated metrics, logs, traces collection, alerting rules, and Amazon Managed Grafana dashboards.

Creating a new workspace

When creating a workspace in Amazon Managed Grafana, you now have the possibility to select between multiple versions including 8.4, 9.4 and now 10.4. The entire process of creating a new workspace can be found in the Amazon Managed Grafana documentation.

Figure 1 – Selecting a Grafana version for a new workspace

Alternatively, you can create a new workspace with your favorite infrastructure as code tools like AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), Terraform or the AWS Command Line Interface (AWS CLI).

Below are examples for creating a workspace with the AWS CLI and Terraform.

Using the AWS CLI

1. Creating an AWS Identity and Access Management (IAM) role for the workspace

Here, we create an IAM role that will be assumed by the Amazon Managed Grafana workspace, enabling it to access AWS data sources like Amazon Managed Prometheus, CloudWatch, and AWS X-Ray. In addition to these data sources, Amazon Managed Grafana supports several other default data sources, as listed on built-in data sources.

cat <<EOF > assume-role.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GrafanaAssume",
            "Effect": "Allow",
            "Principal": {
                "Service": "grafana.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

GRAFANA_WORKSPACE_ROLE_ARN=$(aws iam create-role --role-name grafana-v10-demo \
--assume-role-policy-document file://assume-role.json --query 'Role.Arn' --output text)

# Amazon CloudWatch
aws iam attach-role-policy \                                                       
--role-name $GRAFANA_WORKSPACE_ROLE_ARN \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonGrafanaCloudWatchAccess

# AWS X-Ray
aws iam attach-role-policy \                                                       
--role-name grafana-v10-demo \
--policy-arn arn:aws:iam::aws:policy/AWSXrayReadOnlyAccess

# Amazon Managed Service for Prometheus
cat <<EOF > aps-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aps:ListWorkspaces",
                "aps:DescribeWorkspace",
                "aps:QueryMetrics",
                "aps:GetLabels",
                "aps:GetSeries",
                "aps:GetMetricMetadata"
            ],
            "Resource": "*"
        }
    ]
}
EOF

APS_POLICY_ARN=$(aws iam create-policy \
--policy-name grafana-v10-demo-aps-policy \
--policy-document file://aps-policy.json \
--query 'Policy.Arn')

aws iam attach-role-policy \
    --role-name $GRAFANA_WORKSPACE_ROLE_ARN \
    --policy-arn $APS_POLICY_ARN

2. Creating the workspace

The following code snippet configures the workspace to access resources within the current AWS account. Alternatively, you can configure it at the AWS Organization level. The code reuses the role created in the first step, granting permissions to access CloudWatch, Amazon Managed Service for Prometheus, and X-Ray. For logging into the workspace, we use AWS IAM Identity Center (successor to AWS SSO) as the authentication method for the workspace. Finally we enable the plugin management feature to allow workspace admins to install or uninstall plugins from the plugin catalog.

Check Managing Workspaces, Users, and Policies for more information on how to setup workspaces.

aws grafana create-workspace \
--account-access-type CURRENT_ACCOUNT \
--workspace-role-arn $GRAFANA_WORKSPACE_ROLE_ARN \
--authentication-providers AWS_SSO \
--permission-type SERVICE_MANAGED \
--grafana-version 10.4 \
--configuration '{"plugins": {"pluginAdminEnabled": true}}'

Using Terraform

An alternative approach to deploying Amazon Managed Grafana is through Terraform. We will utilize an example from the AWS Observability Accelerator for Terraform to streamline the deployment of a workspace configured with CloudWatch, X-Ray, and Amazon Managed Service for Prometheus as data sources. Additionally, this workspace will be set up to leverage AWS IAM Identity Center for user authentication. Visit Creating a new Amazon Managed Grafana Workspace for more information on the example.

Replace <AWS_REGION> with your current AWS Region.

git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
cd terraform-aws-observability-accelerator/examples/managed-grafana-workspace
terraform init
export TF_VAR_aws_region=<AWS_REGION>
terraform apply 

After a few minutes your workspace should be ready to use.

Updating workspace version to v10.4

If you are an existing user of Amazon Managed Grafana, you can directly migrate your current workspace from 9.4 to 10.4 in the AWS console. This can be also done with the AWS CLI as shown in the example below. However, it’s important to note that the migration process is not reversible. We recommend testing the upgrade in a non-prod environment, before upgrading your production workspace.

If you are still running 8.4, the upgrade will be done in two steps, first to 9.4 and then 10.4.

aws grafana update-workspace-configuration \
--workspace-id <GRAFANA_WORKSPACE_ID> \
--configuration <GRAFANA_WORKSPACE_CONFIGURATION>
--grafana-version 9.4

aws grafana update-workspace-configuration \
--workspace-id <GRAFANA_WORKSPACE_ID> \
--configuration <GRAFANA_WORKSPACE_CONFIGURATION>
--grafana-version 10.4

Note: We recommend that you review the breaking changes before upgrading the workspace version. Some plugins may need an update, to work properly with Grafana version 10.

Migrating Grafana content to version 10.4

In addition to manually migrating the content such as data sources, folders, and dashboards from your self hosted Grafana yourself, you can easily do this with the amazon-managed-grafana-migrator. This solution can be useful to migrate from self-managed Grafana to Amazon Managed Grafana or to test in a lower environment easily, before upgrading a production Amazon Managed Grafana workspace.

Service Accounts and new APIs

In Grafana v10.4, Service Accounts replace API Keys, as the primary method for authenticating applications that interact with Grafana using Service Account Tokens. To support this change, we have introduced new control plane APIs that allow you to create, read, and delete service accounts and service account tokens. This enhancement enables you to fully automate the provisioning of Amazon Managed Grafana resources such as dashboards, folders, and data sources.

In the below snippet, using the AWS CLI, we will create a service account and token. This will allow us to provision dashboards automatically by a Grafana Operator running in an Amazon EKS cluster in this example.

SERVICE_ACCOUNT_TOKEN_ID=$(aws grafana create-workspace-service-account \
--workspace-id <GRAFANA_WORKSPACE_ID> \
--grafana-role ADMIN \
--name terraform-accelerator-eks \
--query 'id'
--output text)

# Creating a 3600s lived token as we deploy manually.
# In an automated pipeline, security token should be shorter in duration.
export TF_VAR_grafana_api_key=$(aws grafana create-workspace-service-account-token \
--workspace-id <GRAFANA_WORKSPACE_ID> \
--name grafana-operator \
--seconds-to-live 3600 \
--service-account-id $SERVICE_ACCOUNT_TOKEN_ID
--query 'serviceAccountToken.key'
--output text)

Demo environment

We will be utilizing the Amazon EKS cluster metrics example to collect metrics into a new Amazon Managed Service for Prometheus workspace and collect logs into CloudWatch.

cd terraform-aws-observability-accelerator/examples/existing-cluster-with-base-and-infra
terraform init
terraform apply -var eks_cluster_id=<EKS_CLUSTER_NAME> \
-var managed_grafana_workspace_id=<GRAFANA_WORKSPACE_ID>

After a few minutes, we should see new dashboards created in the Amazon Managed Grafana workspace.

Exploring the new features

Navigation

Amazon Managed Grafana 10.4 includes enhancements to the navigation interface, streamlining user workflows and simplifying interactions across the platform. A notable enhancement is the detachable breadcrumb navigation, now universally accessible. This feature allows users to easily backtrack through their activity in complex monitoring setups.

Figure 2 – Configuring AWS Data Sources in the navigation menu

The simplified experience of setting up AWS Data Sources can be found under the Apps section. If you ran the Terraform commands above, you will notice that some data sources have already been provisioned for you.

You can quickly navigate to the explore view with a single keyboard shortcut (g+e) , where you can query your data sources. Let’s now select the Amazon Managed Service for Prometheus metrics and issue a query. We are now presented with a new outline view on the left side which helps navigating between the query, the resulting graphs and raw data.

Figure 3 – Using the outline menu in the Explore section

Correlations

A correlation defines how data in one data source is used to query data in another data source. Correlations are supported when using the explore view with Logs or Table visualizations. The explore view provides you with an editor that allows you to build those correlations for any result field. To better understand correlations, let’s consider the following scenario as an example.

Metrics to logs correlation

In our example, we have Amazon Managed Service for Prometheus metrics for containers running in an Amazon EKS cluster. With the following query, we have an average CPU utilization for containers running in the default namespace, we do an aggregation while preserving only a few labels.

sum(rate(container_cpu_usage_seconds_total{namespace="default", container!=""}[5m])) by (pod, cluster, container, k8s_node_name)

Let’s assume we have CPU utilization spikes for a few containers and we want to understand the root cause.

One natural way to solve this would be be to identify the logs location for this particular container, and then run a query. This process is simplified with the new correlations feature. By clicking on Add and Add correlation button on the top action bar, we can instruct Amazon Managed Grafana do this process for us and save time in further investigations.

Figure 4 – Starting the correlation editor in Explore

This opens the correlation editor, where any table column can be a hyperlink. When selecting the pod, we get presented with a split view where we can select the CloudWatch datasource, and use all the variables in the query as dynamic arguments for querying. In our case, where we have a CloudWatch log group per namespace, and store each pod log in a separate CloudWatch log stream, this allows us to precisely find which log stream to query to get the pod logs.

Figure 5 – Creating a correlation with the explorer editor

With the correlation saved, we can select any pod using the hyperlink in the result table to view the logs with all the correct values provided in the CloudWatch Logs Insights query.

Figure 6 – Clicking on a particular pod column displays associated logs

We can also perform transformations to extract values that exist in a field with other data. For example, using a transformation, you can extract one portion of a log line to use in a correlation. Correlations can also be created, edited and deleted in the dedicated Correlation page in Home > Administration > Plugins and data > Correlations.

Correlations can be done on the same data source, for example we can make a correlation with the k8s_node_name column to find the associated node metrics. This can be useful for a scenario where we want to understand if the node hosting the pod is degraded and affects the pod’s performance.

Logs to trace correlation

It’s worth noting that some plugins have a form of correlation in their settings, introduced before Amazon Managed Grafana v10. For example, the CloudWatch logs plugins allows you to select a log field that contains an X-Ray trace ID value which can be used to query and display the associated transaction trace map. This scenario is explained in the One Observablity Workshop.

Visualization improvements

Trend Visual

The Trend visual in Amazon Managed Grafana is an experimental panel that allows you to visualize data trends where the x-axis represents numeric values instead of time. This panel fills a gap not addressed by traditional Time Series or XY Chart panels, making it ideal for datasets with a sequential, numeric x-axis, such as function graphs, RPM/torque curves, supply/demand relationships, or elevation/heart rate plots along a racecourse.

The Trend Panel supports various visual styles and options similar to the time series visualization, including rendering series as lines, points, or bars. However, it lacks certain features like annotations, time regions, shared cursor/crosshair, multi-timezone x-axis, and drag-selection to change the dashboard time range.

You can leverage the Trend Panel to showcase visually engaging examples, such as plotting function graphs to illustrate mathematical relationships or simulations, visualizing RPM/torque curves of an engine to understand its performance characteristics, analyzing supply/demand relationships in economics or business scenarios, or mapping elevation or heart rate changes along a racecourse, providing valuable insights for athletes and trainers.

Figure 7 – Trend visual

Datagrid Visual

The Datagrid visual allows users to create, edit, and adjust data directly within the platform. It can be used as both a standalone data source and a tabular visualization tool within a dashboard. Users have the flexibility to manipulate existing data, start with a blank slate, or import data via drag-and-drop. This feature enables significant customization and editing, including the deletion of data to reset the state.

Figure 8 – Datagrid visual

XY Chart

The XY Chart plots data points on numerical X and Y axes. Ideal for scientific, engineering, financial, and statistical applications, this chart excels at illustrating the relationships between two variables. Users can effectively analyze correlations, trends, and outliers within their datasets, enabling clear and precise visualizations of complex information.

Figure 9 – XY Chart

Take a look at the Panels and Visualizations documentation to learn more.

Subfolders

The subfolders feature in Amazon Managed Grafana 10.4 improves dashboard management by allowing you to organize dashboards into a hierarchical structure, similar to a file system. This organization aids in clearer, more manageable setups, especially in environments with many dashboards or multiple teams. It also simplifies access control, as permissions can be set at the folder level, streamlining management and enhancing security for different user groups. This feature is particularly useful for larger organizations or those with complex monitoring setups, ensuring that dashboards are easier to find, manage, and secure.

Figure 10 – Organizing folders and subfolders

Grafana alerting preview

With version 10.4, Grafana introduced Grafana alerting upgrade with rule preview, a feature that allows users to view, edit, and delete migrated rules prior to cutting over, with the option to roll back to classic alerting if needed. Users looking to migrate from classic alerting to Grafana Alerting can do so with confidence with the migration preview. Classic alerting will be completely removed from Grafana in version 11.

Figure 11 – Previewing alerting upgrade

Cleanup

If you ran the instructions, make sure to delete resources to stop incurring any charges.

If you deployed Amazon Managed Grafana using the AWS Console you will need to navigate to the Amazon Managed Grafana service section and deleted the workspace you created.

If you deployed using the AWS CLI you will need to execute the below commands. Replace <WORKSPACE_ID> with the actual ID of your workspace.

aws grafana delete-workspace --workspace-id <WORKSPACE_ID>

If you deployed using the Terraform module you will need to execute the below commands.

cd terraform-aws-observability-accelerator/examples/existing-cluster-with-base-and-infra
terraform destroy
cd terraform-aws-observability-accelerator/examples/managed-grafana-workspace
terraform destroy

Finally, if you used the AWS CLI to create your Amazon Managed Grafana workspace, run:

aws iam detach-role-policy \
    --role-name grafana-v10-demo \
    --policy-arn $APS_POLICY_ARN

aws iam delete-policy \
--policy-arn $APS_POLICY_ARN

aws iam detach-role-policy \                                                       
--role-name grafana-v10-demo \
--policy-arn arn:aws:iam::aws:policy/AWSXrayReadOnlyAccess

aws iam detach-role-policy \                                                       
--role-name grafana-v10-demo \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonGrafanaCloudWatchAccess

aws iam delete-role --role-name grafana-v10-demo

aws grafana delete-workspace --workspace-id <WORKSPACE_ID> 

Conclusion

In this blog, we’ve highlighted key new features of Grafana version 10.4 in Amazon Managed Grafana including Correlations, new navigation experience, and panels updates. We’ve seen how to setup a new workspace in 10.4, and how to migrate to the Amazon Managed Grafana v10.4 workspaces. With the replacement of Grafana API keys by Grafana Service Accounts, we’ve also covered our related new control plane APIs to help you perform operations in your workspace.

To explore the complete list of new features in version 10.4, refer to the user documentation. To learn more about Amazon Managed Grafana features and its pricing, visit the product page and pricing page.

• Check out One Observability Workshop aimed at providing a hands-on experience for you on the wide variety of toolsets AWS offers to setup monitoring and observability on your applications.
• Refer to AWS Observability best practices to learn more about prescriptive guidance and recommendations with implementation examples.
• If you are interested in Observability with infrastructure-as-code check out the AWS Observability accelerator which is a set of Terraform modules or CDK patterns to help you configure Observability for your workloads with AWS Observability services.