Setting up Grafana on EC2 to query metrics from Amazon Managed Service for Prometheus

The recently launched Amazon Managed Service for Prometheus (AMP) service provides a highly available and secure environment to ingest, query, and store Prometheus metrics. We can query the metrics from the AMP environment using Amazon Managed Grafana, a self-hosted Grafana server, or using the HTTP APIs.

In this article, we will look at how to configure a self-hosted open source Grafana server on Amazon Elastic Compute Cloud (Amazon EC2) to connect an AMP data source and create a dashboard.

Overview

To set up the environment, we will follow these steps:

• Install Grafana Enterprise server on Amazon EC2.
• Enable SigV4 authentication through built-in AWS SDK in Grafana (v7.3.5 onward).
• Set up an Application Load Balancer (ALB) to access Grafana server externally.
• Add AMP as data source in Grafana using SigV4 authentication.
• Query the metrics from AMP to verify the setup works as expected.

Install Grafana Enterprise server on EC2

To begin, log in to the EC2 console on your AWS account.

Next, create an Amazon Linux 2 Amazon Machine Image (AMI) based EC2 instance and assign an IAM role that has the managed policy arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess attached to it. If you want to use a custom policy, ensure the policy has the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "aps:GetLabels",
                "aps:GetMetricMetadata",
                "aps:GetSeries",
                "aps:QueryMetrics"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Log into the EC2 console using AWS CloudShell or your terminal on macOS/Linux/Windows through SSH.

Option 1: Install using binary .tar.gz file. Download and extract the Grafana server binaries by executing the following commands. We are following instructions in the Grafana documentation.

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-7.3.6.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-7.3.6.linux-amd64.tar.gz

You should be able to see a directory with the name of the Grafana version installed. Because we install Grafana version 7.3.6, we are able to see a directory with name grafana-7.3.6 on the EC2 instance.

Option 2: Install using YUM repository. Optionally, you can also install the Grafana server as a systemd process by following instructions from the Grafana documentation.

After the installation, you should see a screen similar to the following:

Enable SigV4 authentication through built-in AWS SDK in Grafana (v7.3.5 onward)

If you used Option 1 in the previous step, execute the following commands to set the required environment variables:

export AWS_SDK_LOAD_CONFIG=true
export GF_AUTH_SIGV4_AUTH_ENABLED=true

Next, execute the following command to start the Grafana server:

./bin/grafana-server web

If you used Option 2 to install Grafana, you can start the server using systemd. Before that, set the required environment variables to enable SigV4 auth by executing the following command:

sudo systemctl edit grafana-server

This step will open the vi editor, where we need to paste the following configuration. Save the file and exit the editor.

[Service]
Environment="GF_AUTH_SIGV4_AUTH_ENABLED=true"
Environment="AWS_SDK_LOAD_CONFIG=true"

Execute the following command to start the Grafana server:

sudo systemctl start grafana-server

We can check that the server is running as expected by executing the following command:

sudo systemctl status grafana-server

You should see a screen similar to the following, indicating that the server has started successfully:

Your Grafana server is now up and running. You can check this by executing the following command. You should see a response:

curl http://localhost:3000 -L

Set up an ALB to access Grafana server externally

As a best practice, we’ll create an ALB that uses a Target Group to forward HTTP requests to the EC2 instance on port 3000 (which is the default Grafana port).

Add AMP as data source in Grafana using SigV4 authentication

Log in to Grafana using the ALB URL. Provide the default credentials admin/admin to log in.

Once logged in, we go to the Data Sources screen from Settings:

Select Add data source, then Prometheus from the list as shown below:

Next, we paste the AMP Endpoint query URL (find this under the Summary tab on the AMP workspace) leaving out the api/v1/query portion (for example, https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-3aa5f57b-yy11-xx00-12ab-ea86005d6dd7/) in the URL field under HTTP. We need to enable SigV4 auth in the Auth section.

We also need to ensure that AWS SDK Default is selected in Authentication Provider under the Sigv4 Auth Details section, then select the AWS Region in which the AMP workspace was created earlier in the Default Region drop-down. See the following screenshot for details.

Next, we can choose Save & Test. We should see a green banner that says “Data source is working” as shown in the following.

Query the metrics from AMP to verify the setup

Next, we’ll create a new Dashboard from the left navigation bar by choosing the + sign.

We then add a new panel and select the new AMP data source configured previously.

We can write a simple PromQL query in the Metrics textbox, and we should see the metrics in the panel as shown in the screenshot:

Conclusion

In this article, we showed how to configure a self-hosted Grafana server on EC2 to connect to Amazon Manager Service for Prometheus workspace. AWS also offers Amazon Managed Grafana, a fully managed service that is highly available, secure, and natively integrated with other AWS services, such as AWS Identity and Access Management (IAM), AWS Single Sign-On (AWS SSO), Amazon Simple Notification Service (Amazon SNS), among others. Check out the blog post, Amazon Managed Grafana – Getting started, from the AWS Management & Governance Blog, to get started with those services.