Authenticating with Amazon Managed Grafana Using Open Source Keycloak on Amazon EKS

Application modernization and containers have become synonymous, as Kubernetes has become a popular platform choice for container orchestration. Amazon Elastic Kubernetes Service (Amazon EKS) makes it easy for customers to run their container-based workloads on Amazon Web Services (AWS). With Amazon EKS, customers run clusters of varying sizes, both small and large. Regardless of the scale of your containerized environment, it’s crucial to monitor the cluster to understand its health. Whether you are looking to troubleshoot and identify issues or dive deep into your application’s performance, the Grafana observability platform is often the go-to solution for cluster administrators.

Amazon Managed Grafana is a fully managed service for open source Grafana. When configuring authentication, customers can choose between integration with AWS IAM Identity Center, previously AWS Single Sign-On (SSO), or a Security Assertion Markup Language (SAML) provider. A number of software as a service (SaaS) SAML providers, including Azure Active Directory, CyberArk, Okta, OneLogin, and Ping Identity, are supported with Amazon Managed Grafana.

Another popular SAML provider is the free and open source Keycloak, a leading identity and access management solution that offers an alternative to providers with subscription pricing. Keycloak helps manage users and give access to their Grafana workspaces. Additionally, customers looking to fully automate the provisioning of Amazon Managed Grafana workspaces including the authentication layer can do so by installing and configuring Keycloak on Amazon EKS.

This blog post will walk you through the steps to deploy and configure Keycloak on Amazon EKS to serve as the SAML authentication provider for Amazon Managed Grafana. The well-known Bitnami Helm Chart for Keycloak is used to install Keycloak and automate the configuration of the Keycloak SAML provider.

Keycloak

Keycloak is a flexible, free and open source identity and access management solution for modern services and applications. The Keycloak project has a very large and active open source community. It provides various features including single sign-on (SSO), user federation, and identity brokering. It also provides both an admin console and an account management console. Keycloak uses open protocol standards like OpenID Connect and SAML 2.0.

Amazon EKS

Amazon EKS is a managed Kubernetes service that provides highly available and secure Kubernetes clusters. With managed control plane provided by Amazon EKS, the service automatically manages the availability and scalability of control plane components. Amazon EKS achieves this by running the control plane across multiple available zones and automatically detects and replaces unhealthy control plane nodes. Customers have different options for running their data plane with workloads, including Amazon Elastic Compute Cloud (EC2) servers or AWS Fargate. In this blog, Amazon EKS hosts the containerized applications.

Amazon Managed Grafana

Grafana is an open source analytics and interactive visualization application. Amazon Managed Grafana is a fully managed service for open source Grafana. It helps analyze application metrics, logs, and traces without provisioning servers. Amazon takes care of the undifferentiated heavy lifting involved with scaling and managing Grafana. Amazon Managed Grafana supports multiple data sources for observability, such as Prometheus, Amazon CloudWatch, and Amazon OpenSearch Service, along with third-party ISVs, like Datadog and Splunk.

Solution overview

Under the hood, we deploy Keycloak into Amazon EKS, along with the AWS Load Balancer Controller, which provisions the Application Load Balancer to expose the Keycloak application to outside traffic. We use ExternalDNS to update the DNS records to AWS Route 53, into the hosted zone. We also use AWS Certificate Manager (ACM), which will take care of SSL certificate creation and rotation for the domain under use. On a high level, the solution works as shown in the below diagram.

Open source Keycloak on AWS

When the user tries to access the Amazon Managed Grafana service, authentication is achieved using SAML with Keycloak. SAML is a standard which covers federation, identity management, and SSO. SAML is used as a standard for logging users into applications based on their previous sessions.

For this blog, Keycloak is the identity provider, and Amazon Managed Grafana is acting as the service provider. When the user tries to access Amazon Managed Grafana, the following sequence of actions takes place, which is explained in the below diagram.

Amazon Managed Grafana access diagram

Walkthrough

This blog will walk you through the following steps:

Create an Amazon EKS cluster.
Install AWS Load Balancer Controller. This will provision ALB for ingress resources.
Install External DNS service. This will update DNS records for AWS Route 53.
Create an Amazon Managed Grafana workspace.
Install Keycloak.
Update SAML configuration.
Validate login with Keycloak.

Prerequisites

You will need the following resources and tools for this walkthrough:

Route53 Public Hosted Zone. See creating a public hosted zone for detailed instructions if you don’t have an existing zone configured.
ACM Public Certificate. If your hosted zone is example.com, then use either a certificate for keycloak.example.com or a wildcard certificate *.example.com. See requesting a public certificate for detailed instructions. It is recommended to use the AWS console for easy DNS validation.
AWS Command Line Interface (AWS CLI) version 2
eksctl
kubectl
helm
jq

Step 1 — Create an Amazon EKS cluster

Start by setting a few environment variables. Choose your preferred AWS Region. The list of Regions that support Amazon Managed Grafana can be found in the documentation. Replace <your-public-domain> below with a domain that you own and have configured in Route53.

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export AWS_REGION=us-west-2 export CLUSTER_NAME=keycloak-blog export HOSTED_ZONE=<your-public-domain>

Then, create the EKS cluster. The eksctl configuration file below will also create two AWS Identity and Access Management (IAM) roles that will be associated with service accounts for the AWS Load Balancer Controller and External DNS. This is part of the IAM Roles for Service Accounts (IRSA) feature that gives the least privileged access for those two applications.

cat << EOF > keycloak_blog_cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: $CLUSTER_NAME
  region: $AWS_REGION
  version: "1.22"
addons:
- name: vpc-cni
iam:
  withOIDC: true
  serviceAccounts:
 - metadata:
       name: aws-load-balancer-controller
      namespace: aws-lb
    roleName: keycloak-blog-aws-lb-controller
    roleOnly: true
    wellKnownPolicies:
      awsLoadBalancerController: true
  - metadata:
       name: external-dns
      namespace: external-dns
     roleName: keycloak-blog-external-dns
     roleOnly: true
     wellKnownPolicies:
       externalDNS: true
managedNodeGroups:
- name: main
iam:
  attachPolicyARNs:
  - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
  - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
  - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
  instanceType: t3.large
  privateNetworking: true
EOF

eksctl create cluster -f keycloak_blog_cluster.yaml

Step 2 — Install the AWS Load Balancer Controller

Now Install the AWS Load Balancer Controller using Helm. The AWS Load Balancer Controller will create and configure an Application Load Balancer (ALB) for the Ingress defined when Keycloak is installed in Step 5.

cat << EOF > aws_lb_values.yaml
clusterName: $CLUSTER_NAME
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::${ACCOUNT_ID}:role/keycloak-blog-aws-lb-controller
image:
tag: v2.4.1
EOF

helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
--create-namespace \
--namespace aws-lb \
-f aws_lb_values.yaml

Step 3 — Install ExternalDNS

Install ExternalDNS next. ExternalDNS configures DNS servers for Kubernetes Ingress and Service objects. It will configure a Route53 record in your Hosted Zone that will direct traffic to Keycloak running in the Amazon EKS cluster. The name of the record is defined by the Ingress hostname when installing Keycloak in Step 5.

cat << EOF > edns_values.yaml
image:
  tag: v0.11.0
provider: aws
registry: txt
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${ACCOUNT_ID}:role/keycloak-blog-external-dns
txtOwnerId: $CLUSTER_NAME
EOF

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm install external-dns external-dns/external-dns \
    --create-namespace \
    --namespace external-dns \
    -f edns_values.yaml

Step 4 — Create Amazon Managed Grafana workspace

When creating an Amazon Managed Grafana workspace using the AWS CLI, you must first create an IAM Role. This role will be assigned to the workspace and the IAM permissions will be used to access any AWS data sources.

cat << EOF > grafana_trust_policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "grafana.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "${ACCOUNT_ID}"
                },
                "StringLike": {
                    "aws:SourceArn": "arn:aws:grafana:${AWS_REGION}:${ACCOUNT_ID}:/workspaces/*"
                }
            }
        }
    ]
}
EOF

aws iam create-role --role-name keycloak-blog-grafana-role \
    --assume-role-policy-document file://grafana_trust_policy.json

Next, create the Amazon Managed Grafana workspace configured to use SAML for authentication and the IAM role we created above. The commands below will wait for the workspace to be finished creating and assign the ID and URL of the workspace to variables to be used later.

RESULT=$(aws grafana create-workspace \
    --account-access-type="CURRENT_ACCOUNT" \
    --authentication-providers "SAML" \
    --permission-type "CUSTOMER_MANAGED" \
    --workspace-name "keycloak-blog" \
    --workspace-role-arn "keycloak-blog-grafana-role")

export WORKSPACE_ID=$(jq -r .workspace.id <<< $RESULT)

while true; do
    STATUS=$(aws grafana describe-workspace --workspace-id $WORKSPACE_ID | jq -r .workspace.status)
    if [[ "${STATUS}" == "ACTIVE" ]]; then break; fi
    sleep 1
    echo -n '.'
done

export WORKSPACE_ENDPOINT=$(aws grafana describe-workspace --workspace-id $WORKSPACE_ID | jq -r .workspace.endpoint)

Optional: If you would like this Amazon Managed Grafana workspace to access an AWS data source like Amazon Managed Service for Prometheus, you can use the Amazon Managed Grafana console to add permissions. Navigate to the keycloack-blog workspace and choose to the the “Data Sources” tab. Then, click the “Edit permission type” button and change the permission type to “Service managed.” Select your desired data sources and a new IAM role will be created with the permissions for your selected data sources.

Step 5 — Install Keycloak

You will need to choose a password for Keycloak. In this blog post, the password you choose is used both as the Keycloak admin password and as the admin password for the “keycloak-blog” realm. Replace <your-strong-password> below with a password of your choice.

export KEYCLOAK_PASSWORD=<your-strong-password>

Then, install Keycloak. The configuration below includes all the necessary SAML Assertion mapping to integrate Keycloak as a SAML identity provider for Amazon Managed Grafana. For more details about how to configure SAML Assertion Mapping for Amazon Managed Grafana, refer to the documentation.

cat << EOF > keycloak_values.yaml
auth:
  adminUser: admin
  adminPassword: $KEYCLOAK_PASSWORD
keycloakConfigCli:
  enabled: true
  configuration:
    realm.json: |
      {
        "realm": "keycloak-blog",
        "enabled": true,
        "roles": {
          "realm": [
            {
              "name": "admin"
            }
          ]
        },
        "users": [
          {
            "username": "admin",
            "email": "admin@keycloak",
            "enabled": true,
            "firstName": "Admin",
            "realmRoles": [
              "admin"
            ],
            "credentials": [
              {
                "type": "password",
                "value": "${KEYCLOAK_PASSWORD}"
              }
            ]
          }
        ],
        "clients": [
          {
            "clientId": "https://${WORKSPACE_ENDPOINT}/saml/metadata",
            "name": "amazon-managed-grafana",
            "enabled": true,
            "protocol": "saml",
            "adminUrl": "https://${WORKSPACE_ENDPOINT}/login/saml",
            "redirectUris": [
              "https://${WORKSPACE_ENDPOINT}/saml/acs"
            ],
            "attributes": {
              "saml.authnstatement": "true",
              "saml.server.signature": "true",
              "saml_name_id_format": "email",
              "saml_force_name_id_format": "true",
              "saml.assertion.signature": "true",
              "saml.client.signature": "false"
            },
            "defaultClientScopes": [],
            "protocolMappers": [
              {
                "name": "name",
                "protocol": "saml",
                "protocolMapper": "saml-user-property-mapper",
                "consentRequired": false,
                "config": {
                  "attribute.nameformat": "Unspecified",
                  "user.attribute": "firstName",
                  "attribute.name": "displayName"
                }
              },
              {
                "name": "email",
                "protocol": "saml",
                "protocolMapper": "saml-user-property-mapper",
                "consentRequired": false,
                "config": {
                  "attribute.nameformat": "Unspecified",
                  "user.attribute": "email",
                  "attribute.name": "mail"
                }
              },
              {
                "name": "role list",
                "protocol": "saml",
                "protocolMapper": "saml-role-list-mapper",
                "config": {
                  "single": "true",
                  "attribute.nameformat": "Unspecified",
                  "attribute.name": "role"
                }
              }
            ]
          }
        ]
      }
service:
  type: ClusterIP
ingress:
  enabled: true
  ingressClassName: alb
  pathType: Prefix
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: 'ip'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
  hostname: keycloak.${HOSTED_ZONE}
EOF

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install keycloak bitnami/keycloak \
    --create-namespace \
    --namespace keycloak \
    -f keycloak_values.yaml

Step 6 — Update SAML configuration

You will need to complete an additional step to finish SAML configuration before users can log in to the workspace. Update the Amazon Managed Grafana workspace with the Keycloak Identity Provider (IdP) metadata below.

export SAML_URL=https://keycloak.$HOSTED_ZONE/realms/keycloak-blog/protocol/saml/descriptor

aws grafana update-workspace-authentication \
    --authentication-providers SAML \
    --workspace-id $WORKSPACE_ID \
    --saml-configuration \
     idpMetadata={url=$SAML_URL},assertionAttributes={role=role},roleValues={admin=admin}

Step 7 — Validate login with Keycloak

Now, log in to Amazon Managed Grafana. Open your browser and navigate to the URL in the WORKSPACE_ENDPOINT variable. You can view it with the command below.

echo $WORKSPACE_ENDPOINT

Click on the “Sign in with SAML” button. This will redirect to Keycloak for authentication.

Keycloak sign in screen

Enter “admin” for the Username, and enter the Keycloak password you chose in Step 5.

Keycloak successful login screen
Congratulations! You have successfully authenticated with Amazon Managed Grafana using Keycloak as your SAML Identity Provider.

Cleaning up

To avoid incurring future charges, delete the resources you created by following the instructions below. Deleting the keycloak namespace is required for the Elastic Load Balancer and EBS Volume created by the Keycloak install to be automatically deleted as well.

aws grafana delete-workspace --workspace-id $WORKSPACE_ID
aws iam delete-role --role-name keycloak-blog-grafana-role
kubectl delete ns keycloak
eksctl delete cluster -f keycloak_blog_cluster.yaml

Conclusion

In this post, we learned how to authenticate Amazon Managed Grafana with open source Keycloak as the SAML Identity Provider running as a containerized solution on Amazon EKS. This offers customers that have chosen Keycloak as their identity provider a way to integrate that with their Amazon Managed Grafana environment. To learn more about Keycloak and the solutions discussed in the blog, please review the links in the references section below.