Securing Kubecost access with Amazon Cognito

Introduction

Kubecost provides real-time cost visibility and insights for teams using Kubernetes. It has an intuitive dashboard to help you understand and analyze the costs of running your workloads in a Kubernetes cluster. Kubecost is built on OpenCost, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, and is actively supported by AWS.

Amazon EKS optimized bundle of Kubecost

Earlier last year, Amazon Elastic Kubernetes Service (Amazon EKS) announced the availability of an Amazon EKS-optimized bundle of Kubecost for cluster cost visibility. The bundle is available to customers free of charge and includes Kubecost troubleshooting support. Kubernetes platform administrators and finance leaders can use Kubecost to visualize a breakdown of their Amazon EKS charges, allocate costs, and chargeback organizational units (e.g., application teams). Kubecost gives internal teams and business units transparent and accurate cost data based on AWS bill. Customers can also get personalized suggestions for cost optimization tailored to their infrastructure environment and usage patterns.

Using Kubecost’s intuitive dashboard, customers can monitor, analyze, and allocate cluster costs. When customers deploy Kubecost in a cluster, the dashboard is secured by NGINX basic authentication, which isn’t recommended in production environments. This post shows how to make the dashboard accessible to external audiences, such as finance leaders, and secure access using Amazon Cognito.

Solution overview

We make the Kubecost dashboard accessible outside the cluster by exposing it using an ingress, which uses Application Load Balancer (ALB). Integrating Amazon Cognito with the ALB, the solution adds support for authenticating and authorizing users to the Kubecost dashboard. To learn more about how ALB and Cognito integrate, please see How to use Application Load Balancer and Amazon Cognito to authenticate users for your Kubernetes web apps.

In this post, we use the secure ingress-auth-cognito EKS Blueprints pattern to set up:

Application Load Balancer, Amazon Cognito, and a Transport Layer Security (TLS) Certificate on AWS Certificate Manager (ACM) with Amazon Route 53 hosted zone to authenticate users to Kubecost
Deployment of Kubecost application using Kubecost add-on for EKS CDK Blueprints
Kubernetes Ingress with annotations for Amazon Cognito and a TLS Certificate (using Amazon Certificate Manager) for securely authenticating user to Kubecost

Customers can use this pattern to manage multiple clusters across environments with GitOps. Please see Continuous Deployment and GitOps delivery with Amazon EKS Blueprints and ArgoCD to learn about GitOps-driven delivery using EKS Blueprints Patterns.

The secure ingress-auth-cognito Cloud Development Kit (CDK) pattern includes an Amazon EKS cluster configuration, compute capacity configuration, and add-ons required by Kubecost.

Architecture Showing how to Secure Ingress with Cognito for Kubecost application.

Prerequisites

You need the following to complete the steps in this post:

AWS Command Line Interface (AWS CLI) version 2
AWS CDK version 2.80.0 or later
Node version 20.0.0 or later
NPM version 8.19.2 or later
kubectl version 1.24 or later
Git
An Amazon Route 53 public hosted zone

Let’s start by setting a few environment variables:

ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
export AWS_REGION=${AWS_REGION:=us-west-2}

Clone the cdk-eks-blueprints-patterns repository and install dependency packages. This repository contains CDK v2 code written in TypeScript.

git clone https://github.com/aws-samples/cdk-eks-blueprints-patterns.git
cd cdk-eks-blueprints-patterns
npm install
make build

The secure ingress-auth-cognito EKS Blueprints pattern is at lib/secure-ingress-auth-cognito/index.ts. In this file, you can find the blueprint definition with all the configurations above using the blueprints.EksBlueprint.builder() method.

Bootstrap CDK

The first step to any CDK deployment is bootstrapping the environment. Bootstrapping is the process of provisioning resources for the AWS CDK before you can deploy AWS CDK applications into an AWS environment (an AWS environment is a combination of an AWS account and Region). If you already use CDK in a region, then you don’t need to repeat the bootstrapping process.

Execute the commands below to bootstrap the AWS environment in your Region:

cdk bootstrap aws://$ACCOUNT_ID/$AWS_REGION

Deploy Kubecost with secured access

In this solution, we’ll allow access to the Kubecost dashboard based on user email addresses. You can control access to the dashboard by allow-listing an entire domain or individual email addresses.

Users are required to sign-up before they can access the Kubecost dashboard. The pre sign-up Lambda trigger only allows sign-ups when the user’s email domain matches the allow-listed domains. When users sign-up, Amazon Cognito sends a verification code to their email address. Users have to verify access (using the one time valid code) to their email before they get access to the dashboard.

First, we’ll create an AWS Systems Manager (SSM) parameter to store the value of the email domain that users use to sign up. Next, we’ll create an environment variable to store the domain name that hosts the Kubecost dashboard. The email domain and the domain used to host the Kubecost dashboard can be same or different. For example, you may choose to host the dashboard at kubecost.myorg.mycompany.com and use email@mycompany.org to login to the dashboard.

Create below parameters with allowed email addresses and domains in the AWS Systems Manager Parameter Store:

export SSM_PARAMETER_KEY="/secure-ingress-auth-cognito/ALLOWED_DOMAINS"
export SSM_PARAMETER_VALUE="emaildomain1.com,emaildomain2.com"

aws ssm put-parameter \
  --name "$SSM_PARAMETER_KEY" \
  --value "$SSM_PARAMETER_VALUE" \
  --type "String" \
  --region $AWS_REGION

If you’d like to limit access to the dashboard by email addresses, then you can also create a parameter to store allowed email addresses and add a logic to the pre authentication Lambda trigger as shown here.

Next, create a secret in AWS Secrets Manager that you’ll use to access ArgoCD. The argo-admin-password secret must be defined as plain text (not key/value):

aws secretsmanager create-secret --name argo-admin-secret \
    --description "Admin Password for ArgoCD" \
    --secret-string "password123$" \
    --region $AWS_REGION

The CDK code expects the allowed domain and subdomain names in the CDK context file (cdk.json).

Create two environment variables. The PARENT_HOSTED_ZONE variable contains the name of your Route 53 public hosted zone. The DEV_SUBZONE_NAME will be the address for your Kubecost dashboard.

Generate the cdk.json file:

PARENT_HOSTED_ZONE=mycompany.a2z.com
DEV_SUBZONE_NAME=kubecost.mycompany.a2z.com
cat << EOF > cdk.json
{
    "app": "npx ts-node dist/lib/common/default-main.js",
    "context": {
        "parent.hostedzone.name": "${PARENT_HOSTED_ZONE}",
        "dev.subzone.name": "${DEV_SUBZONE_NAME}"
      }
}
EOF

Run the below command from the root of this repository to deploy the solution:

make pattern secure-ingress-cognito deploy secure-ingress-blueprint

This blueprint will deploy the following:

Amazon Virtual Private Cloud (Amazon VPC) with public and private subnets, Network Address Translation (NAT) gateways in each availability zone (AZ), and an Internet Gateway
An Amazon EKS cluster with the following Kubernetes add-ons
- Metrics Server
- Cluster Autoscaler
- Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) Driver Amazon EKS
- AWS Load Balancer Controller
- Amazon VPC CNI
- ExternalDNS
- Kubecost
- Argo CD
Amazon Cognito user pool, user pool client, domain and also pre-sign-up and pre-authentication lambda triggers to run custom logic to validate users before allowing them to either sign-up or authentication.

Once the deployment is complete, you will see the output similar to shown below in your terminal:

Outputs:
secure-ingress-blueprint.secureingressblueprintClusterNameD6A1BE5C = secure-ingress-blueprint
secure-ingress-blueprint.secureingressblueprintConfigCommandD0275968 =  aws eks update-kubeconfig —name secure-ingress-blueprint —region us-west-2 —role-arn arn:aws:iam::<ACCOUNT ID>:role/secure-ingress-blueprint-secureingressblueprintMas-XXXXXXXXXX
secure-ingress-blueprint.secureingressblueprintGetTokenCommand21BE2184 =  aws eks get-token —cluster-name secure-ingress-blueprint —region us-west-2 —role-arn arn:aws:iam::<ACCOUNT ID>:role/secure-ingress-blueprint-secureingressblueprintMas-XXXXXXXXXX

Stack ARN:
arn:aws:cloudformation:us-west-2:<ACCOUNT ID>:stack/secure-ingress-blueprint/XXXXXXXXXX

To update your Kubernetes configuration for your new cluster, copy and run the aws eks update-kubeconfig command (the second command in the output) in your terminal.

export EKS_KUBECONFIG=$(aws cloudformation describe-stacks \
  --stack-name secure-ingress-blueprint \
  --query "Stacks[0].Outputs[?starts_with(OutputKey, 'secureingressblueprintConfigCommand')].OutputValue" \
  --region $AWS_REGION \
  --output text)

eval $EKS_KUBECONFIG

Validate the access to your Amazon EKS cluster using below kubectl listing all namespaces:

kubectl get namespace

You should see the following namespaces in the cluster:

NAME              STATUS   AGE
argocd            Active   30m
default           Active   39m
external-dns      Active   30m
kube-node-lease   Active   39m
kube-public       Active   39m
kube-system       Active   39m
kubecost          Active   30m

The stack deploys Kubecost resources in the kubecost namespace.

kubectl -n kubecost get all
NAME                                                             READY   STATUS    RESTARTS   AGE
pod/kubecost-cost-analyzer-84d5775f7b-zg8mq                      2/2     Running   0          88m
pod/kubecost-cost-analyzer-grafana-69d77ccd6d-9r8rc              2/2     Running   0          88m
pod/kubecost-cost-analyzer-kube-state-metrics-789fc978c8-ch8lb   1/1     Running   0          88m
pod/kubecost-cost-analyzer-prometheus-node-exporter-w9w75        1/1     Running   0          88m
pod/kubecost-cost-analyzer-prometheus-server-6dc99564bf-mz9nw    2/2     Running   0          88m

NAME                                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/kubecost-cost-analyzer-cost-analyzer              ClusterIP   172.20.193.130   <none>        9003/TCP,9090/TCP   88m
service/kubecost-cost-analyzer-grafana                    ClusterIP   172.20.143.32    <none>        80/TCP              88m
service/kubecost-cost-analyzer-kube-state-metrics         ClusterIP   172.20.165.147   <none>        8080/TCP            88m
service/kubecost-cost-analyzer-prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP            88m
service/kubecost-cost-analyzer-prometheus-server          ClusterIP   172.20.54.102    <none>        80/TCP              88m

NAME                                                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/kubecost-cost-analyzer-prometheus-node-exporter   1         1         1       1            1           <none>          88m

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubecost-cost-analyzer                      1/1     1            1           88m
deployment.apps/kubecost-cost-analyzer-grafana              1/1     1            1           88m
deployment.apps/kubecost-cost-analyzer-kube-state-metrics   1/1     1            1           88m
deployment.apps/kubecost-cost-analyzer-prometheus-server    1/1     1            1           88m

Testing the authentication

Point your browsers to the URL you associated with the DEV_SUBZONE_NAME key from the CDK context to access the Kubecost dashboard.

The value is also stored as an environment variable:

echo $DEV_SUBZONE_NAME

Your browser will be redirected to an Amazon Cognito hosted User Interface (UI) sign-in page. Since this is your first time accessing the application, select sign up.

The Pre sign-up AWS Lambda trigger for Amazon Cognito User pool is configured to allow users to register only from certain allow-listed email domains. The allow-listed email domains are configured as an environmental variable in the AWS Lambda function. Let us try sign up a new user using an email id, whose domain is not part of the allow list.

Screenshot of sign-in page

You’ll get an error since the domain is not allow-listed. Screenshot of error message

Let’s sign up as a new user with one of the allow-listed email domains. This time, you’ll get a prompt to confirm your account. Get the verification code sent to your email and confirm your account.

Screenshot of account verification page

After verifying email address, sign in to access the Kubecost dashboard

Once you sign in, the ALB will redirect you to the Kubecost dashboard:

Kubecost dashboard

Cleaning up

You continue to incur cost until deleting the infrastructure that you created for this post. Use the commands below to delete resources created during this post:

make pattern secure-ingress-cognito destroy secure-ingress-blueprint

Conclusion

In this post we showed you how to secure the Kubecost dashboard while making it accessible to users without needing access to the Kubernetes cluster. We used an ALB to expose the dashboard and secured access using Cognito. We also created a record in Route 53 so users can easily access the dashboard.

We used Cognito user pool to store user information. If you already have an identity provider that provides OpenID Connect (OIDC) or SAML 2.0 support, then you can integrate it with Cognito to skip the sign-up and sign-in to the Kubecost dashboard.

Containers