A Container-Free Way to Configure Kubernetes Using AWS Lambda

In a previous blog post Simplifying Kubernetes configurations using AWS Lambda we built a container image that contains eksctl, kubectl, and aws-auth to call the Kubernetes API. This allows you to create and manage resources through a unified control plane. In this post we take a look at an approach to access the Kubernetes API from an AWS Lambda function entirely in code, without the need to build a container image with executables. This provides a simple approach to create your Lambda functions, and also a faster execution time.

The Problem

When you want to send a command to your Amazon Elastic Kubernetes Service (Amazon EKS) cluster from kubectl you can create a kubeconfig file using the AWS Command Line Interface (AWS CLI) command aws eks update-kubeconfig. The kubeconfig created by the AWS CLI contains:

the endpoint for your Amazon EKS cluster
certificate authority data used to verify the certificate on the Amazon EKS endpoint
an exec setting to call “aws eks get-token” to create a bearer token from your IAM identity.

The previous blog post performed the authentication by including the AWS CLI in the container image. But you can do the same authentication from a Lambda function purely in code, and access your cluster with the Kubernetes Python client. This makes for a simpler function and offers some performance improvement.

The Solution

The Kubernetes Python client provides an API to authenticate with a bearer token. The beaker tokens we supplied to an Amazon EKS cluster Kubernetes API look like this:

k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtc291dGh....

If you remove the k8s-aws-v1. prefix and base64 decode the second portion of the token you’ll find a pre-signed STS get-caller-identity URL. Inside your cluster AWS IAM Authenticator for Kubernetes running on the control plane receives the token. The pre-signed URL is decoded and verified by the authenticator. The response from get-caller-identity proves your identity. The bearer token is used for authentication. Supplying the token “proves you are who you say you are”.

What am I allowed to do inside the cluster? Authorization is configured inside the aws-auth configmap. The aws-auth configmap is where you map your AWS Identity and Access Management (IAM) identity to an RBAC subject. In this example we will add an entry that authorizes your IAM role identity to list pods.

Identity Authorization with RBAC

Kubernetes API Authentication from AWS Lambda

How would I do this from a Lambda function? Rather than relying on the “aws eks get-token” being present in my Lambda environment I’d like to investigate a pure code solution for improved performance.

The endpoint and certificate authority data can be retrieved from the Amazon EKS DescribeCluster API. For improved performance the endpoint and certificate authority can be cached in the Lambda execution environment between calls. The bearer token can be generated from a pre-signed STS get-caller-identity URL.

I will use the Kubernetes API to list the pods running in the default namespace. I will need to configure Kubernetes RBAC to allow the list operation by creating a Role and RoleBinding. An entry in the aws-auth configmap maps the Lambda function’s role identity to a subject in Kubernetes.

Prerequisites

Python 3.9
eksctl and kubectl
The Bash shell. For Linux and macOS, this is included by default. In Windows 10, you can install the Windows Subsystem for Linux to get a Windows-integrated version of Ubuntu and Bash.
AWS Command Line Interface (AWS CLI) locally installed for programmatic interaction with AWS
Amazon EKS cluster

The scripts, AWS CloudFormation template, and function code are supplied in a GitHub repository.

├── 1-setup.sh
├── 2-deploy.sh
├── 3-kube-setup.sh
├── 4-invoke.sh
├── 5-cleanup.sh
├── function
│   ├── lambda_function.py
│   └── requirements.txt
└── template.yml

Lambda Function Code

The supplied function/lambda_function.py code configures the Kubernetes Python client. Inspect the get_bearer_token() to see the process used to create the token from a pre-signed STS get-caller-identity URL. More details on this process is available in the aws-iam-authenticator github repository.

The function retrieves endpoint and certificate authority data from the DescribeCluster. Note that this response is cached in the cluster_cache dictionary for subsequent reuse of the execution environment. The cluster information and bearer token are used to build a configuration dictionary. This configuration follows a very similar structure to your kubeconfig file. The dictionary contains a cluster, user, and context information.

kubeconfig = {
    'apiVersion': 'v1',
    'clusters': [{
        'name': 'cluster1',
        'cluster': {
        'certificate-authority-data': cluster["ca"],
        'server': cluster["endpoint"]}
    }],
    'contexts': [{'name': 'context1', 'context': {'cluster': 'cluster1', "user": "user1"}}],
    'current-context': 'context1',
    'kind': 'Config',
    'preferences': {},
    'users': [{'name': 'user1', "user" : {'token': get_bearer_token()}}]
}

Setup

Clone or download the repository at https://github.com/aws-samples/amazon-eks-kubernetes-api-aws-lambda. Run the 1-setup.sh script. The script prompts you for the name of the Amazon EKS cluster the Lambda function will connect to, and creates a bucket for the deployment artifacts.

Deployment

Run the 2-deploy.sh script to build, package and deploy the Lambda function. The template.yml creates a Lambda function lambda-eks-getpods-python and an IAM role lambda-eks-getpods-python-role. The IAM role has only two AWS permissions: logging in AWS CloudTrail and Amazon EKS DescribeCluster.

Kubernetes Setup

The function is now deployed, but our setup is not complete. If you were to run the function now you would see an “Unauthorized” message. If you have authenticator control plane logging enabled you would see a message in your logs similar to this:

level=warning msg="access denied" arn="arn:aws:iam::123456789012:role/lambda-eks-getpods-python-role"
client="127.0.0.1:33872" error="ARN is not mapped" method=POST path=/authenticate

This is telling us there is no entry in the aws-auth config map for the IAM role associated with the Lambda function. Handy tip: Authenticator logging can be very helpful when you are debugging authentication related issues.

I want to set up some permissions for the Lambda function inside the Amazon EKS cluster. To limit the access of the function I’d like to restrict the function to read-only access in the default namespace. I need to create three things:

a Kubernetes Role – contains the permissions settings limited to read-only operations in the default namespace.
a Kubernetes RoleBinding – maps the role to a group read-only-group.
an entry in the aws-auth configmap mapping the IAM role lambda-eks-getpods-python-role to the Kubernetes read-only-group.

Run the 3-kube-setup.sh script to create the read only Role, RoleBinding and aws-auth configmap mapping. The script displays the Role, RoleBinding contents and prompts you before creating them with kubectl. Next, the script prompts you to create the aws-auth configmap mapping with ekctl.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-only
  namespace: default
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-only-binding
  namespace: default
roleRef:
  kind: Role
  name: read-only
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: read-only-group

Invoke the Function

To invoke the function run 4-invoke.sh. You will see the results of the Kubernetes API call to list and count the number of pods in the default namespace. You have successfully deployed a Lambda function that can make an authenticated Kubernetes API call to your cluster!

Clean Up

Run 5-cleanup.sh to remove all the resources created in this article: artifacts bucket, CloudFormation stack, and the function log group.

Summary

You now have a new (and optimized) way to connect to the Kubernetes API endpoint of your Amazon EKS cluster. Lambda’s easy integration with other services like API Gateway, and Amazon CloudWatch events will make it easy to build integrations that use your Amazon EKS cluster.