AWS Open Source Blog

Deploying the Heptio Authenticator to kops

This post has been updated – Deploying the AWS IAM Authenticator to kops



Authenticator Flow

The Kubernetes 1.10 release has included alpha support for the client-go package to process external ExecCredential providers. This is being used to power the authentication against Amazon Elastic Container Service for Kubernetes (EKS) clusters while still following one of the core tenets of EKS; providing a 100% upstream Kubernetes. To make all this possible, we’re using the Heptio Authenticator. This post will walk you through deploying the authenticator to a kops cluster; allowing you to use AWS Identity & Access Management‎ (IAM) to authenticate kubectl

Managing authentication protocols is typically an onerous task, requiring admins to maintain a list of acceptable users, validate permissions on an ongoing basis for each user, prune users that don’t need access, and even periodically recycle token and certificate-based access. The more systems need to be managed, the more complicated these tasks become. That is why heptio, an AWS partner in the Amazon Partner Network, and AWS created the heptio authenticator, which allows you to have federated authentication using AWS IAM.

Getting Started

To get started, we’re going to need a Kubernetes cluster, and the easiest way to get this up and running is to use kops. First, you’re going to need to install the kops binary. The different installation options are explained in the kops documentation. If you are using macOS, you can follow along here.

brew update && brew install kops

After this has installed, verify by running.

$ kops version
Version 1.9.0

You will also need the Kubernetes command line tool, kubectl; you can install this using Homebrew as well.

brew install kubernetes-cli

Next we need to have an IAM user with the following permissions:

  • AmazonEC2FullAccess
  • AmazonRoute53FullAccess
  • AmazonS3FullAccess
  • IAMFullAccess
  • AmazonVPCFullAccess

Alternatively, a new IAM user may be created and the policies attached as explained at

The last dependency we need to install is the heptio authenticator. The easiest way to install this today is using go get. This requires that you have Golang installed on your machine. If you do not, please follow the directions here for your operating system. Once you have Golang installed, we can install the authenticator.

go get -u -v

make sure heptio-authenticator-aws is in your $PATH by trying to run the binary.

heptio-authenticator-aws help

If this fails with -bash: heptio-authenticator-aws: command not found then you will need to export a PATH including the $GOPATH/bin directory. Otherwise, continue to Create Cluster.

Create Cluster

Now that we have all the dependencies out of the way, let’s create your kops cluster. This is as simple as running one command:

export NAME=authenticator.$(cat /dev/random | LC_ALL=C tr -dc "[:alpha:]" | tr '[:upper:]' '[:lower:]' | head -c 10).k8s.local
export KOPS_STATE_STORE=s3://${NAME}
kops create cluster \
    --zones us-west-1a \
    --name ${NAME} \

If you’d like to deploy your cluster in a region besides us-west-1 make sure to change the --zones key to an availability zone in your region.

These commands will make a random $NAME that can be used for the bucket and the cluster, create the Amazon S3 bucket for storing cluster state, and, finally, create the cluster.

Now you can verify the status of the cluster by running the validate command.

kops validate cluster

This process can take five to ten minutes to set up. When the response ends with Your cluster {{NAME}} is ready you are ready to start using the cluster. To get the Heptio authenticator installed, we need to first modify the kops configuration:

kops edit cluster

This will open your $EDITOR to the kops cluster manifest file. In this file, under .spec.api, we’re going to add the following.

# ...
  authenticationTokenWebhookConfigFile: /srv/kubernetes/heptio-authenticator-aws/kubeconfig.yaml
- name: kops-hook-authenticator-config.service
    - kubelet.service
  roles: [Master]
  manifest: |
    Description=Download Heptio AWS Authenticator configs from S3
    ExecStart=/bin/mkdir -p /srv/kubernetes/heptio-authenticator-aws
    ExecStart=/usr/local/bin/aws s3 cp --recursive {{KOPS_STATE_STORE}}/{{NAME}}/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

Make sure to replace {{KOPS_STATE_STORE}} and {{NAME}} with the values from your created cluster. This will add a new systemd unit file that will download all the configurations from your KOPS_STATE_STORE Amazon S3 bucket; we’ll add these files next. This also tells the apiserver to use the Webhook authentication config that we’ll provide.

If you are adding this to an existing cluster and you are using a non-default AMI, you need to verify that the AWS command line interface is available. If not, you can change the ExecStart to use Docker instead.

ExecStart=/usr/bin/docker run --net=host --rm -v /srv/kubernetes/heptio-authenticator-aws:/srv/kubernetes/heptio-authenticator-aws aws s3 cp --recursive {{KOPS_STATE_STORE}}/{{NAME}}/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

After you are done, save and close this file. Next, we’ll create the files needed for the authentication configuration.

heptio-authenticator-aws init -i $NAME
aws s3 cp cert.pem ${KOPS_STATE_STORE}/${NAME}/addons/authenticator/cert.pem
aws s3 cp key.pem ${KOPS_STATE_STORE}/${NAME}/addons/authenticator/key.pem
aws s3 cp heptio-authenticator-aws.kubeconfig ${KOPS_STATE_STORE}/${NAME}/addons/authenticator/kubeconfig.yaml

Now that we have these created and uploaded to the state store, we can issue an update then a rolling-update to release new images. These commands can take five to ten minutes to process.

kops update cluster --yes
kops rolling-update cluster --yes

Create Policy

Now we can test that our default KubernetesAdmin user still has access to the cluster by running kubectl get nodes –this should return the nodes that are connected to your cluster.

Before we can give anyone access to the cluster, we first need to create the AWS IAM Role and Trust Policy for our additional admin user. This can be done either via the AWS Console or using the AWS CLI.

# Get your account ID
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

# Define a role trust policy that opens the role to users in your account (limited by IAM policy)
POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':root"},"Action":"sts:AssumeRole","Condition":{}}]}')

# Create a role named KubernetesAdmin (will print the new role's ARN)
aws iam create-role \
  --role-name KubernetesAdmin \
  --description "Kubernetes administrator role (for Heptio Authenticator for AWS)." \
  --assume-role-policy-document "$POLICY" \
  --output text \
  --query 'Role.Arn'

Now we can create a ConfigMap that defines the AWS IAM roles that have access to the cluster.

# authenticator.yaml
apiVersion: v1
kind: ConfigMap
  namespace: kube-system
  name: heptio-authenticator-aws
    k8s-app: heptio-authenticator-aws
  config.yaml: |    
    clusterID: {{NAME}}
      - roleARN: arn:aws:iam::{{ACCOUNT_ID}}:role/KubernetesAdmin
        username: kubernetes-admin
        - system:masters

apiVersion: extensions/v1beta1
kind: DaemonSet
  namespace: kube-system
  name: heptio-authenticator-aws
    k8s-app: heptio-authenticator-aws
    type: RollingUpdate
      annotations: ""
        k8s-app: heptio-authenticator-aws
      # run on the host network (don't depend on CNI)
      hostNetwork: true

      # run on each master node
      nodeSelector: ""
      - effect: NoSchedule
      - key: CriticalAddonsOnly
        operator: Exists

      # run `heptio-authenticator-aws server` with three volumes
      # - config (mounted from the ConfigMap at /etc/heptio-authenticator-aws/config.yaml)
      # - state (persisted TLS certificate and keys, mounted from the host)
      # - output (output kubeconfig to plug into your apiserver configuration, mounted from the host)
      - name: heptio-authenticator-aws
        - server
        - --config=/etc/heptio-authenticator-aws/config.yaml
        - --state-dir=/var/heptio-authenticator-aws
        - --kubeconfig-pregenerated

            memory: 20Mi
            cpu: 10m
            memory: 20Mi
            cpu: 100m

        - name: config
          mountPath: /etc/heptio-authenticator-aws/
        - name: state
          mountPath: /var/heptio-authenticator-aws/
        - name: output
          mountPath: /etc/kubernetes/heptio-authenticator-aws/
      - name: config
          name: heptio-authenticator-aws
      - name: output
          path: /srv/kubernetes/heptio-authenticator-aws/
      - name: state
          path: /srv/kubernetes/heptio-authenticator-aws/

Now we can apply this config to our cluster to deploy the service and the allowed roles for authentication:

kubectl apply -f authenticator.yaml

Once we have this deployed, we need to make a new user in our kubeconfig. Do so by opening ~/.kube/config with your favorite editor. Next you can create a user by replacing {{NAME}} with your cluster name and {{ACCOUNT_ID}} with your AWS Account ID.

# ...
- name: {{NAME}}.exec
      command: heptio-authenticator-aws
        - "token"
        - "-i"
        - "{{NAME}}"
        - "-r"
        - "arn:aws:iam::{{ACCOUNT_ID}}:role/KubernetesAdmin"

Then we’ll want to modify our context to reference this new user:

kubectl config set-context {{NAME}} --user={{NAME}}.exec

With all of this in place, we can test authenticating against our cluster:

$ kubectl get nodes
NAME                                          STATUS    ROLES     AGE       VERSION   Ready     node      5h        v1.9.3   Ready     master    3h        v1.9.3   Ready     node      5h        v1.9.3

If you see the nodes of your cluster listed, the authenticator was deployed properly and is using STS to verify the users’ identity.


If you’d like to continue to use this cluster, you can leave it running. If you’d like to shut the instances down, you can do so by calling the kop delete cluster command:

kops delete cluster --name ${NAME} --yes


The Heptio authenticator gives you the ability to federate your Kubernetes apiserver authentication out to AWS IAM, allowing you to set up granular IAM role-based groups that grant granular Kubernetes RBAC rules. No longer will you have to issue complex commands to manage keys and certificates that grant kubectl access.

Thanks to Peter Rifel for creating the write-up on Github Heptio Authenticator.

Read more from Chris.

Chris Hein

Chris Hein

Chris Hein is a Sr. Developer Advocate for Kubernetes/EKS at Amazon Web Services. Before Amazon, Chris worked for a number of large and small companies like GoPro, Sproutling, & Mattel. Read More from Chris here and follow him at @christopherhein