Teleport is an open source solution from Gravitational that can be configured as a proxy for administering a Kubernetes cluster. The open source version includes support for GitHub authentication, i.e. you can use GitHub as an identity provider for authenticating users. You can also extended Teleport’s session recording and audit log to Kubernetes. For example, regular kubectl exec commands are logged into the audit log and the interactive commands are recorded as regular sessions that can be stored and replayed in the future. The commercial version of Teleport adds support for RBAC and SSO using OAuth/OIDC or SAML; see Teleport Enterprise for more information.
While IAM authentication is adequate for a majority of use cases, interest in other types of authentication like OIDC has been steadily growing, particularly among organizations where creating new IAM user accounts is politically challenging. With a solution like Teleport, these organizations can use alternate types of authentication, such as GitHub, with EKS.
When you configure the EKS cluster endpoint to be private, the name of the EKS cluster can only be resolved from within the worker node VPC. As an alternative to creating Route 53 inbound endpoints (which cost approximately $90/month), you can run a Teleport proxy [in each worker node VPC] on a t3.small instance for as little as $14/month.
There are times when creating IAM users is not practical, e.g. when the composition of a development team is changing frequently or when the time to create an IAM user is excessively long. Delegating cluster access to a team lead who can control access through Teleport and GitHub can streamline the onboarding process for new team members.
You’ll need a functioning EKS cluster. If you’re unfamiliar with creating an EKS cluster, see eksctl.io.
The first step is to create a cluster role and role binding that will allow the EC2 instance to impersonate other users, groups, and service accounts.
cat << 'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-impersonation
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: teleport-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-impersonation
subjects:
- kind: Group
name: teleport-group
- kind: User
name: system:anonymous
EOF
Requests that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.
Create IAM trust policy document
This is the trust policy that allows the EC2 instance to assume a role.
cat > [filename] << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
Create IAM role
ROLE_ARN=$(aws iam create-role --role-name teleport-role --assume-role-policy-document file://[filename] | jq -r '.Role.Arn')
Create IAM policy granting list-clusters and describe-cluster permissions (optional)
This policy is necessary to create a kubeconfig file using the aws eks update-kubeconfig command. If you have another mechanism to create a kubeconfig file on the instance that runs Teleport, this step is not required.,
cat > [filename] << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters"
],
"Resource": "*"
}
]
}
EOF
POLICY_ARN=$(aws iam create-policy --policy-name teleport-policy --policy-document file://[filename] | jq -r '.Policy.Arn')
aws iam attach-role-policy --role-name teleport-role --policy-arn $POLICY_ARN
Update aws-auth configmap
This maps the IAM role teleport-role to the Kubernetes group teleport-group.
If you used eksctl to create your cluster, you may need to add the mapUsers section to the aws-auth ConfigMap before executing these commands.
ROLE=" - userarn: $ROLE_ARN\n username: teleport\n groups:\n - teleport-group"
kubectl get -n kube-system configmap/aws-auth -o yaml | awk "/mapUsers: \|/{print;print \" $ROLE \ ";next}1" > /tmp/aws-auth-patch.yml
kubectl patch configmap/aws-auth -n kube-system --patch "$(cat /tmp/aws-auth-patch.yml)"
After it’s patched, your aws-auth ConfigMap should look similar to:
mapUsers: |
- userarn: arn:aws:iam::123456789012:role/teleport-role
username: teleport
groups:
- teleport-group
Installing Teleport
Create EC2 instance
Create an EC2 instance using the Amazon Linux 2 (AL2) AMI on a public subnet in your VPC. Modify the security group associated with that instance to allow port 22 inbound, so you can SSH to the instance after it’s running. You will need security group rules to allow access to ports 3080 and 3022-3026 so that users can access the Teleport server from the Internet. This also allows GitHub to post a response back to the Teleport server. You’ll also need to open port 80 to allow Let’s Encrypt to do HTTP validation when issuing SSL certificates.
| Type |
Protocol |
Port Range |
Source |
| Custom TCP |
TCP |
3022-3026 |
0.0.0.0/0 |
| Custom TCP |
TCP |
3080 |
0.0.0.0/0 |
| HTTP |
TCP |
80 |
0.0.0.0/0 |
| SSH |
TCP |
22 |
your IP |
If you don’t modify the EKS control plane security group to allow port 443 inbound from the Teleport security group, your Teleport instance will not be able to communicate with the Kubernetes API.
Assign role to instance:
aws iam create-instance-profile --instance-profile-name teleport-role
aws iam add-role-to-instance-profile --instance-profile-name teleport-role --role-name teleport-role
aws ec2 associate-iam-instance-profile --iam-instance-profile Name=teleport-role --instance-id [instance_id]
instance_id should be replaced with the instance id of the instance where you intend to install Teleport.
SSH to the instance once it’s in a RUNNING state.
Download pip:
curl -O https://bootstrap.pypa.io/get-pip.py
Install pip:
sudo python get-pip.py
Upgrade AWS CLI:
sudo pip install --upgrade awscli
Download kubectl:
curl -o kubectl https://amazon-eks.s3-us-west-2.amazonaws.com/1.12.7/2019-03-27/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin
Download Teleport:
curl https://get.gravitational.com/teleport-v4.0.0-linux-amd64-bin.tar.gz -o teleport-v4.0.0-linux-amd64-bin.tar.gz
tar -xzf ./teleport-v4.0.0-linux-amd64-bin.tar.gz
Install Teleport:
sudo ./teleport/install
PATH=$PATH:/usr/local/bin
source ~/.bashrc
Create kubeconfig:
aws eks update-kubeconfig --name [cluster_name] --region [region]
Configuring Teleport
Create a systemd unit file (by creating a unit file, Teleport will automatically start when the instance is rebooted):
cat > teleport.service << 'EOF'
[Unit]
Description=Teleport SSH Service
After=network.target
[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport start --config=/etc/teleport.yaml --pid-file=/var/run/teleport.pid
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/teleport.pid
[Install]
WantedBy=multi-user.target
EOF
Copy to systemd/system and reload:
sudo mv teleport.service /etc/systemd/system/teleport.service
sudo systemctl daemon-reload
Create SSL certificate for HTTPs. It is absolutely crucial to properly configure TLS for HTTPS when you use Teleport Proxy in production. For simplicity, we are using Let’s Encrypt to issue certificates and nip.io for simple DNS resolution. However, using an Elastic IP and a Route53 domain name would be appropriate for production use cases.
Install certbot from EPEL:
sudo wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/
sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm
sudo yum-config-manager --enable epel*
sudo yum repolist all
sudo yum install -y certbot python2-certbot-apache
Generate a certificate for our nip.io hostname (update your email address below):
export TELEPORT_PUBLIC_DNS_NAME="$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f1).nip.io"
echo $TELEPORT_PUBLIC_DNS_NAME
export EMAIL=[yourname@yourdomain.com]
sudo certbot certonly --standalone \
--preferred-challenges http \
-d $TELEPORT_PUBLIC_DNS_NAME \
-n \
--agree-tos \
--email=$EMAIL
Let’s Encrypt certificates are valid only for a short duration. A cron job can be used to regularly renew certificates (Certbot developers recommend renewing twice daily):
echo "39 1,13 * * * root certbot renew --no-self-upgrade" | sudo tee -a /etc/crontab
sudo systemctl restart crond
Add a renewal hook to reload teleport configuration upon certificate renewal:
echo "renew_hook = systemctl reload teleport" | sudo tee -a /etc/letsencrypt/renewal/${TELEPORT_PUBLIC_DNS_NAME}.conf
Create config file
If you haven’t already, export the public DNS name of the Teleport instance as an environment variable:
export TELEPORT_PUBLIC_DNS_NAME="$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f1).nip.io"
cat > teleport.yaml << EOF
# By default, this file should be stored in /etc/teleport.yaml
## IMPORTANT ##
#When editing YAML configuration, please pay attention to how your editor handles white space. YAML requires consistent handling of tab characters
# This section of the configuration file applies to all teleport
# services.
teleport:
# nodename allows to assign an alternative name this node can be reached by.
# by default it's equal to hostname
nodename: $TELEPORT_PUBLIC_DNS_NAME
# Data directory where Teleport keeps its data, like keys/users for
# authentication (if using the default BoltDB back-end)
data_dir: /var/lib/teleport
# Teleport throttles all connections to avoid abuse. These settings allow
# you to adjust the default limits
connection_limits:
max_connections: 1000
max_users: 250
# Logging configuration. Possible output values are 'stdout', 'stderr' and
# 'syslog'. Possible severity values are INFO, WARN and ERROR (default).
log:
output: stderr
severity: ERROR
# Type of storage used for keys. You need to configure this to use etcd
# backend if you want to run Teleport in HA configuration.
storage:
type: dir
# This section configures the 'auth service':
auth_service:
authentication:
type: github
enabled: yes
# IP and the port to bind to. Other Teleport nodes will be connecting to
# this port (AKA "Auth API" or "Cluster API") to validate client
# certificates
listen_addr: 0.0.0.0:3025
# This section configures the 'node service':
ssh_service:
enabled: yes
# IP and the port for SSH service to bind to.
listen_addr: 0.0.0.0:3022
# See explanation of labels in "Labeling Nodes" section below
public_addr: $TELEPORT_PUBLIC_DNS_NAME:3022
labels:
role: master
type: postgres
# List (YAML array) of commands to periodically execute and use
# their output as labels.
# See explanation of how this works in "Labeling Nodes" section below
commands:
- name: arch
command: [/usr/bin/uname, -p]
period: 1h0m0s
# This section configures the 'proxy servie'
proxy_service:
enabled: yes
# SSH forwarding/proxy address. Command line (CLI) clients always begin their
# SSH sessions by connecting to this port
listen_addr: 0.0.0.0:3023
# Reverse tunnel listening address. An auth server (CA) can establish an
# outbound (from behind the firewall) connection to this address.
# This will allow users of the outside CA to connect to behind-the-firewall
# nodes.
tunnel_listen_addr: 0.0.0.0:3024
# List (array) of other clusters this CA trusts.
# trusted_clusters:
# - key_file: /path/to/main-cluster.ca
# Comma-separated list of OS logins allowed to users of this
# trusted cluster
# allow_logins: john,root
# Establishes a reverse SSH tunnel from this cluster to the trusted
# cluster, allowing the trusted cluster users to access nodes of this
# cluster
#tunnel_addr: 80.10.0.12:3024
# The HTTPS listen address to serve the Web UI and also to authenticate the
# command line (CLI) users via password+HOTP
web_listen_addr: 0.0.0.0:3080
# TLS certificate for the HTTPS connection. Configuring these properly is
# critical for Teleport security.
https_key_file: /etc/letsencrypt/live/$TELEPORT_PUBLIC_DNS_NAME/privkey.pem
https_cert_file: /etc/letsencrypt/live/$TELEPORT_PUBLIC_DNS_NAME/fullchain.pem
kubernetes:
enabled: yes
public_addr: $TELEPORT_PUBLIC_DNS_NAME:3026
listen_addr: 0.0.0.0:3026
kubeconfig_file: /home/ec2-user/.kube/config
EOF
Copy this file to the etc directory:
sudo cp ./teleport.yaml /etc/teleport.yaml
Start Teleport:
sudo systemctl start teleport.service
systemctl status teleport.service
journalctl -u teleport.service
Log out of the EC2 instance.
Configuring GitHub
Follow the instructions to create a new organization from scratch.
Create an OAuth app.
In the Homepage URL field, type the public nip.io DNS name of the teleport instance followed by the port 3080:
https://ec2-12-34-56-78.nip.io:3080
In the Authorization callback URL field type the public DNS name of the Teleport instance followed by /v1/webapi/github/callback :
https://ec2-12-34-56-78.nip.io:3080/v1/webapi/github/callback
Once you’re done creating the OAuth app, copy the client ID and secret from the OAuth Apps page under your Organizations settings. You will need these values when configuring GitHub authentication.
Finish configuring Teleport
SSH into the teleport instance.
Create github.yaml:
export TELEPORT_PUBLIC_DNS_NAME="$(curl http://169.254.169.254/latest/meta-data/public-hostname | cut -d '.' -f1).nip.io"
cat > github.yaml << EOF
kind: github
version: v3
metadata:
# connector name that will be used with 'tsh --auth=github login'
name: github
spec:
# client ID of Github OAuth app
client_id: [your client id]
# client secret of Github OAuth app
client_secret: [your client secret] # connector display name that will be shown on web UI login screen display: Github # callback URL that will be called after successful authentication redirect_url: https://$TELEPORT_PUBLIC_DNS_NAME:3080/v1/webapi/github/callback # mapping of org/team memberships onto allowed logins and roles teams_to_logins: - organization: [your github org name] # Github organization name team: [your github team name] # Github team name within that organization # allowed UNIX logins for team octocats/admins: logins: - ec2-user # list of Kubernetes groups this Github team is allowed to connect to kubernetes_groups: ["system:masters"] EOF
Apply github.yaml:
sudo /usr/local/bin/tctl create -f ./github.yaml
Log out of the EC2 instance.
Configuring the Teleport client
Download Teleport (MAC):
curl -O https://get.gravitational.com/teleport-v4.0.2-darwin-amd64-bin.tar.gz
tar -xzf teleport-v4.0.2-darwin-amd64-bin.tar.gz
Client binaries for other operating systems can be found at Teleport Community Edition.
Copy TSH binary to path:
cp ./teleport/tsh /usr/local/bin/tsh
Accessing the Kubernetes cluster
Before you run the following command, make a backup of your kubeconfig file as it will get overwritten when you log in to Teleport. If you have KUBECONFIG exported to another file run, ‘unset KUBECONFIG.’
In the tsh login command, type the public nip.io DNS name of the teleport instance followed by the port 3080:
tsh login --proxy=ec2-12-34-56-78.nip.io:3080
This will automatically open a new browser tab for you to sign in to GitHub. If you’re already signed in to GitHub, the page will say “Login successful.”
You should see output similar to the following in your terminal:
If browser window does not open automatically, open it by clicking on the link:
http://127.0.0.1:55811/7c22cd2d-c93b-41f4-ad70-a829e247fff9
> Profile URL: https://ec2-12-34-56-78.nip.io:3080
Logged in as: johnsmith
Cluster: ec2-12-23-56-78.nip.io
Roles: admin*
Logins: ec2-user
Valid until: 2019-07-09 05:07:13 +0800 CST [valid for 12h0m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
Now you have access to the Kubernetes API server with kubectl. Verify that your kubeconfig has been replaced and you are connected through the proxy by running kubectl config get-contexts.
Wrapping up
By using Teleport as an authentication proxy, you can use GitHub identities, in addition to IAM roles and users, to authenticate users who require access to an EKS cluster. This can be particularly useful in environments where obtaining an IAM principal is challenging or time-consuming. It may also prove useful in environments where the team composition is in constant flux.
As always, we welcome feedback on this post. If you have a suggestion or question, please click the view comments button below.
