AWS Open Source Blog

Running Bleeding-Edge Kubernetes on AWS with kops

In an earlier blog post, I explained how to set up a Kubernetes cluster on AWS using kops. By default, the kops create cluster command chooses the default Kubernetes version from the stable channel of kops, which would work for most developers. If you want a specific version, the --kubernetes-version option can be used to specify that. But what if you’d like to use kops to create a build using a recently-merged PR in Kubernetes? (Maybe, like me, you’re too impatient to wait through a test cycle for kops to support this in the channel – I want to try out the feature now!) This guest post by Micah Hausler explains how you can use a development build of Kubernetes to spin up your own cluster using kops.

– Arun


One of the easiest tools for creating, running, and managing Kubernetes clusters on AWS is kops. You can create clusters for released versions of Kubernetes easily, as explained in Arun’s earlier post, Manage Kubernetes Clusters on AWS Using Kops. In this post, we’ll show you how to use kops to create a cluster running a development build of Kubernetes. Kops states the version of Kubernetes that it officially supports, and, at the time of writing, Kubernetes 1.8.4 is the latest supported version. The examples in this post all use versions of Kubernetes not supported by kops, so they are only recommended for development, prototyping, and testing.

Before trying these examples yourself, you’ll need kops, kubectl, gsutil, and the AWS command line interface.

Background – release binaries

When creating a cluster with kops, you have the option of specifying a release of Kubernetes that kops supports, or providing kops an HTTP URL address where the Kubernetes binaries are located.

The Kubernetes release team places release binaries in the Google Cloud Storage bucket kubernetes-releases at the following location: https://storage.googleapis.com/kubernetes-release/release/

You can use the gsutil command line tool to verify that all the required binaries are located in a particular path. (The “https://storage.googleapis.com/” of the URL is replaced with “gs://”)

gsutil ls gs://kubernetes-release/release/v1.9.0/

Each pull request to Kubernetes gets tested with kops and shows up in the Github status as the “pull-kubernetes-e2e-kops-aws” test.

When you click on “Details” for the kops test, you’ll see the test output and a version for the specific commit of the given Pull Request.

The Kubernetes binaries used for this test are also stored in a Google Cloud Storage bucket, titled “kubernetes-release-pull”. If you search the raw build log for the kops test for that version string, you’ll see that the release binaries are uploaded to the location:

gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws/<version>

So for pull request #56759, you would set the Kubernetes version in kops to the URL

https://storage.googleapis.com/kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws/v1.10.0-alpha.1.84+dd628f15f46a9c

Read the kops docs if you’re interested in using a custom Kubernetes build and uploading your binaries to S3.

Create the cluster

Before you can create a cluster, you will need the proper AWS permissions. (If you are using a cross-account role, set the environment variable AWS_SDK_LOAD_CONFIG=1) Read through the kops documentation for guidance on creating an AWS user with the IAM permissions required, but the short list of IAM permissions are:

AmazonEC2FullAccess
AmazonRoute53FullAccess
AmazonS3FullAccess
IAMFullAccess
AmazonVPCFullAccess

Once you have a user with the correct permissions, you’re ready to create your cluster. You’ll need a name for the cluster, an S3 bucket for kops to store the state, and the AWS region and availability zones you want to create the cluster in. The example below uses us-east-1 and Kubernetes 1.9.0.

export CLUSTER_NAME="example.cluster.k8s.local"
export KUBERNETES_VERSION="https://storage.googleapis.com/kubernetes-release/release/v1.9.0/"
export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1}
export AWS_AVAILABILITY_ZONES="$(aws ec2 describe-availability-zones --query 'AvailabilityZones[].ZoneName' --output text | awk -v OFS="," '$1=$1')"
export S3_BUCKET=${S3_BUCKET:-kops-state-store-$(cat /dev/random | LC_ALL=C tr -dc "[:alpha:]" | tr '[:upper:]' '[:lower:]' | head -c 32)}
export KOPS_STATE_STORE=s3://$S3_BUCKET

If you are using a new S3 bucket for kops, you’ll need to create it first:

aws s3api create-bucket \
 --bucket $S3_BUCKET \
 --create-bucket-configuration LocationConstraint=$AWS_DEFAULT_REGION

At this point, you are ready to create your cluster! There are a lot of options when creating a cluster, so check the help output for “kops create cluster -h” to see the available options.

kops create cluster \
 --name $CLUSTER_NAME \
 --zones $AWS_AVAILABILITY_ZONES \
 --kubernetes-version $KUBERNETES_VERSION \
 --yes

After running kops create, your kubectl context will be updated to point to your new cluster. It will take a few minutes to bring all the resources online; you can check the status of your cluster using the command kops validate cluster.

validate cluster

You now have a Kubernetes cluster on AWS!

Get Involved

The kops project has grown a lot in the last year, and it has been great to see the community jump in to contribute. If you are looking for help, join the Kubernetes Slack and ask questions in the #kops channel. If you want to contribute, read the kops documentation on contributing.

Here are all the above steps compiled in a single script:

https://gist.github.com/micahhausler/b389cdb2cb5d0c51ee02d01454e9263a


At the time of writing, Micah Hausler was a Senior Site Reliability Engineer at Skuid where he led the DevOps team and was a contributor to Kubernetes. You can (still) find him at @micahhausler on Twitter, Github, and Kubernetes Slack.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

TAGS:
Micah Hausler

Micah Hausler

Micah Hausler is a Systems Development Engineer at Amazon Web Services where he works on the EKS team and is a contributor to Kubernetes. You can find him at @micahhausler on Twitter, Github, and Kubernetes Slack

Arun Gupta

Arun Gupta

Arun Gupta is a Principal Open Source Technologist at Amazon Web Services. He focuses on everything containers and open source at AWS. He is responsible for CNCF strategy within AWS, and participates at CNCF Board and technical meetings actively. He has built and led developer communities for 12+ years at Sun, Oracle, Red Hat and Couchbase. He has extensive speaking experience in more than 40 countries on myriad topics and is a JavaOne Rock Star for four years in a row. Gupta also founded the Devoxx4Kids chapter in the US and continues to promote technology education among children. A prolific blogger, author of several books, an avid runner, a globe trotter, a Docker Captain, a Java Champion, a JUG leader, NetBeans Dream Team member, he is easily accessible at @arungupta.