Run Codefresh Pipelines on Amazon EKS Using Bottlerocket OS

By Kostis Kapelonis, Developer Advocate – Codefresh
By Curtis Rissi, Principal Partner Solutions Architect, Containers – AWS

In August 2020, Amazon Web Services (AWS) announced Bottlerocket OS, a new open source Linux distribution that’s built specifically for running container workloads.

Bottlerocket OS comes with security hardening out of the box and support for transactional updates. This allows for greater ease in automating operating system (OS) updates, maintaining security compliance, and reducing operational costs.

Furthermore, Bottlerocket OS is designed to be able to run anywhere and, at launch, has a pre-built variant for Amazon Elastic Kubernetes Service (Amazon EKS).

In this post, we’ll show you how to deploy Codefresh Runners on Amazon EKS using Bottlerocket-based nodes to help scale your build environments quickly and benefit from Bottlerocket OS’s hardened security and simplified management.

Codefresh is an AWS ISV Partner and deployment platform specifically built for containers and Kubernetes. You can use Codefresh to create CI/CD pipelines that take an application from source code all the way to production.

Codefresh is built with speed and scalability in mind so you can run your builds in a fraction of the time. This is incredibly important today, as the speed with which you’re able to release new features, capabilities, and even hot-fixes to your customers can directly affect the success of your business.

Codefresh’s software-as-a-service (SaaS) offering lets you scale concurrent builds to meet these needs. However, some customers have additional security requirements and needs that require their code and build artifacts to remain in environments they manage.

For these customers, Codefresh offers the ability to employ Codefresh Runners to execute builds in customer-managed environments, including those in AWS (such as Amazon EKS) quickly and securely.

Codefresh Key Concepts

Before we dive into the how-to, let’s review some key Codefresh concepts.

The central component of the Codefresh platform are pipelines. These are workflows composed from individual steps that take care of the whole software lifecycle (code compilation, packaging, container builds, security scans, etc).

Pipelines are container-based, meaning that each step in the pipeline is an individual container that’s launched on-demand on the build node, once it’s needed. This makes the maintenance of build nodes very simple in Codefresh, as there’s nothing to install or upgrade (apart from the container runtime).

Gone are the days when you had to maintain specific build tools on each build node and battle with plugins and their dependencies.

The Codefresh Runner is a standard Kubernetes application that converts any Kubernetes cluster, including those running in EKS, into a Codefresh runtime environment for your pipelines. It’s managed and monitored like any other Kubernetes applications, meaning you can reuse your Kubernetes expertise and knowledge to maintain it.

If, for example, there is auto scaling enabled in your Kubernetes cluster, then the Codefresh Runner will gain auto scaling as well without any extra configuration.

The instructions below walk you through the process of creating an EKS cluster using Bottlerocket nodes and installing the Codefresh Runner on it. This will enable you to then run Codefresh pipelines on that cluster.

Prerequisites

To use Bottlerocket in Codefresh, you need:

Codefresh account—you can use Bottlerocket in any tier (even the free one).
AWS account—note that depending on the resources you create, your account will be billed, so make sure you clean up the resources you don’t need.

You also need the following CLIs:

AWS Command Line Interface (CLI) for accessing your AWS account from the command line.
eksctl for creating/managing your Kubernetes cluster.
Codefresh CLI for installing the Codefresh Runner in the cluster.
kubectl for working with Kubernetes applications.

Creating a Kubernetes Cluster Using Bottlerocket Nodes

You can see a Bottlerocket-based cluster configuration in the example eksctl config. The key point here is that it requires both the amiFamily and specific identity and access management (IAM) policies to allow the cluster nodes to function properly using Bottlerocket.

In addition to that, we’ll add the specific config to enable the node group to leverage Amazon EC2 Spot Instances.

Create an eksctl config file, and set the amiFamily to Bottlerocket in your node group. This tells eksctl to use the Bottlerocket EKS Amazon Machine Image (AMI) for the nodes in the group:

amiFamily: Bottlerocket

Ensure you have the required IAM policies specified for the Bottlerocket nodes:

iam:
     attachPolicyARNs:
         - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
         - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
         - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
         - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Add your Spot Fleet configuration to your node group:

 instancesDistribution:
      maxPrice: 0.017
      instanceTypes: ["t3.small", "t3.medium"] # At least two instance types should be specified
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 50
      spotInstancePools: 2

In this configuration, we’re using a combination of 50 percent On-Demand and Spot instances, two sizes of T3 instances, and a maximum price of $0.017 USD per hour. Learn more about these configuration options.

Your config file should look like this:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: sales-dev-eks-bottlerocket
  region: us-west-2
vpc:
  id: vpc-02c5dbe54afc8ed28
  subnets:
    private:
      us-west-2a:
          id: "subnet-0bde2b07890685c46"
      us-west-2b:
          id: "subnet-0b1d9a2612885d5de"
      us-west-2c:
          id: "subnet-0c69908965e7a2dd7"
    public:
      us-west-2a:
          id: "subnet-07ce9945741de1d4d"
      us-west-2b:
          id: "subnet-0109e5fe671a1290c"
      us-west-2c:
          id: "subnet-0226033eab8e4f954"
nodeGroups:
  - name: ng-bottlerocket
    instanceType: m5.large
    desiredCapacity: 1
    privateNetworking: true
    amiFamily: Bottlerocket
    iam:
        attachPolicyARNs:
         - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
         - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
         - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
         - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    instancesDistribution:
      maxPrice: 0.017
      instanceTypes: ["t3.small", "t3.medium"] # At least two instance types should be specified
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 50
      spotInstancePools: 2

Update the virtual private cloud (VPC) and subnets to match your own configuration and save the above file as eks-cluster.yaml. Next, create your EKS cluster by running the following command:

$ eksctl create cluster -f eks-cluster.yaml

Before installing the Codefresh Runner on your new EKS cluster, it’s a good idea to rename the context of our cluster. This way, the name of the runtime environment that will be created in our Codefresh account will have a more readable name.

To do that, you can run the following command with the context name you want. We used sales-dev-eks-bottlerocket for our context name:

$ kubectl config rename-context <ARN> sales-dev-eks-bottlerocket

Installing Codefresh Runner in the Bottlerocket-Based Cluster

Now, you can go ahead and install the Codefresh Runner on your EKS cluster. Install the Codefresh CLI with:

npm install -g codefresh

Set up authentication with Codefresh using your API Key (get a key from your user settings):

auth create-context --api-key {API_KEY}

Run the installation wizard with:

codefresh runner init.

The wizard will ask you if you would like to run a demo pipeline after install, and choose Yes.

If everything went well with the install, you should see a console output that includes details regarding the demo pipeline status, links to documentation and some Codefresh ascii art.

Figure 1 – Codefresh CLI output.

For more details, see the Codefresh Runner documentation.

We now have the Codefresh Runner installed on our Amazon EKS cluster!

You should be able to go back to the Codefresh console and select the “Runner” project that was created during the install. From there, you’ll see the demo pipeline that was created and executed.

Figure 2 – Codefresh Runner console.

You can click on the gear icon for the “CF_Runner_Demo” pipeline, which will open the pipeline settings console. Clicking the Runtime option from the left menu will show you the new runtime environment that was created in your pipeline settings. See more details for using the Runner.

Figure 3 – Codefresh pipeline runtime details.

Going back to the “CF_Runner_Demo” build, you can click into it to see the details and then again to see the “test” step details.

Figure 4 – Codefresh build output details.

That’s it! Any future builds of your pipeline will run on your Bottlerocket-powered Amazon EKS cluster and be centrally managed from your Codefresh account.

Conclusion

By leveraging the combination of solutions detailed in this post, you can scale quickly to meet the demands of your build pipelines. For workloads that need to remain in a customer-managed environment, you can do so in a way that improves overall security and simplifies management.

This is just the begining, too. For instance, by enabling the Cluster Autoscaler on AWS for your build cluster, you could enable it to automatically add additional cluster hosts in the event your pipelines need more capacity.

If you’re interested in, or already using, Codefresh Runners on EKS clusters using Amazon Linux 2, this is still supported. You can repeat the installation process on an EKS cluster using Amazon Linux 2 nodes, and then run your pipelines on either cluster.

In fact, you can even mix and match pipelines so you can use both types of clusters in Codefresh. This allows you to gradually move workloads from one cluster to the next, allowing you to ease the adoption of Bottlerocket-based nodes.

For more information on Bottlerocket OS, see the project Github page. For more information on the Codefresh Runner, see the official documentation and visit Codefresh.io to try it yourself.

.

.

Codefresh – AWS Partner Spotlight

Codefresh is an AWS ISV Partner and deployment platform specifically built for containers and Kubernetes. You can use Codefresh to create CI/CD pipelines that take an application from source code all the way to production.

Contact Codefresh | Partner Overview | AWS Marketplace

*Already worked with Codefresh? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog