CI/CD on Amazon EKS using AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and FluxCD

This post discusses how we can speed up the development of our Kubernetes infrastructure by using a continuous integration (CI) pipeline to build our Docker images and automatically deploy them to our Amazon Elastic Kubernetes Service (Amazon EKS) cluster using FluxCD and the GitOps philosophy as the continuous delivery (CD) element. To do so, we use an AWS managed services solution.

What is GitOps

GitOps[1] is an approach where infrastructure as code (IaC) is hosted in a git repository and follows the same merge request process as application software code. Git becomes the source of truth, and infrastructure files, are managed through a git repository. This post uses FluxCD, an open source CD system developed by Weaveworks, to keep everything that is written in our Git repositories synchronized with our Kubernetes infrastructure.

Provision the infrastructure

For this post, we created an AWS Cloud Development Kit (AWS CDK) repository written in Python, where you can deploy an infrastructure with multiple components managed by AWS and also the FluxCD installation. For a list of the prerequisites you need before deploying, see the README.md file. You can also install FluxCD yourself; for instructions, see Getting started with Flux.

Our architecture is made of a main pipeline in which developers and infrastructure administrators carry out the CI, and later rely on FluxCD as a CD system.

The pipeline has two main stages:

• Source – AWS CodeCommit stores the application repository and the Kubernetes infrastructure repository. A commit occurring in the application repository triggers the pipeline to start the Source stage, which clones the repository code.
• Build – AWS CodeBuild looks for a buildspec.yaml to build and pushes our container application to Amazon Elastic Container Registry (Amazon ECR).

The following screenshot shows our pipeline.

Our buildspec.yaml lives in the application repository. See the following code:

version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- REPOSITORY_URI=${REPO_ECR}
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=${COMMIT_HASH:=latest}
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG

After the Docker image is pushed to Amazon ECR, FluxCD retrieves the image, synchronizing the Kubernetes infrastructure based on our .yaml definition file, from the Kubernetes infrastructure repository. See the following code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: awsome-app-flux
  labels:
    app: app
  annotations:
    fluxcd.io/automated: "true"  
spec:
  replicas: 2
  selector:
    matchLabels:
      app: awsome-app-flux
  template:
    metadata:
      labels:
        app: awsome-app-flux
    spec:
      containers:
      - name: awsome-app-flux
        image: account_id.dkr.ecr.us-east-1.amazonaws.com/awsome-flux-example:latest

You can check the status of the sync by listing the current pods:

# kubectl get pod
NAME READY STATUS RESTARTS AGE
awsome-app-flux-67b6457b5b-fhz7m 1/1 Running 0 1m
awsome-app-flux-67b6457b5b-ngxhb 1/1 Running 0 1m 
flux-f8c9c99f9-k9wd4 1/1 Running 0 20m 
memcached-5bd7849b84-26xrv 1/1 Running 0 20m 

The key characteristic is the annotation part of the code, which automates the update when there is a new container image:

  annotations:
    fluxcd.io/automated: "true" 

For more information, see Automated deployment of new container images.

FluxCD needs access to the Kubernetes infrastructure repository to check for any changes on this repository so they can listen for all changes and sync with Kubernetes. For example, in the following code, we change the number of replicas from two to three and commit it:

# kubectl get pod
NAME                               READY   STATUS    RESTARTS   AGE
awsome-app-flux-67b6457b5b-bg2l9   1/1     Running   0          25s
awsome-app-flux-67b6457b5b-fhz7m   1/1     Running   0          16m
awsome-app-flux-67b6457b5b-ngxhb   1/1     Running   0          16m
flux-f8c9c99f9-k9wd4               1/1     Running   0          35m 
memcached-5bd7849b84-26xrv         1/1     Running   0          35m 

We can see the refresh on the logs of the FluxCD pod:

# kubectl logs flux-f8c9c99f9-k9wd4

ts=2021-03-06T16:01:37.676604977Z caller=sync.go:540 method=Sync cmd=apply args= count=1
ts=2021-03-06T16:01:37.910480108Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=233.832365ms err=null output="deployment.apps/awsome-app-flux unchanged"
ts=2021-03-06T16:01:37.913245645Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-03-06T16:01:37.929480317Z caller=images.go:106 component=sync-loop workload=default:deployment/awsome-app-flux container=awsome-app-flux repo=account-id.dkr.ecr.us-east-1.amazonaws.com/awsome-flux-example pattern=glob:* current=account-id.dkr.ecr.us-east-1.amazonaws.com/awsome-flux-example warning="image with zero created timestamp" current="account-id.dkr.ecr.us-east-1.amazonaws.com/awsome-flux-example (0001-01-01 00:00:00 +0000 UTC)" latest="account-id.dkr.ecr.us-east-1.amazonaws.com/awsome-flux-example:8ffe182 (2020-11-24 15:55:23.089132354 +0000 UTC)" action="skip container"
ts=2021-03-06T16:04:33.69339545Z caller=loop.go:142 component=sync-loop jobID=e13a92e5-f54c-bfe0-4071-da8c7e15df03 state=in-progress
ts=2021-03-06T16:04:36.647672805Z caller=loop.go:154 component=sync-loop jobID=e13a92e5-f54c-bfe0-4071-da8c7e15df03 state=done success=true
ts=2021-03-06T16:04:36.857335869Z caller=loop.go:134 component=sync-loop event=refreshed url=https://user-account@git-codecommit.us-east-1.amazonaws.com/v1/repos/kubernetes-infra-awsome-flux-example branch=master HEAD=903c14beb6ff52afb6d9d7931c1550a73f6d3370
ts=2021-03-06T16:04:36.85955757Z caller=sync.go:61 component=daemon info="trying to sync git changes to the cluster" old=c989d6031273dda83b8b67b0319410bbfd01a74a new=903c14beb6ff52afb6d9d7931c1550a73f6d3370

Conclusion

This post demonstrated how can we accelerate our container deployments on Kubernetes by building a CI/CD pipeline using self-managed services on AWS and an open-source tool like FluxCD.

[1]https://pages.awscloud.com/rs/112-TZM-766/images/Weaveworks%20GitOps_AWS_eBook_07012020.pdf

About the authors:

César Prieto Ballester is a DevOps Consultant at Amazon Web Services. He enjoys automating everything and building infrastructure using code. Apart from work, he plays electric guitar and loves riding his mountain bike.

Bruno Bardelli is a Senior DevOps Consultant at Amazon Web Services. He loves to build applications and in his free time plays video games, practices aikido, and goes on walks with his dog.