How to Apply GitOps to Everything Using Amazon Elastic Kubernetes Service (Amazon EKS), Crossplane, and Flux

GitOps brought more agility into how developers deploy and manage their cloud-native stack. With GitOps the entire stack is described declaratively, the desired state of the whole system or its building blocks versioned in Git, and approved changes are automatically and reliably applied to the runtime environment.

The open source Crossplane project has enabled developers and operators to provision and manage infrastructure in any cloud service provider using the Kubernetes API. With Crossplane, resources are managed using simple manifests that Kubernetes can apply to cloud providers’ infrastructure. For example, you can now manage RDS databases using Crossplane by creating the necessary manifests and applying them to a Kubernetes cluster; Crossplane will pick up the manifest and provision the Amazon Relational Database Service (Amazon RDS) database using AWS APIs.

Crossplane enables GitOps to be applied virtually everywhere using Kubernetes as a proxy to provision and manage cloud resources. This article will take you in a step-by-step workflow to provision Amazon Elastic Kubernetes Service (Amazon EKS) clusters and an Amazon RDS database the GitOps way using Crossplane and Flux.

The overall workflow

Prepare a Git environment.
Create a management Amazon EKS cluster with eksctl.
Install Flux and Crossplane.
Create the production Amazon EKS management cluster. Push Crossplane manifests to a Git repo and let Flux synchronize them into the management cluster.
Create the application. Push production Amazon EKS cluster Crossplane composite claims to the Git repo and let Flux synchronize them into the management cluster.
a. Push Crossplane composite claims that define a backend app and an in-cluster PostgreSQL database into the Git repo and let Flux synchronize them into the management cluster.
b. Push Crossplane composite claims that define a backend app and an Amazon RDS PostgreSQL database into the Git repo and let Flux synchronize them into the management cluster.
Clean up (optional). Remove manifests from the Git repo and let Flux synchronize (remove) resources from the management cluster. Crossplane, in return, will remove the “real” resources.

Step 1 – Prepare your environment

GitHub Parameters

We need to make sure we have the right GitHub parameters set up. Flux will be looking for these parameters along the way. More on that later on.

# Replace `[...]` with the GitHub organization or user
export GITHUB_ORG=[...]

# Replace `[...]` with the GitHub token
export GITHUB_TOKEN=[...]

# Replace `[...]` with `true` if it is a personal account, or with `false` if it is an GitHub organization
export GITHUB_PERSONAL=[...]

Step 2 – Create a management Amazon EKS cluster

We will need a management cluster to provision our production environment. We will also need to give it access to our AWS account. We create a key and a secret with the necessary credentials to allow our management cluster to provision and manage resources in our AWS environment.

Once we have our key and secret created, we export them as environment variables as shown in the following command line example.

# Replace `[...]` with your access key ID`
export AWS_ACCESS_KEY_ID=[...]# Replace `[...]` with your secret access key
export AWS_SECRET_ACCESS_KEY=[...]

Now we create a management cluster with eksctl.

Note

If you haven’t installed eksctl before, you can find how to set it up in this Amazon EKS user guide.

eksctl create cluster \
--name management \
--region us-east-1

Once our cluster is provisioned, we create a Crossplane namespace with kubectl.

kubectl create namespace crossplane-system

We also need to create a development (dev) namespace. We will use the dev namespace as our dev environment for the sake of this tutorial.

kubectl create namespace dev

Note

It is not recommended to create a dev environment within your management cluster. We are just doing so here to save some resources.

kubectl create namespace clusters

Now we push AWS credentials into our management cluster in the form of a secret.

echo "[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
" >aws-creds.confkubectl --namespace crossplane-system \
create secret generic aws-creds \
--from-file creds=./aws-creds.conf

Step 3 – Install Flux and Crossplane

We can now bootstrap Flux. In this step Flux will install itself into the cluster and bootstrap the GitHub repository for us.

flux bootstrap github \
    --owner $GITHUB_ORG \
    --repository crossplane-flux \
    --branch main \
    --path infra \
    --personal $GITHUB_PERSONAL

We make sure we are pointing Flux to the right directory and using the proper AWS credentials.

git clone \
https://github.com/$GITHUB_ORG/crossplane-fluxcd crossplane-fluxecho "/kubeconfig.yaml
/aws-creds.conf" \
| tee .gitignore

Now, we will install Crossplane managed by Flux. We start by creating a Crossplane Helm release, committing it to our repository, and waiting for Flux to sync it with our just created development cluster.

mkdir infra/crossplane-system

flux create source helm crossplane \
    --interval 1h \
    --url https://charts.crossplane.io/stable \
    --export \
    | tee infra/crossplane-system/source.yaml

flux create helmrelease crossplane \
    --interval 1h \
    --release-name crossplane \
    --target-namespace crossplane-system \
    --create-target-namespace \
    --source HelmRepository/crossplane \
    --chart crossplane \
    --chart-version 1.6.4 \
    --crds CreateReplace \
    --export \
    | tee infra/crossplane-system/release.yaml

git add .

git commit -m "Crossplane"

git push

kubectl --namespace flux-system \
get helmreleases,kustomizations

We wait for a few moments to have it synced with our cluster. Flux syncs every 2 minutes by default.

Now, we will install the Crossplane provider packages. Note that we are careful to not move to the next step until all packages are installed and in a healthy state.

curl -o infra/crossplane-system/providers.yaml \


 https://gist.githubusercontent.com/vfarcic/b5d3ab028fe65cda27438e28415b5c83/raw

git add .git commit -m "Crossplane"git pushkubectl --namespace flux-system \
get helmreleases,kustomizationskubectl get pkgrev

Finally, we will install Crossplane’s AWS provider configurations.

curl -o infra/crossplane-system/provider-config-aws.yaml \

     https://raw.githubusercontent.com/vfarcic/devops-toolkit-crossplane/master/crossplane-config/provider-config-aws.yaml

git add .git commit -m "Crossplane"git pushkubectl --namespace flux-system \
   get helmreleases,kustomizationsexport SA=$(kubectl \
    --namespace crossplane-system \
    get serviceaccount \
    --output name \
    | grep provider-helm \
    | sed -e 's|serviceaccount\/|crossplane-system:|g')kubectl create clusterrolebinding \
    provider-helm-admin-binding \
    --clusterrole cluster-admin \
    --serviceaccount="${SA}"

Step 4 – Create the production management cluster

Now that we have our development cluster created and ready to manage our AWS resources, let’s start by creating our production management cluster.

First, we create a directory to store the cluster’s configurations.

mkdir infra/clusters

We need to create the cluster’s specification file. Instead of creating all needed AWS structures manually, such as VPCs, subnets, etc., we are using a Crossplane composition that explains all of the structures listed in the command line example in the ClusterClaim.

echo "apiVersion: devopstoolkitseries.com/v1alpha1
kind: ClusterClaim
metadata:
name: production
namespace: flux-system
spec:
id: production
compositionSelector:
    matchLabels:
      provider: aws
      cluster: eks
parameters:
    nodeSize: small
    minNodeCount: 3
writeConnectionSecretToRef:
    name: production-cluster" \
    | tee infra/clusters/production.yaml

We add, commit, and push the code for Flux to sync the changes.

git add .

git commit -m "Cluster"

git push

kubectl --namespace flux-system \
get clusterclaims

Note

It will take 20-30 minutes to have the cluster and all needed resources created at AWS. Proceed to the next step while your production cluster is being created.

Step 5 – Create your application

Now we will create an application that will be fully GitOps managed. That application will use an Amazon RDS Postgres database which will be managed using GitOps + Crossplane. We will use our management cluster as our development environment to save some resources. This, however, is not recommended. You should have a dedicated development cluster that is an identical scaled-down replica of your production environment. You should then deploy it to your production cluster after making sure it is fully functional.

First, we create a folder for our development app.

mkdir dev-apps

Next, we create the application claim.

echo "apiVersion: devopstoolkitseries.com/v1alpha1
kind: AppClaim
metadata:
name: silly-demo
namespace: dev
spec:
id: silly-demo-dev
compositionSelector:
    matchLabels:
      type: backend-db
parameters:
    namespace: dev
    image: vfarcic/sql-demo:0.1.10
    port: 8080
    host: dev.backend.acme.com---
apiVersion: devopstoolkitseries.com/v1alpha1
kind: SQLClaim
metadata:
name: silly-demo
namespace: dev
spec:
id: silly-demo-dev
compositionSelector:
    matchLabels:
      provider: local-k8s
      db: postgresql
parameters:
    version: \"13.4\"
    size: small
    namespace: dev
writeConnectionSecretToRef:
    name: silly-demo-dev" \
    | tee dev-apps/backend.yaml

We will now push it to our Git repository and wait for Flux to sync it with the cluster. Please notice that we are telling Flux about the new application repository with the flux create command.

Deploy the application to the development cluster

git add .

git commit -m "Backend"

git push

#Let’s tell flux about the new repository that has our application.

flux create kustomization dev-apps \
    --source GitRepository/flux-system \
    --path dev-apps \
    --prune true \
    --interval 1m

kubectl --namespace dev \
get appclaims,sqlclaims

kubectl --namespace dev \
get all,ingresses,secrets

Now, the application is deployed to the development environment. Let’s check if the production cluster and all required AWS resources are created.

#This command will show you all the resources and their status

kubectl get managed

We need to check if our cluster is ready or not.

kubectl get clusters

Let’s double check that our claims sync and are fully created.

kubectl --namespace crossplane-system \
    get secret production-cluster \
    --output jsonpath="{.data.kubeconfig}" \
    | base64 -d >kubeconfig.yaml

Once the production cluster is ready, we create a secret that has AWS credentials inside our production cluster to enable it to manage production AWS resources, as we did with the development cluster.

kubectl --kubeconfig kubeconfig.yaml \
    get nodesecho "[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
" >aws-creds.confkubectl --kubeconfig kubeconfig.yaml \
    --namespace crossplane-system \
    create secret generic aws-creds \
    --from-file creds=./aws-creds.conf

Deploy the Application to the production cluster

As we did with the development environment, we need to first create the production app directory.

mkdir prod-apps

We create the app’s Claim here.

echo "apiVersion: devopstoolkitseries.com/v1alpha1
kind: AppClaim
metadata:
name: silly-demo
namespace: production
spec:
id: silly-demo
compositionSelector:
    matchLabels:
      type: backend-db
parameters:
    namespace: production
    image: vfarcic/sql-demo:0.1.10
    port: 8080
    host: devops-toolkit.127.0.0.1.nip.io
---
apiVersion: devopstoolkitseries.com/v1alpha1
kind: SQLClaim
metadata:
name: silly-demo
namespace: production
spec:
id: silly-demo
compositionSelector:
    matchLabels:
      provider: aws
      db: postgresql
parameters:
    version: \"13.4\"
    size: small
    namespace: production
writeConnectionSecretToRef:
    name: silly-demo" \
    | tee prod-apps/backend.yaml

We push the Claim to our Git repository and make sure that Flux syncs it with the production cluster.

git add .

git commit -m "Backend"

git push

mkdir -p tmp

flux create kustomization prod-apps \
    --source GitRepository/flux-system \
    --path prod-apps \
    --prune true \
    --interval 1m \
    --export \
    | tee tmp/prod-apps.yaml

Next we need to edit our prod-apps.yaml to reference our production secret by setting `spec.kubeConfig.secretRef.name` to `production-cluster`.

At this point, we need to apply the updated files to our production cluster.

kubectl apply \
--filename tmp/prod-apps.yaml

We need to check if our application is provisioned. There are different ways to do so as explained here.

#In case you want to check the status of your app and SQL claims you just applied

kubectl --kubeconfig kubeconfig.yaml \
--namespace production \
get appclaims,sqlclaims

#In case you want to check all relevant K8s resources
kubectl --kubeconfig kubeconfig.yaml \
--namespace production \
get all,ingresses,secrets

#This will tell you if your cluster, including provisioned resources, is in ready state.
kubectl --kubeconfig kubeconfig.yaml \
get managed

Step 6 – Clean up (optional)

To destroy all of the provisioned resources created in this post, run the following script.

# Removes the dev app manifests

rm -rf delete dev-apps/*.yaml

touch dev-apps/dummy

# Removes the dev app manifests
rm -rf delete prod-apps/*.yaml

touch prod-apps/dummy

# Removes them from the GitHub repositories
git add .

git commit -m "Remove apps"

git push

# Flux will remove all resources using CrossPlane at this point.

#Let’s check the status of clusters and resources
kubectl --kubeconfig kubeconfig.yaml \
get managed

# Repeat the previous command until all the `aws` resources are deleted

kubectl --kubeconfig kubeconfig.yaml \
--namespace ingress-nginx \
delete service production-ingress-ingress-nginx-controller

rm -rf infra/clusters/*.yaml

git add .

git commit -m "Destroy"

git push

kubectl get managed

# Repeat the previous command until all the `aws` resources are deleted

eksctl delete cluster \
--name management \
--region us-east-1

gh repo view --web

# Delete the repo

cd ..

rm -rf crossplane-flux

Conclusion

Amazon EKS combined with Flux and Crossplane enables Kubernetes users to deploy AWS resources (such as Amazon RDS in this example) using Kubernetes APIs (via manifests). Now, Kubernetes users do not have to use an Infrastructure as Code tool that has its own domain specific language to deploy AWS resources, which speeds up infrastructure and application deployments. And finally, when combined with Flux, users can ensure that their infrastructure is consistent because the source of truth is always in their Git repo.

AWS Open Source Blog

How to Apply GitOps to Everything Using Amazon Elastic Kubernetes Service (Amazon EKS), Crossplane, and Flux

The overall workflow

Step 1 – Prepare your environment

GitHub Parameters

Step 2 – Create a management Amazon EKS cluster

Step 3 – Install Flux and Crossplane

Step 4 – Create the production management cluster

Step 5 – Create your application

Deploy the application to the development cluster

Deploy the Application to the production cluster

Step 6 – Clean up (optional)

Conclusion

Resources

Follow