Multi-cluster management for Kubernetes with Cluster API and Argo CD
Customers are moving to cloud thanks to its scalability and cost-effectiveness. Some are adopting Kubernetes to modernize their workloads for a more agile way to iterate and deliver time-to-market services or product features to their customers. As the number of clusters to be managed increases, customers struggle with different deployment mechanisms and application programming interfaces (APIs) for managing Kubernetes clusters. They’re looking for a more centralized approach to create and manage clusters running in different formats, instead of using separate tools to manage each. In this post, we’ll show you how to pair Cluster API and Argo CD to streamline deployment and operation of multiple Kubernetes clusters.
What is Cluster API?
Cluster API is a Kubernetes project that enables declarative management for Kubernetes, using APIs to easily create, configure, and update clusters. In October 2021, the Cloud Native Computing Foundation (CNCF) announced Cluster API v1.0 is production-ready, its growing adoption, feature maturity, and a strong commitment to community and inclusive innovation.
Cluster API Provider for aws (CAPA), maintained by an opensource community, is a SIG Cluster Lifecycle project. You can deploy and upgrade Kubernetes clusters by handling infrastructure provisioning in AWS. CAPA enables provisioning of Amazon EC2 and Amazon EKS based Kubernetes clusters. In this post, we’ll cover both to simulate a multi-cluster scenario where customer needs to operate Kubernetes cluster either running on a self-managed cluster backed by Amazon EC2 or the other on Amazon EKS. Amazon EKS Anywhere uses Cluster API provider for vSphere or bare metal for creating, upgrading, and managing Kubernetes clusters.
For Amazon EKS, CAPA has the following features:
- Provisioning and managing an Amazon EKS Cluster
- Upgrading the Kubernetes version of the Amazon EKS Cluster
- Attaching a self-managed machines as nodes to the Amazon EKS cluster
- Creating a machine pool and attaching it to the Amazon EKS cluster. See machine pool docs for details.
- Creating a managed machine pool and attaching it to the Amazon EKS cluster. See machine pool docs for details
- Managing Amazon “EKS Addons”. See addons for further details
- Creating an Amazon EKS AWS Fargate profile
- Managing an AWS Identity and Access management (AWS IAM) aws-iam-authenticator configuration
What is Argo CD
Argo CD is a declarative, GitOps continuous-delivery tool for Kubernetes. It follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. With Argo CD, you automate the deployment of the application as defined in Git repository to your target cluster environments.
How it works – High-level orchestration flow
In this walk through, we use Kind to create a management cluster in the local docker machine.
(Note: kind isn’t designed for production use. It’s used here for demonstration purpose. For production, it’s recommended to use a dedicated Kubernetes cluster, such as Amazon EKS, with appropriate backup and DR policies and procedures in place. The Kubernetes cluster must be at least v1.20.0.).
We initialize the management cluster with Cluster API CLI. The whole cluster and application deployment process is managed by the ArgoCD installed on the Kind management cluster. Argo CD supports multi-clusters and allows you to define an application to link up the Kind cluster and the to-be-created workload clusters (running on Amazon EC2 and Amazon EKS) to your Git repositories where workload configuration is stored. After the creation of workload clusters, we enroll them to Argo CD as a managed cluster to further deploy a sample application to them.
- An understanding of Amazon EKS, Argo CD, and Kubernetes
- Complete installation of kubectl, kind, clusterctl, clusterawsadm, argocd cli, aws cli, jq, and docker
- An AWS account and a profile with administrator permissions should be configured
You may refer to the following links for installing the necessary CLI toolings:
Fork the Git repository and clone it to your local workstation
This post provides sample manifest file for setting up the environment. To access them, you must create a fork of this GitHub repository and clone.
Change to the cloned directory:
Initialize the management cluster
Kind is used for creating a local Kubernetes cluster for the creation of a temporary bootstrap cluster used to provision a target management cluster on the Cluster API provider for AWS.
Now, let’s create the kind cluster and verify the success of the creation:
You should be able to see the following when the Kind cluster is created successfully.
The Kind Kubernetes cluster is transformed into a management cluster by installing the Cluster API provider components. It’s recommended to separate the Kind Kubernetes cluster from any application workload.
Cluster API Provider for AWS ships with
clusterawsadm, a utility to help you manage IAM objects for this project. The
clusterawsadm binary uses your environment variables and encodes them in a value to be stored in a Kubernetes Secret of the Kind cluster to fetch necessary permissions to create the workload clusters.
Now, let’s predefine the necessary environment parameters:
Since we’ll use Cluster API to help us launch an EKS cluster to, we need to enable the following necessary feature gates:
To install the Cluster API components for AWS, we’ll use the kubeadm bootstrap provider, and the
kubeadm control-plane provider, and transform the Kind cluster into a management cluster with the
clusterctl init command.
If you see this output, then it means that you have successfully initialized the Kind cluster into a management cluster.
Generate the Kubernetes Cluster manifests for Amazon EKS and Amazon EC2
In this step, we provision one Kubernetes cluster on Amazon EC2 and one Amazon EKS cluster, in
ap-southeast-2 Regions, respectively. Since we need to provision the workload clusters via Argo CD, the first step is to prepare the manifest files for each cluster. One quick way to do this is to use
clusterctl generate cluster and the command line to capture the environment variables you defined to formulate the manifests.
The following is a list of environment variables you should define:
Since we haven’t created the secure shell (SSH) key for remote accessing the worker nodes of the clusters, let’s create them:
Now, we can create our workload cluster manifests by running the following command. (If you cloned the Git repository, the following manifests should exist under the
K8S cluster running on Amazon EC2, if you want to generate a new cluster template instead of using the one provided, then run the following command:
For Amazon EKS, if you want to generate a new cluster template instead of using the one provided, then run the following command: (the
eks-managedmachinepool dictates that managed node group should be used when creating the work nodes for Amazon EKS cluster):
Install Argo CD on the management Kubernetes cluster
According to Cluster API documentation, one way to deploy a new Amazon EKS/K8S cluster is to run kubectl apply to apply the generated manifests into the Kind management cluster. However, we’ll use Argo CD to streamline the workload cluster deployment based on the generated manifest stored in Git repository.
First, let’s install Argo CD in the Kind management cluster:
Now, let’s log in to Argo CD’s web portal by using port-forwarding:
Log in to Argo CD to verify the installation is correctly performed:
Create application in Argo CD to deploy a new Amazon EKS cluster and K8S cluster that runs on Amazon EC2
Argo CD provides an Application CRD that maps the configuration code of a given Git repository to a Kubernetes namespace. Before we deploy the Argo CD application in the management cluster, we first need to ensure that Argo CD can be authenticated to Git. To do this, log in Argo CD’s web UI, navigate to the Setting gear on menu bar on the left hand side, then choose CONNECT REPO USING HTTPS. Fill in the proper settings to connect to your Git repo.
https://github.com/aws-samples/eks-ec2-clusterapi-gitops (Your should replace it with your forked git repo URL)
For Argo CD to be authorized to manage Cluster API objects, we create a
ClusterRoleBinding to associate necessary permissions to the service account assumed by Argo CD. For simplicity, let’s add the
cluster-admin role to the
argocd-application-controller ServiceAccount used by Argo CD.
To create a new Amazon EKS cluster or a K8S cluster that runs on Amazon EC2, we define an application in a declarative approach. This represents of a collection of Kubernetes manifests that makes up all the pieces to deploy the new cluster. In this guide, the configurations of the Argo CD application to be added are stored in the
Management folder of the cloned repository. For example, the following is the application manifest file for creating a new K8S cluster that runs on Amazon EC2:
After modifying the Argo CD application
yaml file as above, commit and push the updates to the source repository:
Run the following command to create a new Application to create the K8S cluster that runs on Amazon EC2:
Similarly, we’ll use Argo CD application to help us spin up the Amazon EKS cluster. An application was created for this example. Remember to modify the corresponding
yaml file to ensure
repoURL points to your own Git repository, then commit and push the update to the source repository.
Apply the Argo CD application to spin up the Amazon EKS cluster:
As you deploy the two applications in Argo CD, you should be able to see them created in Argo CD’s web UI, as shown in the following diagram:
Now, we list all the clusters being provisioned:
Check status of the clusters:
For the cluster running on Amazon EC2, you should notice that
READY status of worker nodes remains
False. Since there’s no CNI in place in the cluster, the worker nodes haven’t joined to the cluster successfully. While you could automate the CNI deployment process via Argo CD, let’s deploy the Calico CNI into the Amazon EC2 cluster for demonstration purpose.
Fetch the kubeconfig file for the newly created cluster on Amazon EC2:
Deploy Calico CNI:
As Calico CNI is successfully deployed, check the cluster status again.
Navigate to the Amazon EC2 service in AWS Management Console and select
us-east-1 region. We see that all the control plane and worker nodes have spun up running.
Let’s also verify the Amazon EKS cluster has been deployed successfully:
Navigate to the Amazon EKS service in AWS Management Console, select
ap-southeast-2 region and we should see the cluster created.
When creating an Amazon EKS cluster there are
kubeconfigs generated and stored as secrets in the management cluster. This is different to when you create a non-managed cluster using the AWS provider. The name of the secret that contains the
[cluster-name]-user-kubeconfig, where you need to replace [cluster-name] with the name of your cluster.
To get the user
kubeconfig for a cluster named
capi-eks you can run a command similar to:
List all available worker nodes of the newly created Amazon EKS cluster:
Deploy a sample application into the newly created clusters
For us to be able to deploy a sample application into the workload clusters via Argo CD, we need to add the Amazon EC2/Amazon EKS cluster to Argo CD as a managed cluster. For demonstration purpose, I’ll only show the steps for deploying a sample
Nginx application onto the Amazon EKS cluster. The steps for deploying to Amazon EC2 cluster should be quite similar. First, you need to enroll the Amazon EKS cluster as a managed cluster in Argo CD.
Log in to ArgoCD instance (with port forwarding enabled). We use
argocd CLI to authenticate against Argo CD server:
argocd login localhost:8080 (when asked if to proceed with insecure connection, say yes.)
Get the context name of the newly created Amazon EKS cluster:
Run the following command to add the newly created Amazon EKS cluster as a managed cluster in Argo CD:
yes with the following warning:
Next, we should create a new Application in Argo CD to allow it to streamline the sample application deployment to the Amazon EKS cluster. Before applying the following command, modify the manifest file above to ensure
repoURL points to your Git repository, then commit and push the updates.
Apply the following command to deploy the sample
In Argo CD’s Web portal, you see that the sample
nginx app has been deployed into the Amazon EKS cluster successfully.
To make sure there’s no unwanted cloud cost, please clean up the environment.
Delete the Argo CD applications:
Delete the created clusters:
Delete the created Amazon EC2 SSH key in each Region:
Delete the CloudFormation Stack when running clusterawsadm bootstrap command to create all necessary IAM resources:
Delete management cluster:
In this post, I introduced what Cluster API is and explained why you can use this useful tool for managing multiple Kubernetes clusters instead of struggling with different APIs and tool sets to maintain them. By pairing with another GitOps tool, called Argo CD, you can take Git as the single source of truth to store all your cluster and workload’s manifests file and let Argo CD continuously deliver new workloads or workload configuration updates to the target environment.