Using CDK to perform continuous deployments in multi-region Kubernetes environments
This post also had contributions from Jiwon Yeom, Solutions Architect, AWS.
AWS customers use infrastructure as code (IaC) to create cloud resources in a repeatable and predictable manner. IaC is especially helpful in managing environments with identical stacks, which is a common occurrence in active-active multi-region systems. Instead of managing each regional deployment individually, teams avoid infrastructure sprawl by managing infrastructure in both regions using code and pipelines.
Teams need fewer resources to manage and troubleshoot infrastructure when their system configuration is identical across regions or accounts. Using IaC tools like AWS CloudFormation, Terraform, Pulumi, and AWS Cloud Development Kit (CDK), operators create and manage multiple environments from a single codebase and avoid system failures caused by ad hoc changes. IaC also makes it easier to rollback when a change fails.
Similarly, the GitOps approach allows you to externalize your Kubernetes cluster configuration. GitOps tools like ArgoCD allow you to store your Kubernetes cluster state as code in a Git repository. If you manage multiple replica clusters, which is a common scenario in multi-region deployments, then you can control their configuration by pointing your GitOps operator to the same Git repository.
Unified infrastructure and workload deployments
Customers that want to manage their infrastructure and Kubernetes resources using the same code library with a variety of available solutions. Teams already using Terraform can package their applications using the Helm and Kubernetes provider. Pulumi users have Pulumi Kubernetes Provider.
Similarly, CDK and cdk8s allow you to code both the cloud infrastructure and Kubernetes resources using familiar programming languages. Customers can declare their Virtual Private Clouds (VPCs), Subnets, Amazon EKS clusters, Kubernetes namespaces, role-based access control (RBAC) policies, deployments, services, ingresses, and other tasks in the same code base. The Git repository that hosts the application code also becomes the source of truth for infrastructure and workload configuration.
Imagine you work for an Independent Software Vendor (ISV) with a security sensitive product that can’t be multi-tenant. The system design requires one Amazon EKS cluster per tenant. When the product becomes successful, your organization may be responsible for operating a multitude of replica clusters.
In scenarios where the cluster lifecycle is closely tied to the application’s, you can store the definition of your infrastructure, Amazon EKS cluster, and application resources in the same codebase. To deploy a new stack, you rerun the CDK code, which creates the infrastructure resources required to run the application and then deploys the application in an Amazon EKS cluster.
Day 2 operations, such as deploying new versions of applications and upgrading clusters, are also managed using CDK. We recommend checking code using a tool like cdk-nag, along with using standardization and parametrization to minimize the variability between regions.
The code included in this post creates two proof-of-concept Amazon EKS clusters and the supporting infrastructure in two AWS Regions.
CDK also creates a stack with
CodePipeline to build a container image from the sample application’s source code. Users trigger an automated deployment to the primary AWS Region by checking in code to the AWS CodeCommit Git repository CDK creates. CodePipeline triggers AWS CodeBuild when new changes are checked-in to the Git repository and the result is an image pushed to an Amazon Elastic Container Registry (Amazon ECR) repository.
The CDK code this post provides doesn’t include multi-region traffic routing. Customers can add a Global Accelerator, as explained in this post, or use Domain Name System (DNS) to route traffic between regions.
You will need the following to complete the steps in this post:
- AWS CLI version 2
- AWS CDK version 2.19.0 or later
- Node version 17.8.0 or later
- NPM version 8.5.5 or later
Let’s start by setting a few environment variables:
Clone the sample repository and install dependency packages. This repository contains CDK v2 code written in TypeScript.
The Amazon EKS cluster definition is stored in
lib/cluster-stack.ts. The following is the snippet of the code:
As defined in
bin/multi-cluster-ts.ts, the code creates two clusters and deploys the supporting infrastructure, the
You can customize the
multi-cluster-ts.ts file to change regions and instance types based on your requirements.
Bootstrap AWS Regions
The first step to any CDK deployment is bootstrapping the environment.
cdk bootstrap is a tool in the AWS CDK command-line interface (AWS CLI) responsible for preparing the environment (i.e., a combination of AWS account and AWS Region) with resources required by CDK to perform deployments into that environment. If you already use CDK in a region, you don’t need to repeat the bootstrapping process.
Execute the commands below to bootstrap the AWS environment in
cluster-stackthat creates an Amazon EKS cluster for primary region
container-stackthat deploys sample nginx containers to Amazon EKS cluster for primary region
cluster-stackthat creates an Amazon EKS cluster for secondary region
container-stackthat deploys sample nginx containers to Amazon EKS cluster or secondary region
cicd-stackthat creates a cicd pipeline using AWS CodePipeline and AWS CodeSuites to containerize and deploy a sample flask application that crosses regions of Amazon EKS
cdk list to see the list of the stacks to be created:
You can use
cdk diff to get a list of resources CDK will create:
If the result doesn’t match the output below, then refer to this step and check if npm run watch is running in the background.
You should see the following output: