Comparing AWS Cloud Development Kit and AWS Controllers for Kubernetes

DevOps is a common denominator for software delivery across industries. No matter the software, developers must ensure that infrastructure resources are provisioned; testing and delivery mechanisms are in place; and security, reliability, and scalability requirements are provided. That is why choosing the right DevOps tooling is central to a delivery team’s best practices, particularly in combination with continuous integration and delivery (CI/CD) strategies.

At Amazon Web Services (AWS), we offer tools and technologies to automate infrastructure and software rollout strategies. Two of these tools are the AWS Cloud Development Kit (AWS CDK) and AWS Controllers for Kubernetes. One challenge customers frequently encounter is how to decide which tool to choose. Although AWS Cloud Development Kit and AWS Controllers for Kubernetes both allow you to achieve the same goals of creating AWS resources, both are open source, and both are actively under development, key differences between the two may lead to customers preferring one tool over the other.

In this blog post, we outline comparison points that may help you decide which option works better for your needs. We base our comparison on the following six paradigms:

Programming
Validation
Execution
Interaction
Operation
Reconciliation

1. Programming

At a higher level, the programming paradigm captures the declarative vs. imperative model of defining infrastructure resources in AWS CDK vs. AWS Controllers for Kubernetes.

AWS Controllers for Kubernetes follows the model popularized by Kubernetes in defining the expected end-state shape of required AWS infrastructure as a YAML document, with resource dependencies only loosely defined as references and selectors. The shape of declarative AWS resources in AWS Controllers for Kubernetes follows backend REST endpoints, with little flexibility in tweaking parameters and configurations, making it easier to manage and maintain. Finally, the internal machinery in Kubernetes cobbles resources together based on defined references and as resources become available.

AWS CDK offers higher-level constructs for AWS resources that are intent-based. AWS CDK is built on top of full-blown programming languages (namely, TypeScript, Python, Go, .Net, and Java), which means that the entire feature sets of these programming languages are at your disposal when writing resource provisioning scripts. As such, you are responsible for defining and referencing required resources in your scripts. Although no particular ordering is enforced by AWS CDK when resource provisioning requests are processed, CDK programmers must be mindful of defining their resource objects in the right order to satisfy resource definition construct in the code they write. When writing AWS CDK code, you are also responsible for ensuring readability, maintainability, and reusability by others in the organization.

Consider the example of creating an Amazon Simple Storage Service (Amazon S3) bucket with a sample tag and with versioning enabled. Following is the declarative YAML definition of the resource for AWS Controllers for Kubernetes:

s3-resource.yaml
apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: ack-bucket
spec:
  name: ack-bucket
  versioning:
    status: Enabled
  tagging:
    tagSet:
    - key: "app"
      value: "test"

Creating the same resource using AWS CDK’s TypeScript runtime requires importing the respective libraries, creating the AWS CDK stack, and creating the Amazon S3 bucket and the tag in the context of the stack:

s3-resource.ts
import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import {Tags} from '@aws-cdk/core';

export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'cdk-bucket', {
      versioned: true
    });

    Tags.of(this).add("app", "test");
  }
}

2. Validation

As with other software or hardware artifacts, proper testing and validation can increase your confidence in the infrastructure automation process. In the case of AWS Controllers for Kubernetes, the Kubernetes API server will validate the declaration of custom resources as they are applied. For example, submitting a resource with wrong fields could result in a validation error by the Kubernetes API server as shown in the following:

> kubectl apply -f s3-resource.yaml
error: error validating "resources/bucket.yml": error validating data: 
ValidationError(Bucket.spec): unknown field "wrongKey" in aws.k8s.services.s3.v1alpha1.Bucket.spec; 
if you choose to ignore these errors, turn validation off with --validate=false

Additional verification can be run either against created AWS Controllers for Kubernetes resources in a Kubernetes cluster, or directly against AWS service via API calls.

For AWS CDK, because the runtime is built on top of a full-blown programming language, you can rely on static type checking and even go one step further and write unit tests against your AWS CDK code. Unit tests can check on the AWS CloudFormation outputs and validate resource-specific requirements. This provides stronger assurances around resource dependencies and validating non-functional requirements (for example, security requirements), without needing to interface with external API endpoints.

3. Execution

AWS Controllers for Kubernetes is an extension on top of the Kubernetes APIs and lends itself heavily to the Kubernetes execution model—that is, introducing custom resource definitions (CRDs) for AWS resources. AWS Controllers for Kubernetes uses purpose-built Kubernetes operators that interpret YAML declaration of AWS custom resources, calling AWS API operations that respectively carry out AWS resource creation, update, and deletion tasks.

AWS Controllers for Kubernetes also uses the Kubernetes role-based access control (RBAC) system in combination with AWS identity providers, policies, and trust relations to determine who has the permissions to create any resource. As such, choosing AWS Controllers for Kubernetes, at the minimum, implies that you adhere to maintaining a Kubernetes cluster responsible for executing AWS Controllers for Kubernetes operators and storing the intended state of the world for your infrastructure resources. This Kubernetes cluster could be a part of a fleet of infrastructure for your software, or an entirely separate cluster.

AWS CDK, however, is built on top of common programming language frameworks and by extending their list of modules and libraries. The runtime for AWS CDK is the same programming language runtime with which DevOps engineers are most likely already familiar. Thus, executing the AWS CDK code is as simple as using the language runtime to execute code. Access control is also controlled directly via the AWS Identity and Access Management (IAM) service. AWS CDK’s libraries and modules then output AWS CloudFormation templates, which take care of making the appropriate calls against AWS API operations for resources to get created.

4. Interaction

In the interaction paradigm, we capture how either AWS Controllers for Kubernetes or AWS CDK interface with the AWS APIs to manage AWS resource lifecycles. For AWS Controllers for Kubernetes, this happens through service-specific Kubernetes controllers. AWS custom resource definitions capture AWS resource specifications, and a given controller uses AWS SDK for Go as the underlying library to make direct REST calls against AWS API operations. AWS Controllers for Kubernetes is only responsible for AWS control plane resources and is not intended for uploading data or connecting to resource data planes.

For AWS CDK, however, there is a slight indirection in that the TypeScript or Python code for AWS resources is transpiled to the corresponding AWS CloudFormation templates, which then get submitted to AWS CloudFormation in order for provisioning of AWS resources to happen. Along the way, a copy of the AWS CloudFormation templates is also stored locally on the machine where the transpiling has taken place. To this end, AWS CDK offers the value of imperative definition with declarative provisioning. The following code snippet shows the CloudFormation specification for the Amazon S3 bucket that we defined in s3-resource.ts:

Resources:
  cdkbucketF4700AFF:
    Type: AWS::S3::Bucket
    Properties:
      Tags:
        - Key: app
          Value: test
      VersioningConfiguration:
        Status: Enabled
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    Metadata:
      aws:cdk:path: CdkStack/cdk-bucket/Resource

5. Operation

By fully embracing Kubernetes, AWS Controllers for Kubernetes piggybacks on the operational paradigms available in the Kubernetes ecosystem. This implies that any deployment strategy available to a Kubernetes application also can be used to deploy AWS infrastructure resources. GitOps, as an example, plays natively with AWS Controllers for Kubernetes, where you can use Git to store AWS infrastructure resource specifications and use technologies such as ArgoCD and Weaveworks Flux to manage the lifecycle of those resources. In case of infrastructure provisioning problems, debugging involves searching controller logs for error messages as well as looking for error events or status changes on custom AWS resources. For a detailed list of existing controllers and respective AWS services as well as their levels of maturity, check out the documentation page.

The resource provisioning model in AWS CDK is on demand, where an automated or manual trigger results in the invocation of the AWS CDK runtime against the CDK code to synthesize or deploy code, which in turn leads to making the necessary API calls for the AWS CloudFormation change set to be executed. This flow is generally encoded into a delivery pipeline that captures the trigger and runs a script to launch the runtime and carry on with code execution.

Because AWS CloudFormation templates are the final output of AWS CDK runtime execution, debugging provisioning problems involves inspecting the final output templates, debugging AWS CDK code itself, and reading CloudFormation event logs. When it comes to maturity, AWS CDK has been under development for more than two years. AWS CDK has support for many AWS services and features with rich, high-level constructs, and complete coverage of the lower-level CloudFormation resources. The full list of AWS services supported in AWS CDK is available under the AWS Construct Library API Reference.

6. Reconciliation

Reconciliation is a response to drift, where the perceived state of the world diverges from the expected state of the world, and reconciliation happens to make the latter match the former.

A common case of drift for infrastructure resources happens where out-of-band modifications are made to infrastructure resources. For example, if one bypasses deployment pipelines and manually adjusts the cache size in an Amazon Simple Queue Service (Amazon SQS) instance or changes the number of nodes in a Kubernetes cluster, a drift has been introduced.

Reconciliation is inherent to any Kubernetes controller, and AWS Controllers for Kubernetes are no exception. AWS Controllers for Kubernetes consistently monitor the state of AWS infrastructure resources, contrast them against submitted resource objects in a Kubernetes cluster, and revert any drift in actual AWS resources for them to match the submitted resources. Long-running reconciliation loops that exist in AWS Controllers for Kubernetes serve as an effective way to prevent undesired modifications to resource specifications.

For AWS CDK, although out-of-band resource modifications are considered an anti-pattern, there are no built-in guardrails to prevent such practices. AWS CloudFormation does detect resource definition drifts; however, it does not offer built-in remediation capabilities. As such, reconciliation must be built into the delivery pipelines, or there will be a risk for a drift to happen or go unnoticed.

Conclusion

The following image captures the execution paths for AWS CDK and AWS Controllers for Kubernetes, respectively, and from development all the way to resource creation in AWS infrastructure.

the execution paths for AWS CDK and AWS Controllers for Kubernetes, respectively, and from development all the way to resource creation in AWS infrastructure

As discussed in this post, no one-size-fits-all approach exists for infrastructure tooling. Choosing the right tool for an environment depends on existing software practices, the type of workload you are deploying, and the existing expertise within a delivery team.

If you and your team are already immersed the Kubernetes ecosystem, or have plans to be so, AWS Controllers for Kubernetes could be the preferred tool option. If you have bought into GitOps practices or plan to modernize your delivery practices with GitOps, AWS Controllers for Kubernetes will naturally fit in.

On the other hand, if you have encoded your resource specifications into scripts in a CD pipeline, prefer the tooling and support available to full-blown programming languages, or manage workloads and applications independent of Kubernetes, AWS Cloud Development Kit could get you up and running quickly.

We would like to thank Elad Ben-Israel, Paul Roberts, Nuatu Tseggai, and Vara Bonthu for their help with reviewing this blog post.