Declarative provisioning of AWS resources with Spinnaker and Crossplane
This post was written by Steve Borrelli, Rob Clark, Manabu McCloskey, Vikrant Kahlir, and Nima Kaviani.
In a previous blog post, we discussed how GitOps, declarative definition of infrastructure and application resources, and using technologies such as AWS Controllers for Kubernetes (ACK) and Crossplane have enabled DevOps engineers to reduce complexity and improve visibility into infrastructure rollout and application resources.
For small- and medium-sized cloud native companies, switching to a new GitOps model that takes advantage of modern GitOps tooling—such as Argo CD or Flux CD—is a relatively small barrier to entry. For bigger enterprises, however, with established CI/CD automations in place, transitioning away from their reliable mechanisms of rolling out software may not be as trivial. This is where modernization and enhancements to their existing CI/CD platforms would allow for more easily embracing newer technologies.
One of these battle-tested CI/CD platforms heavily used by some of our enterprise-grade strategic customers—such as Netflix, Autodesk, Airbnb, Salesforce, Pinterest, and Snap—is Spinnaker. In a recent blog post, we discussed the collaboration between Netflix and AWS in enabling GitOps for the Spinnaker community via enhancements done for Managed Delivery and its respective Spinnaker microservice, Keel. The intentions behind supporting managed delivery in Spinnaker are to make it easier for Spinnaker users to embrace GitOps with little to no interruption to their existing software rollout strategies. Also, to provide higher-level abstractions for defining software rollout strategies that would abstract away lower-level infrastructure complexities and allow Spinnaker users to move faster and leaner when it comes to deploying software.
In this blog post, we combine the best of these two worlds, using GitOps support in Spinnaker for declarative rollout of infrastructure and application resources via Crossplane. Crossplane is an open source project that allows teams to manage cloud resources in a Kubernetes-native way. This means you can provision and manage Amazon Simple Storage Service (Amazon S3) buckets, VPCs, databases, and Lambda functions in the same way you manage Pods, Ingresses, and StatefulSets. Because Crossplane managed resources are exposed as Kubernetes Custom Resource Definitions, a GitOps pattern can be used to manage infrastructure with Keel’s native Kubernetes support.
In the rest of this blog post, we will discuss the setup and configuration required for Spinnaker and Crossplane to work together, and steps to take for declarative deployment of infrastructure and application resources to AWS Cloud.
Before getting started, ensure that you completed the following:
- Set up a Spinnaker and Keel managed delivery environment (including GitHub Webhooks).
- The AWS Command Line Interface (AWS CLI) must be installed and configured with administrative permissions.
- Configure your Spinnaker deployment to deploy to the Kubernetes cluster with Crossplane installed on it. Refer to the Spinnaker documentation or the Armory documentation for tutorials.
- The Upbound CLI must be installed.
- The kubectl Crossplane plugin must be installed.
Declarative definition of resources in Crossplane
Crossplane models cloud infrastructure as Kubernetes Custom Resource Definitions (CRDs) called managed resources. For example, in Crossplane an Amazon S3 bucket would be defined using the following YAML, which could be embedded in a Keel declarative k8s/resource:
Like any other Custom Resource, Crossplane resources adhere to Kubernetes API conventions. The spec is validated using OpenAPI, and the forProvider stanza includes the settings that can be configured. DeletionPolicy tells Crossplane what to do when the Kubernetes resource is deleted: Delete the managed resource or orphan it.
Deploying AWS infrastructure using Keel and Crossplane
Installing Universal Crossplane with the AWS Provider
Once Spinnaker and Keel are installed, install Universal Crossplane (UXP), a CNCF-conformant distribution of Crossplane. The easiest method is using the up CLI. First, ensure that your current kubectl context is pointing to the correct cluster, our Amazon Elastic Kubernetes Service (Amazon EKS) Spinnaker cluster:
Next, run the following command to install UXP. Refer to the documentation for configuration options and upgrading from an existing Crossplane installation.
To validate the installation, ensure that all the deployments are in a READY state:
A Crossplane Composition allows an infrastructure team to create an abstraction that combines a number of managed resources in an opinionated way, providing fewer things that end users could configure. For example, one could define a ContainerTestEnvironment that includes a VPC, an Amazon EKS cluster, and Amazon Simple Queue Service (Amazon SQS) queues that only allow testers to define the name of the environment and number of nodes in the cluster.
This package will pull down the AWS and Helm Crossplane providers, and install custom compositions. To validate the package installation, run the following command to ensure that all components are installed and healthy:
This command returns one configuration package and two providers (AWS, Helm).
Setting up AWS authentication to Crossplane
We also want to load our AWS credentials into the control cluster so that Crossplane is able to provision infrastructure on our behalf. We will load the credentials into a Kubernetes secret and define a Crossplane ProviderConfiguration that references the secret.
The Crossplane provider expects a secret to be in the following format:
Then generate a Kubernetes secret using the file:
Apply the following ProviderConfig to the Crossplane Kubernetes cluster:
For more information about setting up AWS provider credentials, refer to the documentation.
The following resources should now exist in the Crossplane cluster:
Deploying infrastructure with Keel
The examples for creating Amazon SQS and Amazon S3 buckets can be found in the following git repository. In this example, we define an Amazon SQS queue and an Amazon S3 bucket to be created. Update your
.spinnaker/spinnaker.yaml file with the following. (Be sure to replace the placeholder value for the bucket’s name.)
Once the file is committed to your repository, Keel will pass the resource definition to Crossplane controllers and monitor its progress to ensure desired states are reached. Running the following commands should show the AWS resources being created:
Verify that resources are being created through the AWS CLI:
The Spinnaker UI will show these resources as being managed.
Creating a VPC via Compositions
When we installed the Crossplane Configuration package, it contained Compositions for CompositeNetwork that includes a VPC with subnets, route tables, and gateways, and a CompositeCluster that includes an Amazon EKS cluster with AWS Identity and Access Management (IAM) roles and a Node pool. This allows an infrastructure team to provide a custom resource that an end user can consume.
To deploy a CompositeNetwork in Keel, include the following in the
This resource creation can be verified with the following commands:
The Spinnaker UI will show it as a managed resource.
Get all of the resources and delete them:
Remove Crossplane providers and configuration packages:
Remove all Spinnaker microservices and its resources:
Conclusion and next steps
For companies looking to modernize their established CI/CD systems, the combination of Spinnaker’s Keel service with Upbound’s Crossplane enables the declarative provisioning of both infrastructure and AWS application resources. By architecting around the Kubernetes API, enterprises can adopt the same methodologies as large cloud providers use for their infrastructure.
This approach enables centralized infrastructure teams to publish safe, secure, and governed infrastructure APIs, making them available to developers for rapid use and self-service. Also, it is a natural evolution from infrastructure as code methodologies and tooling, such as Terraform. Enterprises operating at scale are testing the limitations of the architecture, and moving to a more cloud-native approach could be an evolution of their CI/CD tooling.
Spinnaker Keel and Crossplane are open source projects. We would love for you to join the community and start contributing. Join the Keel community on Slack and GitHub. Join the Crossplane community on Slack and Github.