Enhancing an Internal Developer Platform (IDP) with Crossplane on EKS at SIXT

SIXT is a leading global mobility provider serving customers in over 100 countries worldwide. The company is committed to creating an exciting customer journey by delivering premium service and products. To deliver on its customer promise, SIXT is empowering its software developers to provide new and update existing features quickly. They modernized their applications as containerized microservices running on Kubernetes and invested in a self-service continuous integration and continuous deployment (CI/CD) pipeline which provides developers with control over the create, update, and delete (CRUD) lifecycle of their microservices. They can create new services and run them in production in a matter of minutes, deploy multiple times a day, and decommission services as needed.

“My vision is to provide developers with an equivalent level of control over AWS resources by extending the current Internal Developer Platform (IDP) capabilities” said Boyan Dimitrov, CTO at SIXT.

In this blog post, learn how with AWS’ guidance the company further improved developer velocity by enhancing their IDP using open source tools on Amazon Elastic Kubernetes Service (Amazon EKS) and the key strategy differentiators they used to successfully launch it in production.

Initially, the CI/CD pipeline only managed the applications lifecycle, not the infrastructure dependencies. The dependencies such as a database, a queue, an object store, etc., along with related Identity and Access Management (IAM) permissions were managed by manual processes using a ticketing system. As the software development organization grew, provisioning application dependencies degraded overall developer experience and feature velocity. The AWS resource request tickets were sitting in queues for days, which meant that developers were blocked and could not iterate on their applications. Platform engineers would often get interrupted to resolve routine, repetitive, and low-complexity tasks, distracting them from meaningful projects.

The platform engineering team had tried a couple of different approaches in the past. First, they attempted to build in-house bespoke automation. This initiative was dropped as they quickly realized its complexity, time and effort required to build and maintain it, and the solution’s lack of scalability. Next, the team tried to expose Infrastructure as Code (IaC) automation tooling to developers. This was also challenging due to the tooling and infrastructure automation complexity and the steep learning curve. Creating a single Amazon Simple Storage Service (Amazon S3) bucket required a developer to understand IaC configuration, follow the organization’s compliance requirements, and learn how to configure the necessary access permissions.

SIXT decided to evaluate Crossplane as a potential solution for provisioning AWS resources which balanced the needs of both developers and platform engineers. Crossplane is an open source Cloud Native Computing Foundation (CNCF) project that extends the capabilities of Kubernetes to include infrastructure resources management. It provides a simplified interface for developers to request infrastructure resources via Kubernetes manifests called claims. As shown in this diagram, claims are the only namespace-scoped Crossplane resources, serving as the developer interface and abstracting away implementation details. When a claim is deployed to the cluster, it creates a Composite Resource (XR), a Kubernetes custom resource representing one or more cloud resources defined through templates called Compositions. The Composite Resource creates one or more Managed Resources which interact with the AWS API to request the creation of the desired infrastructure resources. To learn more, review some of our related blog posts Introducing AWS Blueprints for Crossplane and Disaster Recovery When Using Crossplane for Infrastructure Provisioning on AWS.

claim architecture drawing Expressing infrastructure as Kubernetes manifests unlocks the ability to leverage the rich Kubernetes ecosystem. The claim can be deployed with the same CI/CD pipeline used for applications, access can be restricted with already set Role-Based Access Control (RBAC) permissions, and fine-grained rules and policies can be enforced with Open Policy Agent (OPA) Gatekeeper.

When drafting the implementation plan for Crossplane, organizations tend to add every possible use case in the initial implementation. As the scope increases, the timeline increases, and so does the risk. The larger the time frame, the more likely it is for business priorities to change and the project to be put on hold indefinitely. Instead, SIXT took an iterative approach. They decided to implement Crossplane by starting with a concise use case: provisioning an Amazon S3 bucket using Crossplane such that the application running on the Kubernetes cluster has IAM access to the bucket. They also involved the development team early in the process to create a tight feedback loop to ensure successful adoption.

Solution

We will explore the solution from the development, platform, and compliance teams’ perspectives.

Developer Interface

The application is bundled into a Helm chart, a package for deploying applications to Kubernetes. The Helm chart includes conditional logic for generating a Crossplane claim manifest. Developers can control Helm parameters using a Helm values file. To create a new Amazon S3 bucket developers can set `s3irsa.enable` to `true` in the `values.yaml`:

s3irsa:
enabled: true

The Helm chart contains the deployment and the Crossplane claim which can be deployed to the Kubernetes cluster using any pipeline capable of deploying Helm charts. This example uses ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes. Once the claim is deployed to the cluster, it creates an Amazon S3 bucket and other AWS and Kubernetes resources necessary for the application to access the bucket depicted in the diagram.

Helm architecture drawing

The developer interface abstracts away the complexities of creating the Amazon S3 bucket and setting up the required permissions for the application to access it. There is no additional cognitive load for developers because it integrates seamlessly with already existing and familiar tools such as Helm and ArgoCD.

Platform Team Interface

The Platform team is responsible for designing the Composite Resources (XRs) by creating combinations of Composite Resource Definitions (XRDs) and Compositions. An XRD defines the schema of the XR and what parameters can be passed in. The Composition defines the behavior of the XR and what Managed Resources the XR would create. In addition to Managed Resources, XRs can create other XRs called nested XRs. In this example, the XRs are organized in two tiers: base tier and top tier. The base tier XRs are designed to be reused by different top level ones to prevent code repetition. The top tier `S3IRSA` XR contains three base tier nested XRs:

`ObjectStorage` creates an encrypted Amazon S3 bucket with public access blocked and injects the bucket name into a Kubernetes ConfigMap, ready for the application to consume.
`IRSA` creates the IAM Role and Kubernetes Service Account annotated with the IAM Role ARN.
`IAMPolicy` creates the Policy and RolePolicyAttachment.

EKS resource diagram

Dividing the code into smaller components makes it easy to read, understand, and modify. Each of the components is tailored to contain business specific requirements. For example, the `ObjectStorage` creates an Amazon S3 bucket that is encrypted and public access to it is blocked. These settings are not exposed to developers via the claim. If someone tries to change them via the AWS console or an API call, the Kubernetes reconciliation process will put them back into their desired state. The next section explains how to use a policy engine to further refine these guardrails.

You can follow along with this example in the Crossplane on EKS repository.

Compliance Team Interface

They extended Open Policy Agent (OPA) OPA/Gatekeeper capabilities to include governance of AWS infrastructure provisioning by integrating it with Crossplane. By integrating these tools into their workflow, SIXT enforced company-wide policies for provisioning. Gatekeeper ensures that all infrastructure deployments adhere to predefined policies, preventing any non-compliant resources from being provisioned, by enforcing policies during the provisioning process. By employing these policies, SIXT ensures adherence to their standards, and minimizes the risk of misconfigurations.

SIXT implemented the following set of OPA policies:

Enforcement of their internal naming conventions for buckets and IAM resources
Validation that claims can only be created within namespaces labeled with the specific criteria (e.g., `crossplane: allowXRD`)
Ensuring that claims can create XRs that can only select compositions owned by the platform team
Policy that prevents creating more than one managed resource owning the same Amazon S3 bucket

The following OPA rule, expressed in the rego policy language, verifies that the requested bucket name is not already managed by an existing Amazon S3 managed resource. This avoids overwriting each other if they make modifications or delete the the bucket:


        # Helper function to compare resource
        sameBucketMR(obj, review) {
            obj.metadata.name == review.object.metadata.name
            obj.kind == review.object.kind
        }


        violation[{"msg": msg}] {
            review := input.review
            # Checks the input is kind Bucket
            review.object.kind == "Bucket"
            obj := data.inventory.cluster["s3.aws.crossplane.io/v1beta1"].Bucket[_]
            # Use helper function to check if it’s the same object
            not sameBucketMR(obj, review)
            # Gets the bucket name to compare
            newBucket := review.object.metadata.annotations["crossplane.io/external-name"]
            existingBucket := obj.metadata.annotations["crossplane.io/external-name"]
            # Checks if both have the same bucket name
            newBucket == existingBucket
            # If checks get this far there is violation to the rule
            msg := sprintf(
              "MR %v requesting Bucket %v is already managed by Bucket MR %v",
              [review.object.metadata.name,newBucket,obj.metadata.name],
            )
        }

A set of OPA policy examples, including the one mentioned, can be found at the Crossplane on EKS repository.

Conclusion

In this post we discussed how SIXT enhanced their internal developer platform (IDP) and successfully launched the new platform capability in production. They implemented open source Crossplane and integrated it with existing tooling such as Helm, ArgoCD, and Gatekeeper to provision AWS resources for business applications. The new IDP functionality improved developer velocity by extending self-service capabilities to include control over the full lifecycle of application dependencies.

The two key strategic differentiators that ensured Crossplane’s successful production launch were:

Choosing a Scoped Use Case for Initial Implementation:
Instead of attempting to build all features in one go, it’s often more effective to start with a focused and scoped use case. A scoped use case typically targets a specific and manageable aspect of the IDP that addresses a pressing need or provides immediate value.
Involving Developers Early in the Design Process:
By involving developers early in the design process, the platform team ensures that the developer’s perspective and needs are taken into account from the very beginning. Early involvement also allows developers to provide valuable feedback, identify potential issues, and suggest improvements based on their expertise.

With this newfound success, SIXT plans to expand their new internal developer platform with additional use cases to support other business applications that leverage AWS services such as databases (Amazon Relational Database Service, Amazon RDS) and serverless compute (AWS Lambda).