Conformitron: Validate third-party software with Amazon EKS and Amazon EKS Anywhere

Introduction

With the proliferation of open-source and commercial container software products, customers want to know upfront which products are compatible with the container orchestrator of their choice. Customers expect compatibility testing to be continuous and include all the available dimensions, such as Amazon Elastic Kubernetes Service (Amazon EKS) / Amazon EKS Anywhere (EKS-A) versions, OS, hardware and virtualization platforms. Customers also ask for guidelines and best practices to provision and maintain production-ready clusters with partner solutions in place, addressing security, ingress, storage, and software distribution across various edge locations, ranging from a few traditional on-premise data centers to thousands of sites running EKS-A on Snowball Edge.

To address the customer and partner needs, AWS has built a solution called Conformitron, which provides an expandable and extensible framework to run conformance testing and general quality assurance on different Amazon EKS deployment models. Today, it supports EKS-A on VMware vSphere, EKS-A on Bare Metal, EKS-A on Snow, EKS-A on Nutanix, and Amazon EKS on AWS Outposts. This solution doesn’t have dependencies on other AWS services and can be used for on-premises, edge, and cloud deployments. Note that Conformitron is separate from previous announcements we’ve made for running partner software on Amazon EKS and Amazon EKS Anywhere. If you’re interested in onboarding your software to AWS Marketplace, then visit the Getting started as a seller and Container-based products in the AWS documentation.

Conformitron provides a consistent automated approach to run Kubernetes evaluator testing, deployment of containerized third-party products, and validation on Kubernetes environments to help partners and customers validate their hardware and software solutions deployed on a variety of Amazon EKS environments including bare metal, VMware, Snowball Edge, Nutanix and other environments.

The solution applies GitOps mechanisms for continuous quality control of Kubernetes workloads and self-service on-boarding using Git as the source of truth across all environments under test. Git-driven workflows also cover self-service on-boarding and governance.

AWS Partners and third-party software vendors can submit their solutions as pull requests to the public GitHub repository of validated solutions and rely on Conformitron to validate it across a subset or all of the supported environments, which are maintained by AWS. This represents a significant benefit to the software vendors as it eliminates the cost of the lab environment maintenance, which otherwise maybe substantial, especially for EKS-A and AWS Outposts.

In addition to the conformance and quality assurance, Conformitron provides customers with an excellent software distribution mechanism-based on GitOps. Now customers can reuse the repository of validated solutions and deploy/manage them consistently across any range of target environments, which may be in hundreds and even thousands (e.g., EKS-A on Snow).

For Independent Hardware Vendors (IHV) and Industry Vertical Segment Customers such as Telecommunications (TelCo), the framework provides an option to run the core evaluator framework on the target Kubernetes environments and ensures if Independent Software Vendor (ISV) products and stateful workloads can be run on their environment completely via automation and GitOps.

Solution overview

Conformitron solution orchestrator

Conformitron solution orchestrator is an internal repository maintained by AWS, which holds the tools configuration, cluster configuration, GitOps bootstrapping, orchestration of tests, and reporting. This repository is internal and is expected to be maintained by AWS; however, it may be made public in the future if we observe a need for partners to contribute. The framework deploys a list of prerequisite tools on the administrative machine that interacts with EKS-A cluster. The framework validates if the required list of tools are installed on the administrative machine that interacts with target environment.

Sonobuoy

Next, the framework runs a Kubernetes conformance validator tool such as sonobuoy to make sure the EKS-A distribution on a particular hardware conforms to the official Kubernetes specifications. Sonobuoy is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of plugins (including Kubernetes conformance tests) in an accessible and non-destructive manner. It is a customizable, extendable, and cluster-agnostic way to generate clear, informative reports about your cluster. The Framework checks if sonobuoy is run already before proceeding any further process.

GitOps via Flux

Conformitron uses GitOps with Flux for installation and validation process of ISV product across Amazon EKS-deployment models. GitOps is a way of managing application and infrastructure deployments declaratively through a Git repository. It is an operational model that offers you the ability to manage the state of multiple Kubernetes clusters leveraging the best practices of version control, immutable artifacts, and automation. Flux is a declarative, GitOps-based continuous delivery tool that can be integrated into any continuous integration (CI) / continuous delivery (CD) pipeline. The ability to manage deployments to multiple remote Kubernetes clusters from a central management cluster, support for progressive delivery, and multi-tenancy are some of the notable features of Flux. Installation of Flux isn’t required on deployment models, such as vSphere, that support GitOps enabled EKS-A clusters out of the box today if configured.

Reporting

Conformitron supports different types of reports readily available to be shared with partners and customers on demand for different deployment models of Amazon EKS, such as EKS-A on Snow, vSphere, Bare Metal, Nutanix, and EKS on AWS Outposts. Sonobuoy reporting is first of the reports produced by the Conformitronwhich is exported to an Amazon Simple Storage Service (Amazon S3) bucket on a secure and internal AWS account. Secondly, Conformitron creates a nighty report on current state of all Kubernetes resources such as deployments, pods, jobs in a particular deployment model and the report is exported to an Amazon S3 bucket in same internal AWS account. Partners can check the status of their installation and validation process with this reporting containing state of the Kubernetes resources.

Conformitron for hardware vendors

Conformitron has a mechanism that allows hardware partners and providers to certify their hardware for services like EKS-A on vSphere and bare metal. The scope of this category also covers partners certifying their own custom operating systems like RancherOS. For an EKS-A cluster on a particular hardware with custom-built node operating system, Conformance Framework first runs sonobuoy to make sure the EKS-A distribution on a particular hardware conforms to the official Kubernetes specifications and report of the run can be shared with hardware vendor. The vendor can take necessary actions based on the errors reported on the report. ISV products can be installed and validated seamlessly by the Conformance Framework using GitOps with Flux. This ensures the EKS-A cluster on a hardware from a vendor is conformant with Kubernetes specifications and allows onboarding third-party solutions, which includes validation of storage layer and other components.

Conformitron for Industry solutions

Conformitron has a mechanism to enable partners specializing in Industry Vertical segments (e.g., TelCo) to validate their software on Amazon EKS and EKS-A environments. For an EKS-A cluster with a particular Industry vertical segment product installed, Conformitron first runs sonobuoy to make sure the EKS-A distribution with an industry segment product is conformant to the official Kubernetes specifications and report of the run can be shared with Industry Vertical segment vendor. The vendor can take the necessary actions based on the errors reported on the report. ISV products can be installed and validated seamlessly by the Conformitron framework using GitOps with Flux. This ensures the EKS-A cluster with an industry segment product conforms to Kubernetes specifications and has ability to install ISV software, which also includes validation of the storage layer and other components.

Walkthrough

ISV and third-party solutions

Conformitron provides an opinionated, close to production-ready cluster configuration with secrets management, observability, and a public open-source repository of validated software solutions. This repository also offers the GitOps software distribution mechanism for solutions (including partner products) and a mechanism for partner on-boarding and automated testing.

As mentioned previously, the GitHub repository uses Flux constructs for software deployment, covering provisioning, and upgrades. The repository structure is based on the environments, which are represented by the top level directories:

+ eks-anywhere-baremetal
+ eks-anywhere-common
+ eks-anywhere-nutanix
+ eks-anywhere-snow
+ eks-anywhere-vsphere
+ eks-local-clusters-outposts

In this repository structure, the postfix on each folder (e.g., -baremetal, -vsphere, and -snow) represents the target deployment environment.

Partners and software vendors have an option to submit their solutions to a single location and deploy across all environments with the same configuration. Such common solutions require a new solution-specific folder (e.g., <orgname> or <orgname>/<productname>) under the common folder eks-anywhere-common/Addons/Partner containing the GitOps descriptors (e.g., HelmRelease, manifests and/or other supported package management resources) in that folder.

In some cases, a particular solution or its configuration is specific to the target environment. For example, an edge cluster configuration of a software product (e.g., Snowball) may not exhibit all the enterprise features that are common for a full production cluster installation in the cloud or on premises. Such solutions are grouped under the new solution directory under the respective target. For example, a solution targeting EKS-A on Snow uses the path eks-anywhere-snow/Addons/Partner.

Each solution must be deployed in its respective namespace. This is done to isolate resources fairly and for security reasons. That means that each solution should contain a namespace manifest, which must be unique. We recommend using the <vendor>-<solutionname> naming convention.

For reporting purposes, it is important to have a clear and automatic mechanism that links the solution to the vendor. That is why the framework requires solution owners to add labels to their namespaces to indicate vendor and solution name as shown below:

apiVersion: v1
kind: Namespace
metadata:
    name: kubecost
    labels:
        aws.conformance.vendor: kubecost
        aws.conformance.vendor-solution: cost-analyzer

For products that use Helm as the package manager, deployments via FluxCD can use the HelmRelease custom resource as shown in this example. In particular, the example covers specification of Helm repository and Helm release.

Secrets management, such as license key or credentials, is implemented using the External Secrets add-on. Partners and software vendors are expected to share such secrets with the AWS Partner team ahead of the validation. The AWS Partner team stores such secrets in an AWS account and use External Secrets to bring them down to the target deployment cluster. After that, the secrets can be configured in the respective GitOps deployment folder and passed to the deployment using configuration values or used directly by the deployment if the deployment supports pre-created Kubernetes secrets. The example folder also contains an example of a secret with the deployment as well as an example of wiring that secret in the product deployment.

While deployment and validation of your solution on the target environment is helpful, it doesn’t provide the required level of quality assurance for functional verification, which is normally achieved with a test framework and automation included in the CI/CD cycle of the Partner product. We recommend that Partners wrap their functional test as a container and submit as a Kubernetes job along with their deployments to enable broader test coverage and better customer experience. The functional test job should be submitted under eks-anywhere-common/testers (runs on all platforms) or under your respective environment folder such as eks-anywhere-snow/testers (e.g., eks-anywhere-snow/testers/<orgname>/<productname>).

Partners can refer to the section on Local Testing (Linux/MacOS) in our eks-anywhere-addons GitHub repository for further information on local testing and validation before submitting a pull request.

Reporting and notification

Reporting

Hardware vendors and Industry Vertical solution vendors running Conformitron on their EKS-A cluster can configure the framework to send Sonobuoy reporting and daily night job reporting to their dedicated AWS accounts.

Apart from Sonobuoy reporting and daily night job reporting covered in the Conformitron solution orchestrator section, Conformitron produces an automation report that’s exported to eks-anywhere-addons containing the certified ISV partners across different deployment models of Amazon EKS such as EKS-A on Snow, vSphere, Bare Metal, Nutanix, and Amazon EKS on AWS Outposts. Please check on the Validated_Partners folder on the eks-anywhere-addons at any time to know the list certified ISV partners across different deployment models of Amazon EKS.

Alerting

Conformitron supports alerting via sending events happening from a Kubernetes cluster to a Slack channel using BotKube, a messaging bot for monitoring and debugging Kubernetes clusters. BotKube also allows you to run kubectl commands like kubectl get pods right from Slack. A dedicated Slack channel is configured for different deployment models of Amazon EKS Anywhere and Partners can be provided interim access to these Slack channels upon request to validate the alerts from EKS-A clusters. Hardware vendors and Industry Vertical solution vendors running Conformitron can configure the framework to send alerts to their own Slack channels on their Slack workspace. The following figure is a snapshot of an alert from an EKS-A cluster:

Collaboration

Please join us on slack at AWS Developers. Get onboarded to slack by sharing your email to our email box aws-containers-conformance@amazon.com.

Contribution flow

The Open Source Software (OSS) portion of the contribution is defined by the standard mechanisms based on forking. A contributor (AWS Partner or third-party solution vendor) will follow these steps:

1. Plan the target deployment using GitOps and identify any configuration that must exist prior to the solution deployment, such secrets (credentials, license keys, Application programming interface (API) keys, etc.) as well as ambient information needed for the solution to work (cluster name, environment, ingress points, and storage).
2. Initiate a discussion with the AWS team by creating an issue or a discussion in the public GitHub repository. Please keep in mind that the information discussed in the issues or discussions is public, so please limit this only to the publicly allowed information. We’ll provide a way to share credentials securely with the AWS team.
3. Once you receive a notification that your shared configuration is applied, create your deployment artifacts (e.g., Helm release or deployment manifests) and/or any documentation that you intend to submit in your respective fork of the solution public repository.
4. You can test the changes locally using FluxCD. See Local Testing (Linux/MacOS) for details.
5. Submit a pull request with your deployment and functional test job to the main branch of the public repository.
6. You can use a combination of Slack channel notifications and Reporting to get the status of the validation. Please see the above Reporting and notification section on the in-depth coverage of the communication mechanisms. The pull request itself can be used by the AWS team to communicate any feedback directly to the software vendor as well.

The above process outlines the steps for the initial contribution; however, the activity as a whole shouldn’t be viewed as a point in time validation. AWS uses these artifacts to continuously validate the contributed solutions in the new environments, including upgrading existing environments (e.g., upgrading version of EKS-A).

If any solution starts failing in the continuous validation, then an issue is created against it and assigned to one of the partners and/or software vendor contributors.

In addition, contributors are expected to maintain the release of their product, such as patching and upgrading their solutions to the latest versions. The contribution process outlined previously can be used for the changes in the configuration and/or version of the deployed solution.

Conclusion

In this post, we closely examined the Conformitron solution – an expandable and extensible framework to run evaluator testing on different Amazon EKS deployment models (e.g., EKS-A on VMware [VMC], EKS-A on Bare Metal, EKS-A on Snow, EKS-A on Nutanix, and Amazon EKS on Rover [AWS Outposts]). The Conformitron solution helped customers and partners with a consistent automated approach to run Kubernetes evaluator testing, ISV solution deployment, and validation on Kubernetes environments to help partners and customers validate their hardware and software solutions deployed on variety of EKS-A environments.

For ISVs, Conformitron evaluates software products across different Kubernetes deployment models supported by the service provider (i.e., AWS in this case). For IHVs and Segment Customers, Conformitron helped them to run the core evaluator framework on their Kubernetes environment and ensured if minimal ISV products and stateful workloads were run on their environment completely via Automation and GitOps.

We are looking forward to collaborate with AWS ISV Partners using the contribution flow section.

For IHV Partners and Industry Vertical segment customers, please work with AWS to validate the conformance of Amazon EKS environment along with ability to install ISV software, which also includes validations on storage drivers and components on different Amazon EKS deployment models (e.g., EKS-A on VMware [VMC], EKS-A on Bare Metal, EKS-A on Snow, EKS-A on Nutanix, and Amazon EKS on Rover [AWS Outposts]).

Finally, stay tuned for Conformitron for Amazon EKS in the cloud, which provides customers the confidence to run, deploy, and operate third-party solutions for multi-clusters, secrets, networking, security, and storage management of Kubernetes workloads on AWS with the catalog of solutions for Amazon EKS.