Developing an AWS Service Catalog self-managed engine for governance

AWS Service Catalog lets you centrally manage your cloud resources to achieve governance at scale of your Infrastructure as Code (IaC) templates. AWS Service Catalog supports AWS CloudFormation natively and allows customers to use other IaC such as Terraform Community and Terraform Cloud via Service Catalog reference engine. We often hear customers asking how to modify the generic reference engine to work with other IaC tools such as Ansible, Chef, Pulumi, Puppet, and more. In this blog post, we will show you how to deploy your own AWS Service Catalog self-managed engine that is compatible with your IaC tool of choice.

The self-managed engine adheres to the workflow illustrated in the high-level diagram below, utilizing AWS Service Catalog, its core engine, and custom logic.

1. An external product from AWS Service Catalog triggers a request to the self-managed engine core.
2. The core engine parses input parameters and receives the product provisioning request from AWS Service Catalog to a dedicated Amazon SQS queue.
3. Upon receiving these messages, the core engine routes them to the custom logic of the self-managed engine.
4. The custom logic processes requests leveraging the Infrastructure as Code (IaC) of choice.
5. The custom logic then notifies AWS Service Catalog about the provisioning results.

Figure 1: High Level Self-Managed Engine for Service Catalog Architecture

Solution Overview

Now that we understand the high-level concept of self-managed engine, in this section we will look at how to develop self-managed engine. We can break the self-managed engine into two parts as described below:

1. Core – These are common resources that required for all self-managed engines. These are resources that the Service Catalog backend depend on and invoke when you provision a new product in Service Catalog.
2. Custom Logic – The resources for the custom logic will vary depending on your IaC of choice and how you want to implement the IaC workflow. At a high level, this portion of the solution is responsible for receiving the request from core engine queue, provisioning the resources according to the IaC of choice, and sending the provisioning results back to the Service Catalog.

Self-Managed Engine Core

The diagram of the self-managed engine core architecture below describes two tasks:

1. Parsing input parameters (variables) from your IaC template/configuration. This allows Service Catalog to parse and display the input variables, prompt the user to enter the input variables, and pass the input variables as part of the product provisioning request to the self-managed engine custom logic.
2. Publishing the product provisioning request from AWS Service Catalog as a message in a dedicated SQS queue. Depending on the operation being performed, the message will be routed to a Provision, Update or Terminate queue. The Custom Logic will be responsible to process these messages later on.

The SQS queue name must follow the prescribed convention to reflect the Service Catalog name requirements. The Dead Letter queue name should be ServiceCatalogExternalOperationsDLQ while the Provision queue is implied as ServiceCatalogExternalProvisionOperationQueue Similarly, the Update queue name should be ServiceCatalogExternalUpdateOperationQueue and for Terminate, it is ServiceCatalogExternalTerminateOperationQueue.

A Terraform version of self-managed engine core components is available in Terraform Service Catalog Engine – Core. This self-managed engine core follows the sample workflow shown in the diagram below, leveraging AWS Lambda, and Amazon SQS queues.

Figure 2: Self-Managed Engine for Service Catalog Core Architecture

Your IaC template/configuration may include input parameters for user input. This input parameter parsing is accomplished by a Lambda function which is purpose build for the type of IaC supported by the self-managed engine.

When a user starts the product provisioning request, Service Catalog invokes the Lambda function. The function name must follow the convention “ServiceCatalogExternalParameterParser”. The Lambda function must be given permission to be invoked by servicecatalog.amazonaws.com as the IAM principal. Specifics on the event schema used to invoke the function, as well as the expected response can be found in AWS documentation for External Engines for AWS Service Catalog.

Self-Managed Engine Custom Logic

The custom logic in the self-managed engine is responsible, at a minimum, for 4 tasks:

1. Process events placed into the Provision, Update, and Terminate SQS queues.
2. Perform resource provisioning and other operations in the IaC tool of choice.
3. Parse outputs and send them to Service Catalog.
4. Notify Service Catalog of success/failure of provisioning.

Most of the architecture choices to accomplish the tasks above are under the control of the developer. For example, an AWS CodeBuild job can be used to run your IaC tool – however, you may opt to do so differently, perhaps as an Amazon ECS. Likewise, you can opt to add additional tasks to the workflow, such as sending an Amazon SNS notification for failure. Ultimately, the final architecture will vary depending on the requirements of your implementation.

A functional self-managed engine for Terraform Community Edition can be found in Terraform Service Catalog Engine – Terraform Community Edition. This self-managed engine follows the sample workflow shown in the diagram below, leveraging AWS Lambda, AWS Step Functions, and AWS CodeBuild.

Figure 3: Self-Managed Engine for Service Catalog Custom Logic Architecture

The following workflow, depicted in the comprehensive diagram below, elaborates on both self-managed engine components:

1. The Service Catalog invokes a Lambda function to parse parameters within the supplied template.
2. The Service Catalog directs the request to the relevant SQS queue.
3. The Terraform Community Engine retrieves messages from the queues and subsequently forwards them to AWS Step Functions.
4. A Step Function Workflow manages the request, while Terraform operations execute within AWS CodeBuild.
5. The Terraform Engine transmits the provisioning outcome to the Service Catalog service.

Figure 4: Self-Managed Engine for Service Catalog Architecture

Provisioning an External Product Type

Now that we reviewed the self-managed engine, in this section we will review how the Service Catalog admin can create a package of IaC templates as a Service Catalog product. The Service Catalog Admin can create their own custom IaC configuration and package it as an AWS Service Catalog product by using External product type. The self-managed engine will be deployed in hub and spoke model. Service Catalog Admin can share the portfolio to the Service Catalog user to provision the product in one or more spoke accounts. AWS Service Catalog stores the provisioning artifact, but unlike CloudFormation templates, Service Catalog does not validate the External product type IaC template. As such, it’s important that the Service Catalog Admin perform test and validation separately. Service Catalog end users can call the following provisioning operations on the product:

1. DescribeProvisioningParameters
2. ProvisionProduct
3. UpdateProvisionedProduct
4. TerminateProvisionedProduct

In addition to the steps included in this blog, more information about creating and provisioning a Service Catalog product can be found here.

Note: Please be aware that an instance of the self-managed engine must be deployed in every region where third-party/external will be deployed. Furthermore, only a single self-managed engine is supported on a per-region, per-account basis.

Conclusion

In this blog post, we introduced how to develop a self-managed engine which supports Service Catalog external product types with your IaC tooling of choice. By combining external products in a Service Catalog hub and spoke model, you can achieve consistent, reusable, and compliant deployments across your organization, with your IaC tool of choice and achieve consistent governance. For more information on how to get started with Service Catalog, see the AWS Service Catalog Getting Started page.

About the Authors