AWS Cloud Operations Blog

Serverless Governance of Software Deployed with AWS Service Catalog

AWS Service Catalog (Service Catalog) is a powerful tool that empowers organizations to manage and govern approved services and resources. It significantly benefits platform engineering by standardizing environments, accelerating service delivery, and enhancing security. With its automated provisioning and resource management, Service Catalog supports infrastructure as code, enabling scalable, reliable deployments.

Platform engineering teams are responsible for maintaining the security and governance of software deployed within their organizations. These platform teams are challenged with meeting diverse customer requirements for deployment options and software updates. At the same time, they are responsible for security and governance of the software under their management.

Tools like Service Catalog can help facilitate software deployment, but platform teams still need tools that provide visibility into the versions and locations of their organization’s software. This is crucial, as the software may include applications created by the organization, as well as third-party solutions. Ensuring the secure deployment of software with properly scoped permissions using AWS Identity and Access Management (IAM) to grant least privilege is also critical.

In this blog, we’ll review a serverless solution that addresses the key requirements of security, visibility, and governance that platform teams need to safely manage software.

Overview of solution

The solution contains two components, let’s examine those, then break down the architecture in Figure 1 below.

Figure 1: The components distributed across two AWS Accounts

On the left side is the platform engineering account. It contains a Service Catalog portfolio and its extension, the Amazon DynamoDB visibility table. Products can be provisioned once the portfolio is shared with an organization account and AWS CloudFormation stacks are created.

Component One: Platform engineering products portfolio

The platform engineering products portfolio is published in an AWS account called “platform engineering” and contains products, either internal or 3rd party, that the organization deploys to its internal teams’ accounts. A product, in its essence, is a simple Infrastructure as Code (IaC) construct, like a CloudFormation template in this example, or an artifact of Terraform, Ansible, or any other IaC provider. Each product extends the organization’s governance or security practices. For example, one product can deploy a CloudFormation stack that creates a role for a CI/CD pipeline with just the bare minimum permissions required by that pipeline process, implementing the least privileged access practice. Another product could create an organization-approved AWS Web Application Firewall and rules that can be attached to any Amazon API Gateway or Amazon CloudFront, thus enabling the same security and governance mechanisms across the organization’s APIs.

The customers of these products, the organizations internal customers, developers and DevOps engineers, can deploy to AWS accounts across their organization with confidence, knowing that they are using a robust and reliable tool.

As we’ll demonstrate in our example below, Service Catalog can help facilitate software deployment, but platform teams still need tools that provide visibility into the versions and locations of their organization’s software. This is crucial because the software may include applications created by the organization, as well as third-party solutions.

Component Two: Portfolio’s extended visibility component

The second component, the visibility mechanism, extends the portfolio. Service Catalog portfolios provide a managed way to publish and share portfolios and products across the organization while supporting versions. However, once the internal customers deploy the products to their accounts, i.e., provision the products, Service Catalog does not offer native visibility as to which product was deployed to what account, who deployed it, and what version was deployed. The visibility mechanism extends Service Catalog and adds a new layer of visibility and information. For every provisioned product, we can store valuable metadata such as the internal team’s name that deployed it, the portfolio it belongs to (its id), the deployed account and region, and product version. This mechanism is based on serverless technology and, in its center, a DynamoDB table. We will discuss this component’s inner workings in the next section.

For platform teams, maintaining visibility into deployed products is important on multiple fronts. Gaining visibility means knowing what version of each product is deployed, and where. Older versions of products could have negative implications for security and compliance. Platform teams may also be tasked with staying within the licensing terms of third-party products as well, which again means understanding where, and what versions of our software, are deployed.

Increased visibility drives operational efficiency as well acting as a tool to help keep internal teams using the same version of software wherever possible, a best practice that allows us to operate our internal products like SaaS solutions. Most importantly, the platform team, the security team, and the compliance team, all will find their experiences improved by having better governance over the applications they manage.

Solution High Level Design

The solution is divided into two domains represented by two CDK constructs:

  1. Platform engineering products portfolio
  2. Portfolio’s extended visibility

The first construct defines the Service Catalog portfolio and its products.

Let’s focus on the second construct, which extends the capabilities of the first.

Visibility Mechanism Low Level Design

The visibility construct defines an Amazon SNS topic, an Amazon SQS queue that subscribes to the topic, a DynamoDB table that will store product metadata and usage, and a CloudFormation custom resource. Custom resources will be created for each of the portfolio’s products. It is the connecting resource between the portfolio’s products and the visibility mechanism. During creation, the custom resource will send the product’s metadata to the SNS topic, thus providing visibility and information to the platform engineering team. The custom resource supports product update and delete actions, giving complete insights into the products’ lifecycle of the organization account.

Provisioning a Product Flow

Figure 2 below describes the process of an admin user provisioning products from the platform engineering portfolio in an organization account. It demonstrates both parts but focuses on the visibility mechanism.

Please note that before a user can provision a product, the following steps are mandatory:

  1. Share portfolio with specific account or organization.
  2. Import portfolio
  3. Set portfolio access to specific users.

Figure 2: The admin user provisioning flow

Let’s review the diagram steps:

Step 1: The “Admin” initiates the provisioning of a product through Service Catalog and the platform engineering portfolio.

Step 2: Service Catalog triggers the creation of a CloudFormation stack for the provisioned product. For this example, the admin chose the CI/CD product.

Step 3: The CI/CD pipeline role is created.

Step 4: The stack includes a custom resource that is created during the stack creation process.

Step 5: The custom resource publishes a message containing the provisioned product metadata and parameters (consumer name) to the platform engineering’s Amazon SNS topic and enters a wait state.

Step 6: The SNS topic forwards the message to an SQS queue in the Platform Engineering Service Catalog Account.

Step 7: An AWS Lambda function is triggered to handle the message from the SQS queue.

Step 8: The Lambda function adds information about the provisioned product, including its version, region, and user details, to the visibility DynamoDB table for enhanced portfolio visibility.

Step 9: The stack is released from the wait state, and the deployment process is completed.

We have automated the visibility process by utilizing the CloudFormation ability to send SNS messages with custom parameters during the lifecycle of the provisioned products. During product update or deletion, a similar process triggers which results in the visibility table updating the product entry or deleting it.

More detailed information

Custom resource and consumer name parameter construct: https://github.com/ran-isenberg/governance-sample-aws-service-catalog/blob/main/cdk/catalog/products/governance_construct.py

Usage. i.e. CI/CD product creation: https://github.com/ran-isenberg/governance-sample-aws-service-catalog/blob/main/cdk/catalog/products/cicd_product.py

Walkthrough

Prerequisites

To deploy the solution, you will need the following prerequisites:

  1. AWS account with necessary permissions.
  2. AWS CLI installed and configured.
  3. Node.js,npm and npx installed.
  4. AWS CDK 2.141.0 and higher (Cloud Development Kit) installed.
  5. AWS account after running the CDK bootstrap command in the designated region
  6. Python 3.12 installed
  7. Poetry installed
  8. Docker installed

Initial setup

Clone the code from GitHub onto a local machine:

Git clone git@github.com:ran-isenberg/governance-sample-aws-service-catalog.git

To install the packages, utilize a virtual environment:

poetry config --local virtualenvs.in-project true
poetry shell
poetry install

To prepare the services for deployment, execute the following commands to create the Lambda function layer requirements.txt file and export the function code to the .build folder:

mkdir -p .build/lambdas ; cp -r catalog_backend .build/lambdas
mkdir -p .build/common_layer ; poetry export --without=dev --format=requirements.txt > .build/common_layer/requirements.txt

If you are running linux/mac you can run ‘make build’ instead

To deploy the services with AWS CDK, run the following command:

npx cdk deploy --app="${PYTHON} ${PWD}/app.py" --require-approval=never

After deployment, you should see a similar entry to Figure 3 below in the Service Catalog ‘local portfolios’ section under the Administration category.

Figure 3: Service Catalog Portfolios

Account structure

In production portfolio scenarios, products will generally be distributed across several accounts. For demonstration, we can use a single account to deploy a portfolio and some products.

Share portfolio at the organization level or specific accounts.

Then you set access permissions by clicking on ‘Grant Access’.

Figure 4: Granting permission to the portfolio

Set permissions to allow your admin to provision it.

The video Share Portfolios Across Accounts in AWS Service Catalog demonstrates these actions:

You define which users and roles will be able to provision

Figure 5: Adding users and roles

Provisioning a Product

Let’s assume that team A’s DevOps member wants to deploy CI/CD pipeline role product.

We’ll enter the product name ‘Team_A_CICD’ and under ‘consumerName’ parameter, ‘Team A’ and click on ‘Launch Product’.

Figure 6: Launching a Service Catalog product

Deployment of a new CloudFormation stack will begin.

All stack deployments start with a prefix ‘SC’. Here’s the deployed stack below.

Figure 7: CloudFormation stack resulting from product launch

The stack contains the CI/CD role that we wanted to deploy and in addition, the platform engineering custom role that extends the catalog’s visibility.

Let’s view the visibility table back at the platform’s engineering account.

We’ll open DynamoDB in the console and view the visibility table.

Figure 8: Entry in DynamoDB displaying product details

As expected, a new entry was added upon deployment completion. The entry holds the following information:

  • portfolio id
  • provisioned product stack ARN
  • account id that deployed sqs
  • product (Team’s A account number)
  • product name
  • time of deployment
  • region
  • version

Cleaning up

To delete the service stack, run the following commands:

Poetry shell
npx cdk destroy --app="${PYTHON} ${PWD}/app.py" --force

For those who use mac or linux, you can use ‘make destroy’

Conclusion

In this blog, we’ve explored a serverless governance solution for platform engineering teams that enhances Service Catalog and its product provisioning capabilities. The solution addresses the platform engineering team’s critical security, visibility, and governance needs.

We’ve explored how a Service Catalog product portfolio and an extended visibility mechanism improve governance of applications and can help drive best practices for consumers and producers of the software they manage. By leveraging serverless services, the solution offers automated provisioning, consistent deployment practices, and enhanced monitoring across the organization. Make sure to check out the Governance Sample AWS Service Catalog repo for more details.

About the authors:

Ran Isenberg

Ran Isenberg is an AWS Serverless Hero, a Principal Software Architect at CyberArk, a blogger, and a public speaker. He maintains a blog (RanTheBuilder.cloud) where he shares knowledge and experience in the Serverless world.

Bill Tarr

Bill Tarr is a Principal Partner Solutions Architect at the AWS SaaS Factory, enabling SaaS Builders on their software-as-a-service (SaaS) journey. Bill evangelizes the best practices for designing, building, and operating SaaS solution through direct engagement, speaking at events, and live streamed shows like Building SaaS on AWS on Twitch.