Use Python to manage third-party resources in AWS CloudFormation

In this post, I demonstrate how to use Python and the AWS CloudFormation registry to manage third-party resources, which are common components of modern cloud architectures. The benefits of using CloudFormation to manage third-party resources include reducing deployment complexity and enabling the inherent benefits of CloudFormation, such as rollback if a failure occurs.

Using the CloudFormation Command Line Interface (CLI), I create a CloudFormation resource provider, which enables you to manage GitHub repositories and AWS resources from a single template.

Solution overview

The provided example is modeled after a GitHub repository, which enables you to manage repositories as part of an AWS CloudFormation template. This post is divided into the following steps:

1. Set up development environment
2. Initialize provider
3. Add handler code
4. Declare dependencies
5. Submit resource to AWS CloudFormation
6. Test resource

Prerequisites

This post assumes that you’re familiar with AWS CloudFormation template, Python, and GitHub. For this walkthrough, you must have the following:

• An AWS account
• A GitHub account

1. Set up development environment

Get your development environment running by completing the following steps:

a. Install Python 3.6 or later by either downloading Python or using your operating system’s package manager.

b. Use the following command to install both the AWS CloudFormation CLI and Python language plugin:

pip3 install cloudformation-cli cloudformation-cli-python-plugin

2. Initialize provider

a. The CloudFormation CLI provides a command to bootstrap a resource provider. Use the following commands to create an empty folder and initialize a new provider:

mkdir demo-github-repository
cd demo-github-repository
cfn init

c. The CloudFormation CLI prompts you for the name of the resource type, which maps to the Type attribute for resources in an AWS CloudFormation template. For this example, use Demo::GitHub::Repository, and choose python37 for the language.

d. Choose whether to use Docker to package Python dependencies. This is useful for resources that have binary dependencies that use a specific platform or architecture. For this resource, all of the dependencies are Python based, so you can choose to disable Docker builds.

You have now initialized your project.

Resource schema

AWS CloudFormation resource providers use JSON to declare a schema. The schema primarily declares which properties the resource accepts and which outputs it provides to template authors (via !GetAtt). For more information, see Resource type schema.

For modeling GitHub repositories, use any text editor to open demo-github-repository.json, and replace its contents with the following JSON code:

{
  "typeName": "Demo::GitHub::Repository",
  "description": "Demo of the python plugin that models a GitHub repository",
  "sourceUrl": "https://github.com/aws-cloudformation/aws-cloudformation-rpdk.git",
  "properties": {
    "AccessToken": {
      "description": "GitHub Access Token",
      "type": "string",
      "pattern": "^[A-Za-z0-9_]{40,259}$"
    },
    "Name": {
      "description": "Name of the GitHub Repository",
      "type": "string",
      "pattern": "^[0-9a-zA-Z-]*$"
    },
    "Org": {
      "description": "Name of the GitHub Organisation, if not provided the repo will be created in user namespace",
      "type": "string",
      "pattern": "^[0-9a-zA-Z]*$"
    },
    "Visibility": {
      "description": "Visibility, can be public or private",
      "type": "string",
      "enum": [ "private", "public" ]
    },
    "SshUrl": { "description": "SSH git URL", "type": "string" },
    "HttpsUrl": { "description": "HTTPS git URL", "type": "string" },
    "Namespace": {
      "description": "Namespace of repo (org/name)",
      "type": "string",
      "pattern": "^[0-9a-zA-Z]*/[0-9a-zA-Z]$"
    },
    "Id": { "description": "GitHub repo ID", "type": "integer" }
  },
  "additionalProperties": false,
  "required": [ "AccessToken", "Name" ],
  "readOnlyProperties": [
    "/properties/SshUrl",
    "/properties/HttpsUrl",
    "/properties/Namespace",
    "/properties/Id"
  ],
  "writeOnlyProperties": [ "/properties/AccessToken" ],
  "createOnlyProperties": [ "/properties/Org", "/properties/AccessToken" ],
  "primaryIdentifier": [ "/properties/Id", "/properties/AccessToken" ],
  "handlers": {
    "create": { "permissions": ["github:CreateRepo"] },
    "read": { "permissions": ["github:ReadRepo"] },
    "update": { "permissions": ["github:UpdateRepo"] },
    "delete": { "permissions": ["github:DeleteRepo"] },
    "list": { "permissions": ["github:ListRepo"] }
  }
}

The properties that are not included in the readOnlyProperties section are available when declaring the resource in an AWS CloudFormation template.

AccessToken: Token used to authenticate the GitHub API. Note that AccessToken is included in the primaryIdentifier section of the schema. This ensures that the token is passed to read and list handlers.

Name: Repository name.

Org: GitHub organization where the repository is created. If this is unspecified, the repository is created in your account and not in an org. This property is defined in the createOnlyProperties section, which means it cannot be changed using the GitHub API. If a stack update changes this value, the AWS CloudFormation API attempts to create a repository in org, which replaces the previous repository. For more information, see Replacement.

Visibility: Defines whether the repository is private or public. By default, the repository is set to private. Note that this behavior is defined in the handler source code in the following section.

Properties that are marked readOnly cannot be defined by the user, but the properties are available using !GetAtt in the template. This resource defines the following read-only properties:

HttpsUrl/SshUrl: URLs that can be used to clone the repository.

Namespace: Full name of the repository, which joins org with the repository name, separated by a forward slash.

Id: When a repository is created, this ID is returned by the GitHub API. It is used as the PrimaryIdentifier for the resource, which can be retrieved by using the !Ref function in the CloudFormation template.

Each time the schema updates, the project must regenerate to ensure that the CloudFormation CLI project code is in sync with the schema. To do this, run the following command from within the folder you used to initialize the project:

cfn generate

3. Add handler code

Handler code manages CREATE, UPDATE, DELETE, and READ operations on the stack that contains the resource type. To simplify this, the CloudFormation CLI generates example code in the src/demo_github_repository/handlers.py file.

Open this file, and replace its contents with the following code, which implements the CREATE, UPDATE, DELETE, and READ handlers required by AWS CloudFormation to facilitate the modeling of the resource:

import logging
from cloudformation_cli_python_lib import (
    Action,
    OperationStatus,
    ProgressEvent,
    Resource,
    exceptions,
)
from .models import ResourceHandlerRequest, ResourceModel
from github import Github, GithubException
# Use this logger to forward log messages to CloudWatch Logs.
LOG = logging.getLogger(__name__)
TYPE_NAME = "Demo::GitHub::Repository"
resource = Resource(TYPE_NAME, ResourceModel)
test_entrypoint = resource.test_entrypoint
def init_gh_client(model):
    progress: ProgressEvent = ProgressEvent(
        status=OperationStatus.SUCCESS, resourceModel=model
    )
    gh_client = Github(model.AccessToken)
    return model, progress, gh_client
def get_repo(id, gh_client):
    try:
        return gh_client.get_repo(int(id))
    except GithubException as e:
        if e._GithubException__status == 404:
            raise exceptions.NotFound(TYPE_NAME, id)
        raise
@resource.handler(Action.CREATE)
def create_handler(_, request: ResourceHandlerRequest, __) -> ProgressEvent:
    model, progress, gh_client = init_gh_client(request.desiredResourceState)
    private = False if model.Visibility == "public" else True
    if model.Org:
        repo_parent = gh_client.get_organization(model.Org)
    else:
        repo_parent = gh_client.get_user()
    model.Namespace = f"{repo_parent.login}/{model.Name}"
    try:
        repo = repo_parent.create_repo(model.Name, private=private)
        repo.replace_topics(["created-with-cloudformation"])
    except GithubException as e:
        # when the repo name already exists, return a specific error
        if isinstance(e.data["errors"][0], dict):
            message = e.data["errors"][0].get("message")
            if message == "name already exists on this account":
                raise exceptions.AlreadyExists(TYPE_NAME, model.Namespace)
        raise exceptions.InternalFailure(str(e.data["errors"]))
    model.Id = int(repo.id)
    model.HttpsUrl = repo.clone_url
    model.SshUrl = repo.ssh_url
    return progress
@resource.handler(Action.UPDATE)
def update_handler(_s, request: ResourceHandlerRequest, _c) -> ProgressEvent:
    model, progress, gh_client = init_gh_client(request.desiredResourceState)
    repo = get_repo(model.Id, gh_client)
    private = model.Visibility == "private"
    repo.edit(name=model.Name, private=private)
    return progress
@resource.handler(Action.DELETE)
def delete_handler(_s, request: ResourceHandlerRequest, _c) -> ProgressEvent:
    model, progress, gh_client = init_gh_client(request.desiredResourceState)
    repo = get_repo(model.Id, gh_client)
    repo.delete()
    return progress
@resource.handler(Action.READ)
def read_handler(_s, request: ResourceHandlerRequest, _c) -> ProgressEvent:
    model, progress, gh_client = init_gh_client(request.desiredResourceState)
    repo = get_repo(model.Id, gh_client)
    model.HttpsUrl = repo.clone_url
    model.SshUrl = repo.ssh_url
    model.Namespace = repo.full_name
    model.Visibility = "private" if repo.private else "public"
    return progress
@resource.handler(Action.LIST)
def list_handler(_s, _r, _c):
    raise NotImplementedError("LIST handler not implemented")

In the Create handler, I catch errors that indicate a repository with the same name already exists. I then raise an AlreadyExists exception to let AWS CloudFormation know what kind of error it is. Similarly, in the get_repo function—called by the UPDATE, DELETE, and READ handlers—I trap exceptions that indicate that the resource doesn’t exist and to raise a NotFound exception. By using the exception types provided by the AWS CloudFormation resource library, I can control behaviors such as retries and the information returned to the user.

4. Declare dependencies

I must use the Python support library to enable the handler (cloudformation-cli-python-lib). In this case, I also need the PyGithub repository, which simplifies interactions with the GitHub API. To do this, I use a standard Python requirements.txt file, which is created when the project initializes. It exists in the root folder and already contains the CloudFormation CLI support library, so I only need to add PyGithub. When it’s complete, the file contains the following:

cloudformation-cli-python-lib==2.1.5
PyGithub==1.54.1

5. Submit resource to AWS CloudFormation

In this example, I specify US West (Oregon) as the AWS Region, but you can use any Region that supports AWS CloudFormation.

cfn submit --set-default --region us-west-2

6. Test resource

Create a template that launches a stack, and save the YAML file as github-repo.yaml:

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  AWSSecretsManagerSecretName:
    Type: String
    Default: "github-access-token"
    Description: The name of the AWS Secrets Manager Secret that contains your GitHub Access Token
    NoEcho: True
Resources:
  MyRepo:
    Type: "Demo::GitHub::Repository"
    Properties:
      Name: "test-repo-from-cfn"
      AccessToken: !Sub "{{resolve:secretsmanager:${AWSSecretsManagerSecretName}}}"
Outputs:
  SshUrl:
    Value: !GetAtt MyRepo.SshUrl
  HttpsUrl:
    Value: !GetAtt MyRepo.HttpsUrl
  FullName:
    Value: !GetAtt MyRepo.Namespace

The template uses a dynamic reference to an AWS Secrets Manager secret to store the GitHub access token. So, before I launch the stack, I create and store an access token. For more information, see Creating a personal access token.

aws secretsmanager create-secret \
  --region us-west-2 \
  --name github-access-token \
  --secret-string <ACCESS_TOKEN>

Replace <ACCESS_TOKEN> with your token value. If you changed the AWS Region when you registered the resource, ensure that you update it to reflect the AWS Region for your resource.

Now you can use the following command to launch the stack:

aws cloudformation deploy \
  --stack-name github-repo \
  --template-file github-repo.yaml \
  --region us-west-2

Stack creation takes a few minutes. The stack output contains the repository details, which can be retrieved using the following command:

aws cloudformation describe-stacks \
  --stack-name github-repo \
  --region us-west-2 \
  --query 'Stacks[0].Outputs'

Sign in to GitHub. You should see a new repository called test-repo-from-cfn.

From here, experiment by updating the stack to change the repository name, set the visibility to public, and specify a GitHub organization.

Cleanup

To avoid incurring future costs, use the following commands to delete the stack and resource type:

aws cloudformation delete-stack \
  --region us-west-2 --stack-name github-repo
aws cloudformation wait stack-delete-complete \
  --region us-west-2 --stack-name github-repo
aws secretsmanager delete-secret \
  --region us-west-2 --secret-id github-access-token
aws cloudformation deregister-type \
  --type-name Demo::GitHub::Repository \
  --type RESOURCE --region us-west-2

In this post, I used Python to create, register, and use a resource type. AWS CloudFormation manages all of your application’s components and makes the template a single source of truth that extends the benefits of CloudFormation to your third-party party components.

To start building your own resource, see Creating resource types. Use the comments to provide feedback about this post or to let me know which resources you’re looking for.