Building a CI/CD pipeline for cross-account deployment of an AWS Lambda API with the Serverless Framework

Modern-day applications that reside on AWS have several distinct environments and accounts, such as dev, test, and staging. An application has to go through an elaborate process of deployment and testing in these environments before reaching its final destination. To achieve automated deployment of the application across different environments, you must use CI/CD pipelines.

Different DevOps models have been proposed that depict how a CI/CD pipeline deploys and promotes an application from one environment to another. In a typical model, pipelines are locally situated in each AWS account where deployment needs to happen. This post, however, focuses on a different model, in which CI/CD pipelines reside in a central AWS account called tools, and carry out deployments across other AWS accounts. This model has several advantages:

All pipelines are now located in a centralized account, which consolidates the security controls and grants increased visibility.
The AWS Identity and Access Management (IAM) permission model is greatly simplified because the pipelines can now share common IAM roles and policies. In addition, there is a clear demarcation between deployment-specific roles that pipelines assume and basic pipeline permissions.
Logs for all pipelines are located in a single account under Amazon CloudWatch.

For more information about CI/CD cross-account pipeline strategies, see Building a Secure Cross-Account Continuous Delivery Pipeline. In this post, we apply this strategy to deploying AWS Lambda-based APIs using the third-party Serverless Framework.

Prerequisites

Before proceeding any further, you need to identify and designate two AWS accounts required for the solution to work:

Tools – The AWS account where pipeline resides
Target – The AWS account where deployment occurs

Solution overview

Start by building the necessary resources in the target account, as shown in the following architecture diagram. This consists of an IAM role that trusts the tools account and provides the required deployment-specific permissions. This IAM role is assumed by AWS CodeBuild in the tools account to carry out deployment. For simplicity, we refer to this role as the cross-account role, as specified in the architecture diagram.

In addition, you also create an AWS CloudFormation execution role to be assumed by AWS CloudFormation in the target account. This role has permissions to create your API resources, such as a Lambda function and Amazon API Gateway, in the target account.

You then build a CI/CD pipeline in the tools account using AWS CloudFormation, AWS CodePipeline, AWS CodeBuild, and AWS CodeCommit. After completing all the steps in this post, you will have a fully functioning CI/CD pipeline that deploys your API in the target account. The pipeline starts automatically every time you check in your changes into your CodeCommit repository.

The following architecture diagram shows the AWS resources across the tools and target accounts. Red arrows depict the flow of events that lead to cross-account deployment of the Lambda-based API.

Serverless Framework

Your API is serverless, which consists of a Lambda function fronted by an API Gateway REST API. You use the Serverless Framework to carry out API deployment. The Serverless Framework is designed to help you define the function and its associated infrastructure components, such as Amazon API Gateway, Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), and Amazon DynamoDB, with minimal configurations.

For instructions in installing the Serverless Framework, see Get Started with Serverless Framework Open Source & AWS.

Creating the infrastructure in the target account

In this step, you create the following resources in the target account:

A cross-account role that has a trust relationship with the tools account. This role provides necessary permissions to CodeBuild in the tools account to carry out deployment.
An AWS CloudFormation execution IAM role that has permissions to create CloudFormation stack resources for your API.

Download the following CloudFormation templates corresponding to the defined roles:

Creating the CloudFormation stacks

To create the CloudFormation stacks for the templates you downloaded, complete the following steps:

Log in to your designated target AWS account.
On the AWS CloudFormation console, choose Create stack.
Select Upload a template to Amazon S3.
Choose Choose File and choose the template you downloaded.
For Stack name, enter an appropriate stack name (for example, myapp-cicd-pipeline).
Review the default parameters.
- For the cross-account role CloudFormation stack, specify your designated AWS account ID for the tools account in the ToolsAccountID parameter.
Choose Next.
Under Options, set any options that you need.
Choose Next.
Select I acknowledge that AWS CloudFormation might create IAM resources.
Choose Create.

AWS CloudFormation creates your stack. After it’s finished, the stack status shows as CREATE_COMPLETE.

Cloudformation stacks successfully created in target account

Creating the CI/CD pipeline in the tools account

You create the CI/CD pipeline in the tools account using the CloudFormation template, which provisions the following required resources:

The CodeCommit repository to store the source code for the API
IAM roles and policies required for the CodePipeline and CodeBuild projects
The Amazon Simple Storage Service (Amazon S3) bucket for the code pipeline to hold source code and build artifacts
The following CodePipeline resources:
- Source – The CodeCommit repository you created earlier
- Build and deploy – The CodeBuild project that packages the Lambda code, assumes the cross-account role, and deploys the Lambda function and API Gateway to the target AWS account
- Pipeline trigger – A CloudWatch event to monitor changes done to the CodeCommit repository and trigger the code pipeline when code is checked into the appropriate branch of the repository (for this post, master)

Setting up the code pipeline

To set up the code pipeline, complete the following steps:

Download the following CloudFormation template.
On the AWS CloudFormation console, choose Create stack.
Select Upload a template to Amazon S3.
Choose Choose File and choose template you downloaded.
For Stack name, enter an appropriate stack name (for example, myapp-cicd-pipeline).
Review the default parameters.
- Specify your designated AWS account ID for the target Account in the TargetAccountID parameter.
Choose Next.
Under Options, set any options that you need.
Choose Next.
Select I acknowledge that AWS CloudFormation might create IAM resources.
Choose Create.
AWS CloudFormation creates your stack. After it’s complete, the stack is in the CREATE_COMPLETE status.
Choose your CloudFormation stack.
On the Outputs tab, record the URL for OutCodeCommitRepoUrl, which you use in the next section.

CodePipeline Cloudformation stack created successfully

Deploying the API

You need to have following utilities installed on your workstation before proceeding further:

Python – You need Python version 3.7 because your Lambda function is written in Python.
Git – This is required to interact with CodeCommit.
- For installation instructions for Linux or Mac, see Setup Steps for HTTPS Connections to AWS CodeCommit Repositories on Linux, macOS, or Unix with the AWS CLI Credential Helper.
- For installation instructions for Windows, see Setup Steps for HTTPS Connections to AWS CodeCommit Repositories on Windows with the AWS CLI Credential Helper.

To deploy the API, complete the following steps:

Step 1: In your workspace, clone the CodeCommit repository you created earlier

git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my-serverless-api

You can get the CodeCommit URL from output parameter OutCodeCommitRepoUrl. The following screenshot shows your output; you have created a new folder in your workspace.

Git clone empty repository

Step 2: Copy all the Serverless API code base from the GitHub repo into this folder.

Your project structure should now look like the following screenshot. Serverless project structure

Per Serverless Framework specifications, every serverless project should have serverless.yml at the project root location. This file defines specifications for deploying the Lambda function and its associated resources (such as API Gateway and DynamoDB) that are required for API implementation. For more information, see Serverless.yml Reference.

The folder /lambda consists of two Lambda APIs written in Python:

create-product.py – Creates a new product in DynamoDB
find-product.py – Searches for the product in DynamoDB

These Lambda source code locations are referenced in serverless.yml.

The pipeline you created in the previous step defines the deploy stage, which is driven by the CodeBuild project. It specifies configuration on how to package and deploy the code. The configuration includes two important properties:

The location of a build-specification .yml file in the source code project. If not specified, the default settings expect a file named buildspec.yml to be at the root of the source code folder structure. You need to specify a file with any other name or location.
The environment variables available to the build environment in which buildspec.yml runs.

See the following code:

  CodeDeploy:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Join [ '-', [ 'Serverless-CodeBuild-Deploy', !GetAtt CodeCommitRepo.Name, !Join [ '-', !Split [ '/', !Ref CodeCommitRepoBranch ] ] ]]
      Artifacts:
        Type: CODEPIPELINE
      Source:
        Type: CODEPIPELINE
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Environment:
        Type: LINUX_CONTAINER
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
        ComputeType: BUILD_GENERAL1_SMALL
        EnvironmentVariables:
          -
            Name: CROSS_ACCOUNT_ROLE
            Type: PLAINTEXT
            Value: !Sub 'arn:aws:iam::${TargetAccountID}:role/${CodePipelineAssumeRoleName}'
          -
            Name: CF_EXECUTION_ROLE
            Type: PLAINTEXT
            Value: !Sub 'arn:aws:iam::${TargetAccountID}:role/${CFExecutionRoleName}'
          -
            Name: TARGET_ACCOUNT_ID
            Type: PLAINTEXT
            Value: !Ref TargetAccountID
          -
            Name: STAGE
            Type: PLAINTEXT
            Value: !Ref DeploymentEnvironment
      Tags:
        - Key: category
          Value: goldmine
        - Key: project_name
          Value: serverless-cross-account-deployment

The crux of the deployment logic resides in the buildspec.yml. It has instructions on how to build, package, and deploy the serverless project. Instructions are in the form of bash commands to be executed in a Linux container provisioned as part of the CodeBuild project. You can choose an appropriate Docker image for the container. For sample images, see Docker images provided by CodeBuild. Additionally, you can specify a runtime version such as Python or Node.js in the buildspec.yml, which gets installed in the container during the install phase.

The buildspec.yml file consists of two phases: install and build. The install phase defines instructions to install prerequisites, which for this use case is the serverless npm package needed to build and deploy the API using Serverless Framework.

In the build phase, you first set up an AWS cross-account profile (as the default profile) using the cross-account role you passed as the environment variable. This post provides a shell script aws-profile-setup.sh (located at the root location of the project) that creates this profile for you.

You then create a serverless package using the sls package command. This takes the configurations you defined in serverless.yml, packages the entire infrastructure into the serverless-package directory, and makes it ready for deployment. You can also pass custom command line parameters and use them inside serverless.yml. One of the parameters you pass here is cfnRoleArn. This represents the AWS CloudFormation execution role to be assumed by AWS CloudFormation service to deploy the stack resources in the target account.

Lastly, you deploy the package using the sls deploy command. This takes the prebuilt package located in the /serverless-package directory and uses the cross-account profile you set up to create a CloudFormation stack in the target account. AWS CloudFormation assumes the IAM role you supplied as cfnRoleArn to provision all the resources that are part of your stack.

You also extract the API endpoint URL from the output of sls info command and pass it on as output artifact so it’s available to subsequent stages of your pipeline. As a future enhancement, you may append another stage to this pipeline to test this API endpoint.

Step 3: Switch to this new repository folder and check in the code you added. See the following code:

cd my-serverless-api
git add –A
git commit -a -m "Initial code checkin"
git push origin master

The following screenshot shows the output.

Git push code to master branch of code commit repository

On the CodePipeline console, you should see that the pipeline kicked off automatically and eventually deploys the API to the target account.

CodePipeline successfull run

In the target account, you can see a stack created with name in the format <SERVICE>-<STAGE>, where <SERVICE> is what you define in serverless.yml and <STAGE> is the environment variable you choose when creating CodePipeline stack. If you used the default configuration, the stack name becomes product-catalog-service-DEV. This stack outputs an API endpoint with the key ServiceEndpoint. In the next step, you test the API endpoint.

Testing the API endpoint

In this step, you use a postman client to test the API you just deployed. You first create a new product and then query that product using its ID. See the following code:

Create Product Endpoint: <api-endpoint>/product/create

Method: POST

Request:

{
    "id": "1001",
    "title": "Amazon Echo",
    "description": "Amazon Echo",
    "price": "89.99"
}

The following screenshot shows the successful response you receive from the create product API endpoint when above request is sent.

Create Product API request/response

Use the following code to search the product you created using its ID:

Find Product Endpoint: <api-endpoint>/product/find/1001

Method: GET

The following screenshot shows that the product information is retrieved successfully when you find the product using its ID.

Find Product API request/response

Security considerations

Cross-account IAM roles are very powerful and therefore need to be handled very carefully. For this post, we have strictly limited the cross-account role to specific Amazon S3, AWS CloudFormation, and API Gateway permissions. This makes sure the role has limited access. Actual creation of resources such as Lambda, API Gateway, and DynamoDB happens via the CloudFormation execution role, which is assumed by AWS CloudFormation in the target account.

Cleanup

To avoid incurring charges please execute following steps in their given order to remove all the resources created in target and tools account.

Delete CloudFormation stack product-catalog-service-DEV from target account. This removes DynamoDB, Lambda, API Gateway resources and their associated IAM roles created by serverless framework. Note that the stack name is in the format <SERVICE>-<STAGE>. Name may differ based on the values you set for service and stage as explained in the above section Deploying the API.
Find the names of CloudFormation stacks you created in the target account for creating cross-account IAM role and CloudFormation execution IAM role; and delete them.
Log into the tools account and delete the CloudFormation stack you created to spin up CI/CD deployment pipeline. This removes CodePipeline, CodeBuild, CodeCommit repository and their associated resources such as IAM Roles and CloudWatch Event.

Conclusion

In this post, we showed how to integrate AWS developer tools such as CodeCommit, CodePipeline, and CodeBuild with the Serverless framework to create a CI/CD pipeline that can deploy a Lambda-based REST API from a centralized tools account to any number of target AWS accounts.

Build your own CI/CD pipeline as shown in this blog using AWS developer tools and Serverless framework.

AWS DevOps & Developer Productivity Blog