Quickly deploy a CI/CD pipeline for Serverless .NET applications on AWS

This is part 3 in the blog post series that is designed to introduce .NET Quick Starts for AWS. In this post, we will introduce you to the Quick Start that builds a complete CI/CD pipeline using AWS CodePipeline. This CI/CD pipeline is for a serverless .NET application built using AWS Lambda, Amazon API Gateway, and AWS Serverless Application Model (AWS SAM).

In part 2 of this series, we explored how by using the AWS CI/CD for .NET Applications on AWS Fargate Quick Start, .NET developers can create a fully managed CI/CD pipeline to build, test, and deploy their .NET (Core) applications to Amazon Elastic Container Service (Amazon ECS) on AWS Fargate.

Consider going through the first post in this series that explains what an AWS Quick Start is and how it helps .NET developers get started quickly with .NET development on AWS. It also walks you through a simple .NET Development Environment setup Quick Start.

Introduction

Modern applications are usually built using serverless-first strategy – a strategy that prioritizes the adoption of serverless services, which allow you to increase agility throughout your application stack. They are built with modular architectural patterns, serverless operational models, and agile developer processes. This allows organizations to innovate faster, reduce risk, accelerate time to market, and decrease their total cost of ownership (TCO) – all without managing servers. The concept of serverless computing on AWS is based on the following tenets: no server management, pay-for-value services, continuous scaling, and built-in fault tolerance.

There are many AWS services that are at the forefront of serverless development. We are going to introduce and use a few of those technologies in this post, including AWS Lambda, Amazon API Gateway, and AWS Serverless Application Model with this Quick Start.

Architecture

The diagram in Figure 1 captures the architecture of the CI/CD pipeline and the serverless workload deployed by the Quick Start. Let’s explore this architecture at a high level before diving deeper into the details.

Figure 1: Reference architecture diagram

This Quick Start builds a CI/CD pipeline using a sample .NET serverless application. You can learn more about this application and its structure by visiting the GitHub repository. The link is also available on the Quick Start home page and in the deployment guide. The folder dotnet-app inside the repo has the application source code and all the other artifacts that will aid in building your serverless application.

The Quick Start sets up the entire CI/CD pipeline for your serverless application using AWS CodePipeline. AWS CodePipeline is a fully managed continuous delivery service that enables you to model, visualize, and automate the steps required to release your software.

It creates three stages in your AWS CodePipeline that orchestrate the complete flow between version control, application build, and deployment. Each stage produces artifacts that are used by the next stage.

The Source stage uses AWS CodeCommit, a highly secure, scalable, and managed source control service that hosts private Git repositories. It supports all the standard functionality of Git, so it works seamlessly with your existing Git-based tools. The Quick Start creates an AWS CodeCommit repository for your sample serverless API application.

The Build stage uses AWS CodeBuild, a fully managed build service. CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy. These artifacts that can include binaries, AWS CloudFormation templates among others are uploaded to an S3 bucket to be consumed by stages further down in the AWS CodePipeline. The application in our reference deployment contains a buildspec.yml  that includes the instructions for AWS CodeBuild on building your serverless app. We will examine and understand the contents of the buildspec file later in the blog.

The Deploy stage uses the CloudFormation template produced by the build stage. The template gets deployed during this stage to your environment. It creates an AWS Lambda application consisting of an AWS Lambda function to host the API, and a REST API in Amazon API Gateway to act as a “front door” for our Lambda function.

The reference application uses the AWS Serverless Application Model (AWS SAM). AWS SAM is an open-source framework that you can use to build serverless applications on AWS. It provides you with a shorthand syntax to model your serverless applications that include Lambda functions, APIs, databases, and event source mappings. You can learn more about AWS SAM from the developer guide.

The Quick Start also creates several AWS Identity and Access Management (IAM) roles to provide fine-grained access to resources requiring specific permissions on other resources to do their job.

Deployment

Before you begin your deployment, make sure to go through the “Planning the deployment” section in the deployment guide. After you have reviewed the section and made sure you meet all the requirements, follow the instructions in the section “Launch the Quick Start.”

The instructions provide a link to the AWS CloudFormation template that you deploy into your account. The template models the AWS resources that make up the CI/CD pipeline for the .NET serverless application deployed by the Quick Start.

Figure 2: Stack details for the AWS CloudFormation template that comes with the Quick Start

As you launch the template following the instructions from the deployment guide, note that on the Specify stack details step, you can customize the stack name or leave the default, dotnet-lambda-cicd. The rest of this blog post assumes the use of the default stack name as shown in Figure 2. The parameters Quick Start S3 bucket name and Quick Start S3 key prefix provide the name of the Amazon S3 bucket and the prefix that point to the application’s source code location within the bucket. During deployment, the CloudFormation template pulls down the source code of the sample application from this location to populate the AWS CodeCommit repository that it creates. You can explore application files on the Github repository mentioned in the Architecture section above.

These parameters also provide the flexibility of using your own source code. Package your source code, upload to an S3 bucket, and change the two parameters accordingly.

Finish following the instructions in “Launch the Quick Start” section to launch the CloudFormation Stack.

Figure 3: CloudFormation stacks deployed by the Quick Start

The CloudFormation Stack you just launched, dotnet-lambda-cicd, will take a couple of minutes to deploy. A few minutes after this stack is deployed, you will observe another CloudFormation Stack, dotnet-lambda-cicd-Deploy, that shows as deploying. This CloudFormation stack is the output from running our CI/CD pipeline that creates and updates our application infrastructure and deploys code with every change.  We will dive deeper into this a little later. Once dotnet-lambda-cicd is deployed, navigate to the resources, and outputs tabs for the stack on the CloudFormation console.

You will observe from Figure 4 that CloudFormation has built the entire CI/CD infrastructure for your project along with the IAM roles that give fine-grained permissions to various services to build and deploy your application. You can visit the IAM service inside your AWS console and review these roles and the permission policies attached to them.

Figure 4: Resources deployed by the Quick Start’s CloudFormation Stack

As shown in Figure 5, the outputs tab will provide

• CodePipelineName – The name of the AWS CodePipeline.
• CodePipelineURL – Link to the AWS CodePipeline.
• Postdeployment – Link to the Quick Start deployment guide.
• RepoUrl – URL for the code repository on AWS CodeCommit. (You have to create your HTTPS git credentials to access the repo outside of the AWS Developer Tools console)

Figure 5: Outputs from the Quick Start’s CloudFormation Stack

Now, let’s talk about the other CloudFormation stack, dotnet-lambda-cicd-Deploy.

After deploying your Quick Start, a build of your application starts. Navigate to the CodePipeline URL output to monitor the build process. The build process creates the CloudFormation template named dotnet-lambda-cicd-Deploy (or <Quick Start stack name>-Deploy if you named your stack differently). This CloudFormation stack will deploy the sample serverless application that is part of the Quick Start.

Once the CodePipeline completes its execution, navigate back to the CloudFormation console and look at the Resources tab and the outputs tab for the newly deployed stack.

As shown in Figure 6, the template has provisioned the infrastructure for your serverless application and deployed your code. It created a Lambda function to host your API and fronted your Lambda function with a REST API on Amazon API Gateway.

Figure 6: Resources deployed by the CloudFormation template produced by AWS CodeBuild

When you navigate to the Outputs tab, you’ll find

• Link to the API
• Amazon Resource Name (ARN) for the lambda function hosting the API.
• ARN for the IAM role created. This role provides the Lambda function permissions to CloudWatch Logs.

Figure 7: Outputs from the CloudFormation stack produced by AWS CodeBuild

Navigate to the ServerlessExampleAPI link on the Outputs tab, presented on Figure 7, to verify your sample serverless API is deployed and running. You should see the output from Figure 8.

Figure 8: Deployed serverless application

Usage

Now that we have successfully deployed our Quick Start, let’s dive deeper into the resources it created to understand the architecture better.

From the outputs section of the dotnet-lambda-cicd stack, as presented in Figure 5, select the CodePipelineUrl to navigate to the AWS CodePipeline console. You can see the three different stages within the Pipeline that the Quick Start created for you. On this Pipeline visualization, presented in Figure 9, you will also find details such as execution ID and a link to the last pipeline execution, latest execution status of each stage, time of execution, link to learn more details about execution of each stage, and details about what triggered the pipeline execution.

Figure 9: Pipeline deployed by the Quick Start in AWS

On the source stage, select the AWS CodeCommit link to navigate to the source code repository. Explore the repo that was created for you by the Quick Start with the sample .NET Lambda function. The root of the repo is presented in Figure 10. You also have the option to select the source code commit hash (in this case, 25ff3af1 from the screen shot in Figure 9), and navigate to the commit details that triggered the pipeline.

Figure 10: Source code repository created in CodeCommit

We will take a closer look at two of the files, buildspec.yml and template.yaml, in the following section when we talk about the CodeBuild stage.

Navigate to the build project created by the Quick Start as part of the CodePipeline pipeline. You can do this either by selecting the AWS CodeBuild link from the CodePipeline pipeline page or visiting the AWS CodeBuild console directly. As presented in Figure 11, from here you can view the build configuration, build history, links to the build runs that provide more details about each run, the build triggers and metrics such as total builds, failed builds, successful builds, and the duration of builds.

Figure 11: Build Project created in CodeBuild as part of the pipeline

AWS CodeBuild provides build environments with various configurations of compute type, memory, vCPUs, and disk space. Refer to the Build environment compute types documentation to learn more about the options. Let’s look at the CloudFormation stack that is deployed by the Quick Start, dotnet-lambda-cicd, to learn more about the current build project.

The build environment for the project is a native dotnet 5.0 image-based Linux environment with a BUILD_GENERAL1_SMALL compute type. This compute type provides an environment with 2 vCPUs, 3 GB of memory, and 64 GB of storage.

Mappings:
  CodeBuildImages:
    DotNet:
      ImageName: aws/codebuild/standard:5.0
Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: CODEPIPELINE
      Source:
        Type: CODEPIPELINE
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        PrivilegedMode: true # Allows access to Docker daemon
        Image: !FindInMap [ CodeBuildImages, DotNet, ImageName ]
        EnvironmentVariables:
          - Name: AWS_ACCOUNT_ID
            Value: !Ref AWS::AccountId
          - Name: AWS_DEFAULT_REGION
            Value: !Ref AWS::Region
          - Name: ARTIFACT_BUCKET
            Value: !Ref ArtifactStore
      Name: !Ref AWS::StackName
      ServiceRole: !Ref CodeBuildServiceRole

Now that you are familiar with your build environment, let us learn more about the build definition and how it’s building your serverless application.

AWS CodeBuild uses a build specification file to run a build. This file is a collection of build commands and related settings, in YAML format. You can include it as part of the source code or you can define it when you create a build project. If you include it as part of the source code, by default, the file must be named buildspec.yml and placed in the root of your source directory.

At the root of the repo that AWSCodeBuild uses to build your application, you will find a buildspec.yml file with the following contents:

version: 0.2
phases:
  install:
    runtime-versions:
        dotnet: latest
  build:
    commands:
      # Compile our .NET code - the compiled assemblies will be delivered
      # to the build sub-folder.
      - sam build -b build -t template.yaml
      # Package the compiled assemblies, upload to S3, and create a new
      # CloudFormation template that uses the newly-generated Zip file.
      - sam package -t build/template.yaml --s3-bucket $ARTIFACT_BUCKET --output-template-file packaged-template.yml
artifacts:
  files:
    - packaged-template.yml

Please visit the Build specification reference for CodeBuild section of the AWS CodeBuild user guide to learn more about the file and its syntax. In our buildspec.yml file, phases represent the required sequence and lists the commands CodeBuild runs during each phase of the build. We make sure we are using the latest version of dotnet by specifying dotnet: latest at phases/install/runtime-versions.

The build commands for our project are specified in the phases/build/commands section. CodeBuild runs these commands during the build, one at a time, in the order listed, from beginning to end. Our buildspec file has two SAM commands, sam build and sam package, in phases/build/commands section. Let’s try to understand what they are.

As we learned earlier, our application and its infrastructure are modeled using AWS SAM, an open-source framework to build serverless applications. You can define your SAM-based applications using AWS SAM templates or AWS Cloud Development Kit (AWS CDK). The quick start uses a SAM template. These templates are essentially an extension of AWS CloudFormation templates and they make it easy to work with serverless applications. The developer guide provides more details about the SAM template anatomy.

Our application includes a SAM template, template.yaml, at the root of the repository. Here we were able to define the entire set of services that make up our serverless application with only a few lines of YAML code. This shows you the power of the shorthand syntax provide by AWS SAM.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  SAM Template for dotnet-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 10
  Api:
    BinaryMediaTypes:
    - image~1vnd.microsoft.icon
Resources:
  ServerlessExampleFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: ./src/ServerlessExample/
      Handler: ServerlessExample::ServerlessExample.Function::FunctionHandler
      Runtime: dotnetcore3.1
      MemorySize: 256
      Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
        Variables:
          PARAM1: VALUE
      Events:
        ServerlessExample:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
        Favicon:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /favicon.ico
            Method: get
Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  ServerlessExampleApi:
    Description: "API Gateway endpoint URL for Prod stage forvServerless Example function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  ServerlessExampleFunction:
    Description: "Serverless Example Lambda Function ARN"
    Value: !GetAtt ServerlessExampleFunction.Arn
  ServerlessExampleFunctionIamRole:
    Description: "Implicit IAM Role created for Serverless Example function"
    Value: !GetAtt ServerlessExampleFunctionRole.Arn

To locally build, test, and debug serverless applications defined using SAM templates, you use the AWS SAM CLI. It provides a lambda-like local execution environment to test your application and catch issues before deployment to AWS. You can also use SAM CLI to deploy your applications to AWS.

The sam build command builds the serverless application that is defined by the SAM template file, template.yaml, at the root of the application directory and delivers it to the build sub-folder in our repo.

The next command sam package packages the compiled assemblies built by sam build into a zip file, uploads them to the S3 artifact bucket, and creates a new CloudFormation template that references the zipped artifacts from S3. This new CloudFormation template is then saved as packaged-template.yml, which then gets saved as an output artifact to S3 from the build project.

You can visit the AWS SAM CLI command reference section of the AWS SAM developer guide to learn more about the SAM CLI commands.

Finally, to deploy our application, our pipeline configured a Deploy stage with an action that uses AWS CloudFormation as the action provider. It uses the output artifact from our build stage, the CloudFormation template packaged-template.yml, to deploy the application. If you have used the default names you will now find the CloudFormation stack, created by our Deploy stage, with the name dotnet-lambda-cicd-Deploy on your CloudFormation console.

The deploy stage is defined by the following lines of code in the Quick Start template dotnet-lambda-cicd.

        - Name: Deploy
          Actions:
            - Name: Deploy
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: '1'
                Provider: CloudFormation
              Configuration:
                ActionMode: REPLACE_ON_FAILURE
                StackName: !Sub ${AWS::StackName}-Deploy
                Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND
                RoleArn: !GetAtt DeploymentRole.Arn
                TemplatePath: BuildOutput::packaged-template.yml
              InputArtifacts:
                - Name: BuildOutput
              RunOrder: 1

Now, let’s take a look at the Lambda and the Amazon API Gateway services on the console to see what the CodePipeline deployment created for us.

The SAM CLI used by the pipeline has created an AWS Lambda application stack, as shown in Figure 12, with the combination of an AWS Lambda function (that hosts our C# Lambda function code), and a REST API in the Amazon API Gateway service that integrates with our backend Lambda function. You can select the Logical IDs of the Lambda function and the ApiGateway RestApi from the Lambda Applications page to explore them.

Figure 12: Lambda Application deployed by our pipeline

Figure 13: REST API created in API Gateway service

Figure 14: Lambda function created as part of the serverless application

Cost and licenses

You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start reference deployment. There is no additional cost for using the Quick Start itself.

The AWS CloudFormation templates for this Quick Start include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.

Tip: After you deploy the Quick Start, create AWS Cost and Usage Reports to track costs associated with the Quick Start. These reports deliver billing metrics to an Amazon Simple Storage Service (Amazon S3) bucket in your account. These reports provide cost estimates based on usage throughout each month and aggregate the data at the end of the month. For more information, see What are AWS Cost and Usage Reports?

Cleanup

If you no longer want to keep the resources created as part of this Quick Start in your account, you need to navigate into the CloudFormation console. This Quick Start uses CloudFormation to deploy an application using an IAM role created by the Quick Start template. To avoid an IAM role error when deleting the Quick Start template stack, ensure that you first delete the deployed application stack, dotnet-lambda-cicd-Deploy, before deploying the Quick Start template stack, dotnet-lambda-cicd.

This process will delete most of the resources that are used by the Quick Start. The one resource that it will not delete is the Amazon S3 bucket used as an Artifact store by the AWS CodePipeline. Navigate to the S3 bucket and follow the instructions in the product documentation on deleting a bucket.

Conclusion

In this blog post, we looked at how we can leverage CI/CD Pipeline for Serverless .NET application AWS Quick Start to quickly and easily get started with developing .NET serverless applications on AWS within minutes. A natural next step would be to use this reference deployment as a baseline and further customize it for your needs. Pull down your own source code as part of this Quick Start by customizing the CloudFormation template parameters.

Visit the AWS Serverless Application Repository to discover, deploy, and publish serverless applications. Follow the blog channel Windows on AWS for easy to consume, focused blog posts, and case studies about Windows and .NET workloads on AWS. You can also follow the .NET on AWS YouTube playlist, and the official .NET on AWS Twitter handle to get the latest news about .NET on AWS.

If you have any other ideas on how the AWS Quick Start for .NET can help you get started quickly and be more productive, please share them on the official .NET on AWS Twitter handle.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your modernization journey today.