Implementing a CI/CD pipeline for AWS IoT Greengrass projects

This post demonstrates how to build a continuous integration and continuous deployment (CI/CD) pipeline for AWS IoT Greengrass projects. Readers should be familiar with AWS IoT Greengrass.

This post focuses on streamlining the development process for AWS IoT Greengrass projects rather than explaining the technology itself. If you are not familiar with AWS IoT Greengrass, you can learn about it using the tutorials at Getting Started with AWS IoT Greengrass.

CI/CD is a modern software development practice that enables the frequent release of small, tested changes into a production software environment. Delivering changes frequently makes their business value available to users as soon as it is ready. Small change sets make it easier to identify the root causes of errors and recover from them. For more information about the benefits and adoption of CI/CD for your team, see Practicing Continuous Integration and Continuous Delivery on AWS.

In this post’s example:

A developer pushes a code change to the master branch of the project’s AWS CodeCommit Git repository.
This change triggers an AWS CodePipeline pipeline that uses AWS CodeBuild to perform the following tasks:
- Deploys the change to a test AWS IoT Greengrass group.
- Executes integration tests for the AWS IoT Greengrass application in the test environment.
- If the Integration Tests succeed, Deploys the changes to the prod AWS IoT Greengrass group, making the new capabilities available to users.

The example builds on the AWS CloudFormation patterns established in the post Automating AWS IoT Greengrass Setup with AWS CloudFormation to establish test and prod AWS IoT Greengrass groups. Each group resides on its own dedicated Amazon EC2 instance. In your project, your AWS IoT Greengrass groups may reside on the edge devices that you chose for your implementation.

Prerequisites

Complete the following steps before proceeding:

Create a non-production account in which to create your resources at the AWS Account Signup Page. Alternatively, you can use AWS Organizations to create and govern new accounts easily and securely.
Install the AWS Command Line Interface (AWS CLI) and configure it to use the access credentials and AWS Region that you have chosen for this exercise. For more information, see Installing the AWS CLI and Configuring the AWS CLI.
Create an EC2 key pair for creating AWS IoT Greengrass groups. For more information, see Creating a Key Pair Using Amazon EC2. Key pairs are not necessary to use AWS IoT Greengrass itself, but allow you to log in to the EC2 instances that act as your deployment environments.
Create the test AWS IoT Greengrass group by launching the test group CloudFormation template and completing the following steps:
- Select your key pair.
- To restrict SSH access to the created EC2 instance, set the SecurityAccessCIDR.
- Acknowledge that the template creates IAM roles and choose Create.
- Let the stack creation finish.

Create the prod AWS IoT Greengrass group by launching the prod group CloudFormation template and repeating the preceding steps.
To create a source repository and a build/deployment pipeline, launch the repo and pipeline AWS CloudFormation template.
- You can use the default parameter values.
- Acknowledge that the template creates IAM roles and choose Create.

Navigating the template

At this point, you should see the following resources in their respective areas of the AWS Console:

The following stacks in CloudFormation:
- gg-cicd-pipeline
- gg-cicd-prod-environment
- gg-cicd-test-environment
A CodeCommit repository (greengrass-cicd-project) to hold the project code.
A CodePipeline pipeline to build, test, and deploy the project code (gg-cicd-pipeline-BuildAndDeployPipeline, ending with your own custom suffix generated by CloudFormation). It is normal that the pipeline be in an error state when first created.
Test and prod AWS IoT Greengrass groups (gg-cicd-test and gg-cicd-prod) in which to run integration tests and production code, respectively. These newly created groups only have cores associated with them. The pipeline updates their definitions to add AWS Lambda functions and subscriptions based on your project code.

Creating a sample AWS IoT Greengrass project

To start, create a directory to work in, using the following commands in your terminal window:

$> mkdir my_work_dir $> cd my_work_dir

In the working directory, clone the empty Git repo to your computer. For help with setting up access to your repo, see Setting Up for AWS CodeCommit. Your Git URL may look a little different than the following:

$> git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/greengrass-cicd-project

Download this project archive and unzip it into your local repository directory. The archive may download to a different directory than the following:

$> cp ~/Downloads/greengrass-cicd-project.zip . $> cd greengrass-cicd-project $> unzip ../greengrass-cicd-project.zip

This archive contains a sample AWS IoT Greengrass project:

$> ls -R LICENSE README.md buildspec.yml requirements.txt src/ test/ ./src: deploy.sh* sample_function/ template.yml ./src/sample_function: __init__.py requirements.txt sample_function.py ./test: init.py certs/ requirements.txt integration_tests.py setup_parameter_store.sh* ./test/certs: root.ca.pem

This project contains a simple Lambda function in src/sample_function/sample_function.py, as follows.

import os
import json
from datetime import datetime
import greengrasssdk


counter = 0
client = greengrasssdk.client('iot-data')

def function_handler(event, context):
    '''Echo message on /in topic to /out topic'''

    response = json.loads(event)
    
    # maybe do something with the event before sending it back
    
    response_string = json.dumps(response)

    client.publish(
        topic='{}/out'.format(os.environ['CORE_NAME']),
        payload=response_string
    )

In AWS IoT Greengrass, you communicate with Lambda functions using the MQTT protocol over the Message Broker for AWS IoT. The set of topics to which a Lambda function subscribes and the set of topics to which it publishes its results acts as its API.

AWS IoT provides a single endpoint per Region within an AWS account. Because a single endpoint can serve multiple applications and deployment environments, it is a best practice to use a pattern like /{application}-{environment}/... when defining topics for your AWS IoT Greengrass applications.

The application prefix allows you to segregate traffic for multiple MQTT-based applications running in a single AWS account and Region.
The environment path element allows you to segregate environments such as prod and test.

In each of your environments, the topic gg-cicd-<env>/in serves as an input to your Lambda function, where <env> is either test or prod, depending on the AWS IoT Greengrass group where the function runs. The Lambda function echoes its input to the gg-cicd-<env>/out topic. You pass the CORE_NAME value into the function’s execution environment and format it into the MQTT topic that you are using.

The definition of AWS IoT Greengrass groups updates via the AWS Serverless Application Model (AWS SAM) template in src/template.yml, shown in the following code. This template defines the AWS IoT Greengrass Lambda function and topics that comprise your AWS IoT Greengrass groups. For more details about defining AWS IoT Greengrass Groups with CloudFormation, see Automating AWS IoT Greengrass Setup with AWS CloudFormation.

The AutoPublishAlias property in the definition of GGSampleFunction causes your Lambda function to publish a new version for deployment to your AWS IoT Greengrass groups whenever the function code changes. If you don’t publish a new version, the previous Lambda definition continues to be active.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Our Application

Parameters:
  CoreName:
    Description: Greengrass Core on which your resources are deployed
    Default: "coreName"
    Type: String

Resources:
  GreengrassGroupVersion:
    Type: AWS::Greengrass::GroupVersion
    Properties:
      GroupId: {'Fn::ImportValue': !Sub '${CoreName}-environment-GreengrassGroupId'}
      CoreDefinitionVersionArn: !Ref GreengrassCoreDefinitionVersion
      FunctionDefinitionVersionArn: !GetAtt FunctionDefinition.LatestVersionArn
      SubscriptionDefinitionVersionArn: !GetAtt SubscriptionDefinition.LatestVersionArn

  GreengrassCoreDefinition:
    Type: AWS::Greengrass::CoreDefinition
    Properties:
      # use CoreName + "_Core" as "thingName"
      Name: !Join ["_", [!Ref CoreName, "Core"] ]

  GreengrassCoreDefinitionVersion:
    # Example of using GreengrassCoreDefinition referring to this
    # "Version" resource
    Type: AWS::Greengrass::CoreDefinitionVersion
    Properties:
      CoreDefinitionId: !Ref GreengrassCoreDefinition
      Cores:
        - Id: !Join ["_", [!Ref CoreName, "Core"] ]
          ThingArn: !Join
            - ":"
            - - "arn:aws:iot"
              - !Ref AWS::Region
              - !Ref AWS::AccountId
              - !Join
                - "/"
                - - "thing"
                  - !Join ["_", [!Ref CoreName, "Core"] ]
          CertificateArn: !Join
            - ":"
            - - "arn:aws:iot"
              - !Ref AWS::Region
              - !Ref AWS::AccountId
              - !Join
                - "/"
                - - "cert"
                  - {'Fn::ImportValue': !Sub '${CoreName}-environment-IoTThingCertificateId'}
          SyncShadow: "false"

  FunctionDefinition:
    Type: 'AWS::Greengrass::FunctionDefinition'
    Properties:
      Name: FunctionDefinition
      InitialVersion:
        DefaultConfig:
          Execution:
            IsolationMode: GreengrassContainer
        Functions:
          - Id: !Join ["_", [!Ref CoreName, "sample"] ]
            FunctionArn: !Ref GGSampleFunction.Version
            FunctionConfiguration:
              Pinned: 'true'
              Executable: index.py
              MemorySize: '65536'
              Timeout: '300'
              EncodingType: binary
              Environment:
                Variables:
                  CORE_NAME: !Ref CoreName
                AccessSysfs: 'false'
                Execution:
                  IsolationMode: GreengrassContainer
                  RunAs:
                    Uid: '1'
                    Gid: '10'

  GGSampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      AutoPublishAlias: live
      CodeUri: sample_function
      Handler: sample_function.function_handler
      Runtime: python2.7
      Role: {'Fn::ImportValue': !Sub '${CoreName}-environment-LambdaExecutionArn'}

  SubscriptionDefinition:
    Type: 'AWS::Greengrass::SubscriptionDefinition'
    Properties:
      Name: SubscriptionDefinition
      InitialVersion:
        # Example of one-to-many subscriptions in single definition version
        Subscriptions:
          - Id: Subscription1
            Source: 'cloud'
            Subject: !Join
              - "/"
              - - !Ref CoreName
                - "in"
            Target: !Ref GGSampleFunction.Version
          - Id: Subscription2
            Source: !Ref GGSampleFunction.Version
            Subject: !Join
              - "/"
              - - !Ref CoreName
                - "out"
            Target: 'cloud'
          - Id: Subscription3
            Source: !Ref GGSampleFunction.Version
            Subject: !Join
              - "/"
              - - !Ref CoreName
                - "telem"
            Target: 'cloud'

Outputs:
  GroupId:
    Value: {'Fn::ImportValue': !Sub '${CoreName}-environment-GreengrassGroupId'}

The pipeline executes the actions in the buildspec.yml file, located in the top level of your project directory, using CodeBuild. The template does the following:

Installs build/test dependencies.
Deploys code to the test group using the helper script src/deploy.sh.
Runs integration tests in the test environment.
If the tests succeed, deploys the code to the prod group using src/deploy.sh.

Deploying to test, running the integration tests, and deploying to prod could be done in three separate CodeBuild actions. Doing so would allow you to independently retry each step of the pipeline. In this example, all three phases are in one build step for simplicity:

version: 0.2

env:
  parameter-store:
    IOT_ENDPOINT: /gg-cicd-test/IOT_ENDPOINT

phases:
  install:
    commands:
      - pip install -q --upgrade pip
      - pip install -q aws-sam-cli
      -
      - cd test
      - pip install -q -r requirements.txt
      - cd ..
  build:
    commands:
      - echo "deploying to test core"
      - cd src
      - ./deploy.sh test
      - cd ..
      -
      - echo "running integration tests"
      - cd test
      - aws ssm get-parameter --name /gg-cicd-test/integration-testing-client.cert.pem --with-decryption --output text --query Parameter.Value > certs/integration-testing-client.cert.pem
      - aws ssm get-parameter --name /gg-cicd-test/integration-testing-client.private.key --with-decryption --output text --query Parameter.Value > certs/integration-testing-client.private.key
      - python -m unittest integration_tests.py
      - cd ..
      -
      - echo "deploying to production core"
      - cd src
      - ./deploy.sh prod
      - cd ..

The deployment script src/deploy.sh uses the AWS SAM CLI to package the Lambda function and execute the AWS SAM template. The template deploys the resources to the AWS IoT Greengrass group specified as its command line argument. The script uses the following code:

#!/bin/bash -e

if [[ $# -ne 1 ]] || ([[ $1 != "test" ]] && [[ $1 != "prod" ]]); then
  echo "Usage $0 <deployment environment (test|prod)>"
  exit 1
fi

DEPLOYMENT_ENVIRONMENT=$1
STACK_NAME="gg-cicd-application-${DEPLOYMENT_ENVIRONMENT}-stack"

sam package \
    --template-file template.yml \
    --output-template-file packaged.yml \
    --s3-bucket ${ARTIFACTS_BUCKET}

sam deploy \
    --template-file packaged.yml \
    --stack-name ${STACK_NAME} \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides CoreName=gg-cicd-${DEPLOYMENT_ENVIRONMENT} \
    --no-fail-on-empty-changeset

GROUP_ID=`aws cloudformation describe-stacks --stack-name ${STACK_NAME} --query 'Stacks[0].Outputs[0].OutputValue' --output text`
GROUP_VERSION_ID=`aws greengrass list-group-versions --group-id ${GROUP_ID} --query 'Versions[0].Version' --output text`

aws greengrass create-deployment --group-id ${GROUP_ID} --group-version-id "${GROUP_VERSION_ID}" --deployment-type NewDeployment

AWS CloudFormation for AWS IoT does not currently support deployment to AWS IoT Greengrass groups. As such, deploy.sh retrieves the GROUP_ID of the target AWS IoT Greengrass group and the GROUP_VERSION_ID of the newly created group version. It supplies them to the AWS CLI to perform the group deployment.

Integration testing

Integration testing determines if your code functions correctly when connected to your other application components in an environment that approximates your production systems. In this case, those components are the Lambda code and the two MQTT topics with which it interacts. The test AWS IoT Greengrass group, deployed to its corresponding EC2 instance, is the environment that approximates production.

The test_echo() function, which resides in test/integration_tests.py, implements the integration test. The test starts by subscribing to the gg-cicd-test/out topic, then publishes a unique message to the gg-cicd-test/in topic. If it sees the published message on gg-cicd-test/out within 25 seconds, the test is successful.

To see how the tests use the AWS IoT Python SDK to subscribe to MQTT topics and register a callback function to process received messages, look at the listen_on_topic() function in this file. Use the AWS IoT SDK to subscribe to topics because the AWS SDK for Python (Boto 3) does not currently have that functionality.

class TestSampleFunction(unittest.TestCase):

    def setUp(self):
        set_received_messages({})

    def test_echo(self):
        input_topic = 'gg-cicd-test/in'
        output_topic = 'gg-cicd-test/out'

        listen_on_topic(client_id='TestSampleFunction.test_echo', response_topic=output_topic)

        iot_client = boto3.client('iot-data')

        message = {
            'id': str(uuid.uuid4()),
            'processed_at': str(datetime.now())
        }

        received_valid_response = False
        for i in range(5):
            if received_valid_response:
                break

            iot_client.publish(topic=input_topic, qos=1, payload=json.dumps(message))
            time.sleep(MESSAGE_WAIT_TIME)

            responses = get_received_messages()

            key = ':'.join([message['id'], output_topic])
            if key in responses:
                received_valid_response = echo_response_is_valid(message, responses[key])

        assert received_valid_response

Deploying the project

The integration tests interact with your deployed Lambda function using the AWS IoT Core MQTT topics. Before you deploy for the first time, run the following commands to create the necessary AWS IoT certificates for the test’s MQTT client, and store them in Systems Manager Parameter Store. This makes them available to the tests in CodeBuild securely.

$> cd test
$> ./setup_parameter_store.sh
$> cd ..

Next, push the project code to your repo to kick off the pipeline:

$> git add .
$> git commit -m 'adding project code'
$> git push -u origin master

After the pipeline has completed successfully, you see actions for PullSource and CodeBuildAndDeployStage in the CodePipeline console for your pipeline. From BuildAndDeploy, choose Details to see the output from CodeBuild.

In the CloudFormation stacks list, there are two new Application stacks: gg-cicd-application-prod-stack and gg-cicd-application-test-stack. Each deploys your project to the test and prod AWS IoT Greengrass groups, respectively.

Your AWS IoT Greengrass groups now show a recent deployment and have a Lambda function and subscriptions associated with them.

Interacting with the deployed Lambda function

Using the AWS IoT MQTT client, subscribe to the gg-cicd-test/out topic and send messages to the gg-cicd-test/in topic to interact with your deployed test group. To interact with your deployed prod group, use gg-cicd-prod/out and gg-cicd-prod/in instead.

Pushing a change

Use your favorite editor or integrated development environment (IDE) to add a timestamp to the responses generated by the Lambda function in src/sample_function/sample_function.py. It should look like the following:

import os
import json
from datetime import datetime
import greengrasssdk


counter = 0
client = greengrasssdk.client('iot-data')

def function_handler(event, context):
    '''Echo message on /in topic to /out topic'''

    response = json.loads(event)
        
    # Add the time we processed the message to our response
    response['processed_at'] = str(datetime.now())
    
    response_string = json.dumps(response)

    client.publish(
        topic='{}/out'.format(os.environ['CORE_NAME']),
        payload=response_string
    )

While in the greengrass-cicd-project directory, commit the change and push it to the master branch of your CodeCommit repo:

$> git add .
$> git commit -m 'adding a timestamp to our Lambda response'
$> git push -u origin master

This change kicks off your pipeline to test and deploy the change, but this time the pipeline fails. The CodeBuild output, shows that the message sent by the integration test has a slightly different processed_at timestamp than that in the response. This difference is because the Lambda function and your test both use processed_at to record their timestamps. The Lambda function is overwriting the timestamp set by the test, so the response differs from the original message.

Fixing the bug

To fix the bug, change the key that your Lambda function uses from processed_at to echo_processed_at. Now your test and Lambda function store their timestamps in different message fields.

Edit src/sample_function/sample_function.py so that it matches the following code:

import os
import json
from datetime import datetime
import greengrasssdk


counter = 0
client = greengrasssdk.client('iot-data')

def function_handler(event, context):
    '''Echo message on /in topic to /out topic'''

    response = json.loads(event)
        
    # Add the time you processed the message to the response
    response['echo_processed_at'] = str(datetime.now()) # use echo_processed_at
    
    response_string = json.dumps(response)

    client.publish(
        topic='{}/out'.format(os.environ['CORE_NAME']),
        payload=response_string
    )

Commit and push the change to CodeCommit:

$> git add .
$> git commit -m 'put the echo timestamp in its own message field'
$> git push -u origin master

This time, the pipeline succeeds. You fixed the bug!

Verifying the fix

You can verify that your new code is deployed to your prod group using the IoT MQTT client. Subscribe to the gg-cicd-prod/out topic and send messages to the gg-cicd-prod/in topic. Your Lambda function has added the echo_processed_at timestamp to your input message, as shown in the following screenshot.

Cleaning up

When you are finished with the resources created in this post, you can delete the CloudFormation stacks that you used to create them. Keep the following in mind when doing so:

When deleting the gg-cicd-pipeline stack, you might first have to empty the artifact bucket that it has created.
Before deleting the gg-cicd-<test|prod>-environment stacks, reset the group deployments.
When deleting the gg-cicd-<test|prod>-environment stacks, you may encounter a GroupDeploymentReset issue. If so, delete the stack again.

Additionally, you can manually delete the following resources using the console:

Parameter Store entries—Delete /gg-cicd-test/IOT_ENDPOINT, /gg-cicd-test/integration-testing-client.cert.pem and /gg-cicd-test/integration-testing-client.private.key.
Integration test AWS IoT certificate—Delete the certificate with the gg-cicd-test-integration-test-client policy attached.
Integration testing AWS IoT policy—Delete the policy named gg-cicd-test-integration-test-client.

Troubleshooting errors

If you encounter errors when you deploy the stack, review the Status reason column in the Events section. For more information, see Automating AWS IoT Greengrass Setup with AWS CloudFormation.

Here are some common errors and their resolution:

GroupDeploymentReset—This error usually occurs when an IAM service role is not associated with AWS IoT Greengrass in the AWS Region deploying the stack. Check the CloudWatch stream logs for errors. For information about associating an IAM service role, see Greengrass Service Role.
Template Format Error: Unrecognized resource types —This error occurs when the template launches and indicates that AWS IoT Greengrass is not available in the AWS Region. Be sure to use a Region that supports AWS IoT Greengrass. For more information, see AWS Regions and Endpoints.
Maximum VPCs reached—This error occurs when you cannot create additional VPCs in the AWS Region. You can either request a service limit increase in the same AWS Region, or deploy the stack in a different AWS Region that supports AWS IoT Greengrass.
Incorrect root.ca.pem—If the checked root.ca.pem is not the right one for your certificates, select the appropriate one from the Amazon Trust Services Repository.

Summary

This post covers the architecture of a CI/CD pipeline for an AWS IoT Greengrass project and demonstrates:

Using AWS CloudFormation stacks to create separate test and production AWS IoT Greengrass environments
Using CodePipeline and CodeBuild to update the test AWS IoT Greengrass environment when pushing a code change to the master branch of your CodeCommit Git project repository
Using the Boto3 and AWS IoT Python SDKs to implement an integration test in the test environment
Updating the production environment only when tests have passed.
Introducing a bug and preventing deployment to production with an integration test
Fixing the bug and deploying the fix from the pipeline to production

Adopting these techniques should enable you to start using CI/CD practices to quickly deliver business value and reduce operational risk in your AWS IoT Greengrass projects!

The Internet of Things on AWS – Official Blog