Containerize robot applications for testing in the cloud and deploying at the edge

Robotics developers use the open source Robot Operating System (ROS) to build distributed and complex applications for their robots. Robot developers benefit from using the cloud for building, training, and testing robotic applications at scale with Continuous integration and continuous deployment (CI/CD) workflows for physical testing and updates pushed over the air. Containers provide a mechanism to build and package robot applications and simulations for use at the edge and in the cloud with unprecedented ease and flexibility.

Overview

In this blog, you will build and package your ROS and ROS2 applications in a container image. You will use the AWS for Robotics cloud extension to store ROSbags in Amazon Simple Storage Service (Amazon S3) as an example of a ROS package that you will containerize for use with AWS IoT Greengrass V2. AWS IoT Greengrass V2 provides an open source edge runtime, a rich set of pre-built software components, tools for local software development, and new features for managing software on large fleets of devices. We will go through the following steps:

1. Clone the ROS package.
2. Create a Standard Dockerfile with entrypoint script.
3. Create a Docker compose file that builds the image for your robot application.
4. Publish the Docker images to Amazon Elastic Container Registry (Amazon ECR), a container registry in the cloud.
5. Optional: Deploy the ROSbags to Amazon S3 docker container as an AWS IoT Greengrass component.

Prerequisites

The following are requirements to follow along with this blog:

• An AWS account with permissions to AWS Cloud9, Amazon Elastic Compute Cloud (Amazon EC2), AWS Identity and Access Management (AWS IAM), Amazon Simple Storage Service (Amazon S3), Amazon ECR, and AWS IoT Greengrass (optional).
• Basic understanding of software containers.

Prepare the environment

This tutorial uses AWS Cloud9 Integrated Development Environment (IDE). You can also develop locally on your own Linux-based machine. If you are developing on your local machine, make sure to install the AWS CLI and configure an AWS profile with administrative permissions, per Installing or updating the latest version of the AWS CLI – AWS Command Line Interface (amazon.com).

1. Setting up the development environment with AWS Cloud9.
   • In the AWS Cloud9 console, choose Create environment.
   • Enter a name and description for your development environment, then choose Next step.
   • Under Configure settings, change the platform to Ubuntu. Keep the rest as default and then choose Next step.
   • Choose Create environment.
   • The Amazon Elastic Block Storage (EBS) volume is only 10GiB so you must increase the volume for this walkthrough.
   • In a new window, open the Amazon Elastic Compute Cloud (EC2) console, choose Instances, and select the checkbox next to the AWS Cloud9 instance with the name you created in the previous step.
   • In the details below, choose Storage, then scroll down and choose the Volume ID.
   • This will take you to the Volumes page. Select the checkbox next to your volume and then choose Actions, Modify volume.
   • Increase the size of your volume from 10GiB to 30GiB, then choose Modify.
   • Go back to the window with your AWS Cloud9 development environment.
   • Choose the plus symbol (+) and then choose New terminal.
   • Reboot your instance. This will temporarily stop your AWS Cloud9 development environment:

     sudo reboot

   • After a few minutes, check that the volume has been updated. Enter:

     df -h

   • You will see an output that shows your EBS volume was increased to around 30G from 10G:
     $ df -h
     Filesystem      Size  Used Avail Use% Mounted on
     devtmpfs        484M     0  484M   0% /dev
     tmpfs           492M     0  492M   0% /dev/shm
     tmpfs           492M  472K  491M   1% /run
     tmpfs           492M     0  492M   0% /sys/fs/cgroup
     /dev/xvda1       30G 1G   22G  27% /
     tmpfs            99M     0   99M   0% /run/user/1000

3. Install Colcon build tool: Please refer to colcon installation guide:

   sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu `lsb_release -cs` main" > /etc/apt/sources.list.d/ros-latest.list'
   sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
   sudo apt update
   sudo apt install python3-colcon-common-extensions

4. Install Docker Compose:

   sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   sudo chmod +x /usr/local/bin/docker-compose

Clone ROS cloud extensions repository

1. Create a ROS workspace and a source directory:

   mkdir -p ~/environment/ros-workspace/src

2. Clone the package into the source directory:

   cd ~/environment/ros-workspace/src
   git clone https://github.com/aws-robotics/rosbag-uploader-ros1.git

Build docker containers

1. Create a new file named Dockerfile in the directory where you created the ROS workspace. If you are following along, you will be using  ~/environment. Copy/paste the following into the Dockerfile:
   Note: To get started, you can use the sample Dockerfile below that will work with ROS workspaces. If you are using your own ROS workspace, you can adapt this file with your own instructions.
   When building Docker images there are a few helpful terms and tricks to know. First, a Dockerfile is an instruction set used to build Docker images. This approach uses a multi-stage build and integrated caching with Docker BuildKit. Multi-stage builds allow workflows with separate build steps, so the build dependencies and source code is not copied into the runtime image. This reduces the size of the Docker image and improves performance. The caching operations speed up future builds by storing previously built files. To learn more about Docker BuildKit, click here.
   Read through the comments in the Dockerfile to get a sense for what is being built and adapt as necessary. For ease of development, the Dockerfile is based on the official ROS Docker images maintained by Open Source Robotics Foundation (OSRF). However, when running in production, you may choose to build the ROS base image with the OSRF source instruction-set in GitHub to prevent impact from upstream changes.
   

   # ======== ROS/Colcon Dockerfile ========
   # This sample Dockerfile will build a Docker image
   # in any ROS workspace where all of the dependencies are managed by rosdep.
   # 
   # Adapt the file below to include your additional dependencies/configuration outside of rosdep.
   # =======================================
   
   # ==== Arguments ====
   # Override the below arguments to match your application configuration.
   # ===================
   
   # ROS Distribution (ex: melodic, foxy, etc.)
   ARG ROS_DISTRO=melodic
   # Application Name (ex: helloworld)
   ARG APP_NAME=robomaker_app
   # Path to workspace directory on the host (ex: ./robot_ws)
   ARG LOCAL_WS_DIR=workspace
   # User to create and use (default: robomaker)
   ARG USERNAME=robomaker
   # The gazebo version to use if applicable (ex: gazebo-9, gazebo-11)
   ARG GAZEBO_VERSION=gazebo-9
   # Where to store the built application in the runtime image.
   ARG IMAGE_WS_DIR=/home/$USERNAME/workspace
   
   # ======== ROS Build Stages ========
   # ${ROS_DISTRO}-ros-base
   #   -> ros-robomaker-base 
   #      -> ros-robomaker-application-base
   #         -> ros-robomaker-build-stage
   #         -> ros-robomaker-app-runtime-image
   # ==================================
   
   # ==== ROS Base Image ============
   # If running in production, you may choose to build the ROS base image 
   # from the source instruction-set to prevent impact from upstream changes.
   # ARG UBUNTU_DISTRO=focal
   # FROM public.ecr.aws/lts/ubuntu:${UBUNTU_DISTRO} as ros-base
   # Instruction for each ROS release maintained by OSRF can be found here: https://github.com/osrf/docker_images
   # ==================================
   
   # ==== Build Stage with AWS RoboMaker Dependencies ====
   # This stage creates the robomaker user and installs dependencies required to run applications in RoboMaker.
   # ==================================
   
   FROM public.ecr.aws/docker/library/ros:${ROS_DISTRO}-ros-base AS ros-robomaker-base
   ARG USERNAME
   ARG IMAGE_WS_DIR
   
   RUN apt-get clean
   RUN apt-get update && apt-get install -y \
       lsb  \
       unzip \
       wget \
       curl \
       xterm \
       python3-colcon-common-extensions \
       devilspie \
       xfce4-terminal
   
   RUN groupadd $USERNAME && \
       useradd -ms /bin/bash -g $USERNAME $USERNAME && \
       sh -c 'echo "$USERNAME ALL=(root) NOPASSWD:ALL" >> /etc/sudoers'
       
   USER $USERNAME
   WORKDIR /home/$USERNAME
   
   RUN mkdir -p $IMAGE_WS_DIR
   
   # ==== ROS Application Base ====
   # This section installs exec dependencies for your ROS application.
   # Note: Make sure you have defined 'exec' and 'build' dependencies correctly in your package.xml files.
   # ========================================
   FROM ros-robomaker-base as ros-robomaker-application-base
   ARG LOCAL_WS_DIR
   ARG IMAGE_WS_DIR
   ARG ROS_DISTRO
   ARG USERNAME
   
   WORKDIR $IMAGE_WS_DIR
   COPY --chown=$USERNAME:$USERNAME $LOCAL_WS_DIR/src $IMAGE_WS_DIR/src
   
   RUN sudo apt update && \ 
       rosdep update && \
       rosdep fix-permissions
   
   # Note: This will install all dependencies. 
   # You could further optimize this by only defining the exec dependencies. 
   # Then, install the build dependencies in the build image.
   RUN rosdep install --from-paths src --ignore-src -r -y
   
   # ==== ROS Workspace Build Stage ==== 
   # In this stage, you install copy source files, install build dependencies and run a build. 
   # ===================================
   FROM ros-robomaker-application-base AS ros-robomaker-build-stage
   LABEL build_step="${APP_NAME}Workspace_Build"
   ARG APP_NAME
   ARG LOCAL_WS_DIR
   ARG IMAGE_WS_DIR
   
   RUN  . /opt/ros/$ROS_DISTRO/setup.sh && \
       colcon build \
        --install-base $IMAGE_WS_DIR/$APP_NAME
        
   # ==== ROS Robot Runtime Image ====
   # In the final stage, you copy the staged install directory to the runtime image.
   # =================================
   FROM ros-robomaker-application-base AS ros-robomaker-app-runtime-image
   ARG APP_NAME
   ARG USERNAME
   ARG GAZEBO_VERSION
   
   ENV USERNAME=$USERNAME
   ENV APP_NAME=$APP_NAME
   ENV GAZEBO_VERSION=$GAZEBO_VERSION
   
   RUN rm -rf $IMAGE_WS_DIR/src
   
   COPY --from=ros-robomaker-build-stage $IMAGE_WS_DIR/$APP_NAME $IMAGE_WS_DIR/$APP_NAME
   
   # Add the application source file to the entrypoint.
   WORKDIR /
   COPY entrypoint.sh /entrypoint.sh
   RUN sudo chmod +x /entrypoint.sh && \
       sudo chown -R $USERNAME /entrypoint.sh && \
       sudo chown -R $USERNAME $IMAGE_WS_DIR/$APP_NAME
       
   ENTRYPOINT ["/entrypoint.sh"]

2. You need to make one edit to the Dockerfile for this particular use case to place a copy of the example python script into your container. Add the following on line 119:

   COPY --chown=$USERNAME:$USERNAME $LOCAL_WS_DIR/src/rosbag-uploader-ros1/examples $IMAGE_WS_DIR/$APP_NAME/examples

3. Create a new file called entrypoint.sh in the same directory (~/environment).
   Note: An ENTRYPOINT file is an executable that will run when the docker container is spawned. You use an entry-point script to source the ROS workspace, so you can easily run roslaunch commands in the container.
4. Copy/paste the following into the new file:

   #!/bin/bash
   set -e
   source "/home/$USERNAME/workspace/$APP_NAME/setup.bash"
   printenv
   exec "${@:1}"

5. Create a new file docker-compose.yaml in the same directory (~/environment).
   Note: Docker compose is a tool for defining and running multi-container Docker applications. You can use Docker Compose to build the Docker image, then run the application container on the robot. To learn more about Docker compose, see Overview of Docker Compose.
   In the Docker Compose file, take note of the environment variables defined in the service definition. These variables are passed from AWS IoT Greengrass V2 and specify the required credentials. For more on configuring Docker containers with AWS IoT Greengrass V2, go to Run a Docker container.
6. Copy/paste the following into the new file (You will edit the bucket name later.)

   version: "3"
   services:
       uploader:
           build:
               context: ./
           image: rosbag-s3-app:latest
           network_mode: "host"
           command: bash -c "roslaunch rosbag_cloud_recorders rolling_recorder_sample.launch s3_bucket:=<MY_BUCKET_NAME> & (sleep 60 && python ~/workspace/gg-s3-app/examples/recorder_client.py rolling_recorder)"
           environment:
               - AWS_CONTAINER_AUTHORIZATION_TOKEN
               - AWS_CONTAINER_CREDENTIALS_FULL_URI

7. For this particular ROS package, open the file ros-workspace/src/rosbag-uploader-ros1/s3_file_uploader/config/node_sample_configuration.json and enter your AWS region.
8. Now, run the following commands to build the robot application container image:

   cd ~/environment
   DOCKER_BUILDKIT=1 docker build . \
   --build-arg ROS_DISTRO=melodic \
   --build-arg LOCAL_WS_DIR=./ros-workspace \
   --build-arg APP_NAME=gg-s3-app \
   -t rosbag-s3-app

9. Run the command docker images to confirm the docker image has been successfully built. The output should include a repository named rosbag-s3-app:

Congratulations! You have successfully built your Docker images and can now use them as AWS IoT Greengrass components to deploy your applications to your robots or run them in AWS RoboMaker for testing and validation with simulations.
Note: To use this approach with your own ROS application, run through the same process above and change the build arguments to match your application.

Publish to Amazon ECR

Containers can be stored in Amazon Elastic Container Registry (ECR), a fully-managed container registry. Upload your container image in the steps following:

1. Set four environment variables that can be reused in the next set of commands.
   Note: Remember to replace <YOUR AWS ACCOUNT NUMBER> and <YOUR AWS REGION> with your account number and region.
   

   export robotapp=rosbag-s3-app
   export account=<YOUR AWS ACCOUNT NUMBER>
   export region=<YOUR AWS REGION>
   export ecruri=$account.dkr.ecr.$region.amazonaws.com

2. Sign-in to Amazon ECR and create a new repositories:

   aws ecr get-login-password --region $region | docker login --username AWS --password-stdin $ecruri
   aws ecr create-repository --repository-name $robotapp

3. Tag Docker images with the Amazon ECR URI:

   docker tag $robotapp $ecruri/$robotapp:latest

4. Upload the Docker images to Amazon ECR:

   docker push $ecruri/$robotapp

5. Run the following describe statements to ensure your images are uploaded to ECR:
   aws ecr list-images --repository-name $robotapp
   Expected Output:

Congratulations! You have now pushed your ROS-based robot docker image to Amazon ECR.

Optional – Continue to create an AWS IoT Greengrass component from this ROS package:

Install and provision AWS IoT Greengrass V2

In order to create the cloud-based resources required to connect robots with AWS IoT Greengrass V2, the provisioning process will need elevated AWS permissions. We have packaged the required permissions along with pre-requisite AWS resources for deployment in an AWS CloudFormation template. You can look at the template here: greengrass/greengrass_bootstramp.template.yaml.

1. In your development environment Change directories:

   cd ~/environment

2. Run the following to download the AWS CloudFormation template:

   wget https://raw.githubusercontent.com/aws-samples/greengrass-v2-docker-ros-demo/main/greengrass/greengrass_bootstrap.template.yaml

3. Run the following AWS CloudFormation create stack command:

   aws cloudformation create-stack --stack-name GG-Provisioning --template-body file://greengrass_bootstrap.template.yaml --capabilities CAPABILITY_NAMED_IAM

4. Check the status of the CloudFormation stack with the following command. Wait a minute or two for it create the resources. Once done, the response will look similar to the following:

   aws cloudformation describe-stacks --stack-name GG-Provisioning

   

5. In a new browser window or tab, open the AWS IAM Console. Click Users to view the new user that was created from the previous steps. The name of the user follows this format: GG-Provisioning-GreengrassProvisioningUser-N12345678. Click on this user and open the security credentials tab.
   

6. Next, scroll to the Access keys section and select Create access key. A pop-up window will open with a new access key. Download or copy/paste these credentials to a secure location.
   

7. Create a new terminal tab in the IDE by pressing the green (+) symbol beside the open terminal.
   

8. Run the following setup commands in the new IDE terminal tab, replacing <INSERT_YOUR_AWS_ACCESS_KEY_ID_HERE> and <INSERT_YOUR_AWS_SECRET_KEY> with the credentials created preceding: (If you are following along in AWS Cloud9, you can skip installing the dependencies because OpenJDK is already installed.)

   # Install dependencies (Java JRE)
   sudo apt-get update
   sudo apt-get install default-jre -y 
   
   # Install and provision AWS IoT Greengrass V2
   export AWS_ACCESS_KEY_ID=INSERT_YOUR_AWS_ACCESS_KEY_ID_HERE
   export AWS_SECRET_ACCESS_KEY=INSERT_YOUR_AWS_SECRET_KEY
   
   curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip > greengrass-nucleus-latest.zip; unzip greengrass-nucleus-latest.zip -d GreengrassCore

9. Run the AWS IoT Greengrass V2 provisioning command:

   sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar \
   --thing-name ROS_IDE_1 \
   --thing-group-name ROS_IDE_Machines \
   --component-default-user ggc_user:ggc_group \
   --provision true \
   --setup-system-service true \
   --deploy-dev-tools true
   
   sudo usermod -aG docker ggc_user

10. Once finished, close the additional terminal by pressing “x” on the right corner of the terminal tab.

Create the sample application component in AWS IoT Greengrass V2

1. Create a new file named recipe.yaml in the directory with the name you gave your Development Environment. That is also the  ~/environment directory. Copy/paste the following into the recipe file:

   ---
   RecipeFormatVersion: '2020-01-25'
   ComponentName: com.s3.uploader.demo
   ComponentVersion: '1.0.0'
   ComponentDescription: 'A basic component that runs ROS s3 uploader cloud extension'
   ComponentPublisher: Amazon
   ComponentDependencies:
     aws.greengrass.DockerApplicationManager:
       VersionRequirement: ~2.0.0
     aws.greengrass.TokenExchangeService:
       VersionRequirement: ~2.0.0
   Manifests:
     - Platform:
         os: all
       Lifecycle:
           Install: |
              docker tag <YOUR_PRIVATE_ECR_IMAGE_ID_ROS_GREENGRASS_DEMO> robomaker-gg-s3-app:latest
           Run: |
              docker-compose -f {artifacts:path}/docker-compose.yaml up
       Artifacts:
         - URI: "docker:<YOUR_PRIVATE_ECR_IMAGE_ID_ROS_GREENGRASS_DEMO>"
         - URI: "s3://<YOUR_S3_BUCKET>/com.s3.uploader.demo/1.0.0/artifacts/docker-compose.yaml"

2. In the AWS Cloud9 IDE terminal, run the following command to retrieve the S3 bucket name created by the CloudFormation template:

   aws cloudformation describe-stacks --stack-name GG-Provisioning
   

   
   Note: If there is an “AccessDenied” error at this stage, it is likely because you are still using the terminal with the provisioning credentials preceding. Close this terminal and open a new one

3. Modify the recipe file by replacing <YOUR_PRIVATE_ECR_IMAGE_ID_ROS_GREENGRASS_DEMO> fields with the respective Amazon ECR Image IDs that you have created. These values should be similar to this: <ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/rosbag-s3-app then, update <YOUR_S3_BUCKET> with the S3 bucket name from the previous step and save the recipe file. You can find your ECR image ID in the ECR console. Navigate to repositories to find your ECR URI.
4. Open the docker-compose.yaml file and replace <MY_BUCKET_NAME> with the same Amazon S3 bucket using the object key defined in the recipe. Save the file.
5. Upload the docker compose file to the same Amazon S3 bucket using the object key defined in the recipe:

   cd ~/environment
   aws s3 cp ./docker-compose.yaml s3://<MY_BUCKET_NAME>/com.s3.uploader.demo/1.0.0/artifacts/docker-compose.yaml

6. Open the AWS IoT Greengrass V2 console in a new browser tab. On the left menu, click Components. Then, press Create Component.
7. Click Enter recipe as YAML, then copy/paste the recipe from preceding into the editor window.
8. Press Create component.

Congratulations! The Docker-based ROS sample application component has been successfully created in AWS IoT Greengrass V2.

Deploy the sample application component with AWS IoT Greengrass V2

Using AWS IoT Greengrass V2, software components and configurations can be deployed to individual robots or fleets of robots. The provisioning process preceding this created an IoT Thing Group named ROS_IDE_Machines to use as a deployment target and executed an initial deployment to download and install local developer tools. All robots added to the ROS_IDE_Machines group will receive the components defined in this deployment. To ensure consistency, AWS IoT Greengrass V2 overwrites previous deployments when deploying new components to targets that have existing deployments. Therefore, in this next section, you will revise the initial deployment to add the new ROS sample application component you created above. Once done, the ROS sample application will be deployed and running on the IDE host. To learn more about AWS IoT Greengrass V2 deployments, go to Deploy AWS IoT Greengrass components to devices.

1. Open the AWS IoT Greengrass console and click Deployments. Here, you will see the initial deployment that was created when the robot was provisioned.
2. Click the check box for Deployment for ROS_IDE_Machines and press Revise in the top-right corner.
3. In the wizard, the first step specifies deployment name and the deployment target (either a Greengrass Core Device or IoT Thing Group). When deployments are revised, the deployment target must stay the same. However, the name could be changed if the new set of components no longer aligns with the defined naming scheme. Leave these details as the default values. Press Next.
4. In the next step of the wizard, find the new sample application component (s3.uploader.demo) under My components. Then, click the check box beside it to include the new component in the revised deployment.
   Note: The dependency components aws.greengrass.DockerApplicationManager andaws.greengrass.TokenExchangeService do not need to be specified directly as all dependencies defined in the component recipe will be deployed automatically with the component.
5. In the third step of the wizard, you can customize the configuration (environment variables, component versions, system user, etc.) of the components in the deployment. For this tutorial, use the default settings. Click Next.
6. In the final step, you have the option to configure advanced deployment settings. To learn more about the different advanced settings, see Deploy AWS IoT Greengrass components to devices. Leave these options as default for now. Press  Next, then press Deploy to start the deployment.
7. After the deployment is complete, run the following commands in the AWS Cloud9 IDE with the Greengrass CLI to see if the ROS containers are running:

   cd /greengrass/v2/bin/
   sudo ./greengrass-cli component list

8. You will see a rosbag file uploaded to your S3 bucket in the folder rolling_recorder_test.

Congratulations! You have successfully dockerized and deployed a ROS application as a component to a robot using AWS IoT Greengrass V2 and uploaded a ROSbag to Amazon S3!

Clean-up

1. In the AWS Cloud9 Console, select your development environment, the choose Delete.
2. In the pop-up, confirm by typing Delete, then choose Delete.
3. In the AWS CloudFormation Console, select your stack name, GG-Provisioning…, then choose Delete, Delete stack.
4. In the Amazon ECR Console, select your repository, rosbag-s3-app, then choose Delete.
5. In the pop-up, confirm by typing delete, then choose
6. In the AWS IoT Core Console, choose All devices, Things, then select your thing, ROS_IDE_1.
7. Choose Delete. Follow the pop-up to confirm deletion of your IoT thing.
8. From the navigation menu, choose All devices, Thing groups, then select your thing group,

Summary

In this blog, you learned how to prepare ROS application Docker images for the cloud. Robot developers can benefit from using the cloud for building, training, and testing robotic applications at scale. You then walked through running the ROS Docker image as an AWS IoT Greengrass V2 component on a device for easy deployment. We look forward to hearing about the different ROS applications you are deploying onto your robots with AWS IoT Greengrass V2.

To find out more, visit AWS RoboMaker, AWS IoT Greengrass or contact AWS for further information.