Hardware and operating systems for edge devices are built and delivered to customers in a wide range of form factors, such as consumer gateways, industrial controllers, and in-vehicle infotainment systems.

Original design manufacturers (ODMs) and original equipment manufacturers (OEMs) building commercial off-the-shelf products decide which hardware and software combinations to use. System integrators (SIs) developing broad solutions and customers creating customized hardware solutions also design edge solutions. Depending on the target use case, they may decide to use a specific type of Embedded Linux distribution that provides core capabilities to benefit system operation and the lifespan of the device itself.

In this post, I explore an option for ODMs, OEMs, SIs, and customers to deliver on these requirements when using AWS IoT Greengrass.

Overview

Today, AWS announced OpenWrt support for AWS IoT Greengrass for the ARMv7l and ARMv8 architectures. AWS IoT Greengrass seamlessly extends AWS to edge devices so they can act locally on the data that they generate, while still using the cloud for management, analytics, and durable storage.

When working on solutions with customers, I found that OpenWrt can be an ideal operating system to help narrow the system footprint and gives strong options for running wear-leveling file systems.

Previously, to run AWS IoT Greengrass, you installed the glibc C runtime package, causing additional footprint. Today, using OpenWrt, you can design the operating system for AWS IoT Greengrass from the beginning to minimize footprint.

Customers, partners, and SIs now have an option to choose from smaller C runtime libraries for their OpenWrt image, completely eliminating the libc dependencies for AWS IoT Greengrass without increasing its footprint.

These improvements enable customers to deliver AWS Lambda functions and Amazon SageMaker Neo-compiled machine learning (ML) models (100x smaller footprint) with AWS IoT Greengrass, creating a smaller footprint IoT solution that is optimal for the resource-constrained OpenWrt-based gateway devices.

The AWS IoT Greengrass packages for OpenWrt are now available for download.

Optimized for ML at the edge

Many of our customers and partners are building compelling and innovative solutions on AWS and use AWS IoT Greengrass as part of their architecture.

The example described here is from one of our Advanced Tier AWS Partners, Smartiply, Inc. Smartiply has integrated AWS IoT Greengrass with the Smartiply IoT Gateway to deliver advanced AWS capabilities at the edge to enable new use cases and opportunities for two mutual customers.

The Smartiply IoT Gateway accelerates the adoption of IoT in environments that are challenged in infrastructure, or challenged geographically and economically. Target use cases are video surveillance, enterprise security, video streaming, and smart city. The core capabilities of the gateway include the following:

• Smart connectivity to deliver an uninterrupted video experience
• Embedded intelligence (video, voice, data, and sensor analytics) in the fog for local decision-making
• Data thinning to optimize traffic to the cloud
• Sensor integration for monitoring
• Remote device management

Beyond smart connectivity and edge intelligence, Smartiply’s customers want to perform object classification inferencing at the edge, driven by AWS IoT Greengrass. Smartiply’s device runs OpenWrt to optimize the device lifespan and operations. Smartiply has decided to use AWS IoT Greengrass to provide its customers seamless and secure IoT connectivity and more effective edge-to-cloud computing. AWS IoT Greengrass enables customers to continuously extend the value of their devices through programmatic extensions using Lambda functions and ML inference.

A mutual customer in India is in the e-surveillance space. They most recently also received the Frost Sullivan award for technology leadership and for catering to the most diverse set of customers in India.

Together, we enable their customers to diversify and enter into new markets beyond bank ATM monitoring. These include environments that require higher-order capabilities to fulfill advanced needs, such as retail, logistics, and construction. Using the Smartiply IoT Gateway and building the solution on the AWS Cloud, our mutual customer uses IoT to deliver smart security solutions that include basic and advanced video analytics.

There is a camera in every location, primarily intended for security purposes. However, using advanced capabilities of AWS (specifically Amazon Rekognition) through AWS IoT Greengrass, Smartiply can deliver innovative computer vision solutions rapidly and efficiently through the edge-to-cloud framework.

Value-added microservices could include any of the following:

• Detecting whether an employee in a fast-food establishment is wearing an apron or cap for compliance.
• Identifying disturbing behavior (fights) in public facilities.
• Calculating average time to bill at a restaurant.
• Monitoring table occupancy.
• Improving queue management at POS workstations.

All of these open up new revenue opportunities for Smartiply and improve operational efficiency.

“The beauty of partnering with Amazon and integrating with AWS IoT Greengrass is that all stars get aligned perfectly. We don’t have to reinvent the wheel, and we get to deliver solutions rapidly and most efficiently, starting from the edge itself. The hybrid model of enabling basic capabilities right at the edge but accessing the cloud for advanced computing is a perfect formula for many mission-critical applications. It not only saves costs but also improves performance, helping accelerate ROI that is direly needed across IoT deployments.” – Kaushik Pillalamarri, cofounder and CEO of Smartiply, Inc.

Another mutual customer, one of the largest local governments in Mexico, needs the Smartiply IoT Gateway to combine the ML capabilities of the AWS Cloud with video recognition for a smart city. The capability for the device to perform person and object recognition at the edge that continuously improves through automated, AWS Cloud-driven model training, and testing provides a powerful solution.

Building system requirements

When you build an Embedded Linux system—whether it’s Yocto, Buildroot, or OpenWrt—all the system requirements are built into the image.

Normally, there is no package manager tooling available on production, such as apt, yum, or opkg, except when building developer images. This means that, beyond kernel configuration requirements, all dependent packages for your target use case are prebuilt into the distribution image. Updates to packages mean constructing a firmware upgrade deliverable for customers.

It is out of scope for this post to have you construct a custom package, so install opkg and manually install AWS IoT Greengrass. Look for a future blog post about how to custom package AWS IoT Greengrass to OpenWrt.

Launch an Amazon EC2 instance

If you have a build machine for Embedded Linux, you may choose to use that machine. Make sure that all the system build dependencies are met.

If you don’t have a build machine, launch an Amazon EC2 instance to build OpenWrt, and one that meets AWS IoT Greengrass container mode requirements. Follow the directions in the Launch instance wizard, but specify the following:

• On the Choose an Amazon Machine Image (AMI) page, choose Public images and filter for Debian. Choose the latest Stretch distribution.
• To build faster, choose a larger instance.
• Change the storage allocation to 120 GB.

To connect to your instance, run the following command while logged into your build system:

ssh -i ~/.ssh/xxx.pem admin@xx.xxx.xxx.x

Many of the following steps come directly from the OpenWrt Project Quick Image Building Guide. After you connect to the EC2 instance, run the following commands to install the dependencies:

sudo apt-get update
sudo apt-get -y dist-upgrade
sudo apt-get install -y subversion g++ zlib1g-dev build-essential git python time
sudo apt-get install -y libncurses5-dev gawk gettext unzip file libssl-dev wget
sudo apt-get install -y libelf-dev

Then, check out the OpenWrt sources.

git clone https://git.openwrt.org/openwrt/openwrt.git/
cd openwrt

./scripts/feeds update -a
./scripts/feeds install -a

Run the following command to enter the menu system:

make menuconfig

Choosing the correct board

Follow these steps to choose the correct board.

1. Select Target System, scroll to Broadcom BCM27xx, and choose Enter.
2. Scroll to Subtarget and choose Enter.
3. Because this runs on a Raspberry Pi 3 Model B, scroll to BCM2710, and choose Enter.

Install utilities

Follow these steps to install several utilities and the ca-certificates package.

1. Install the lsblk utility, used to list disks and partitions on the device:
   1. Select Utilities and press Enter.
   2. Select Disc and press Enter.
   3. Select lsblk and enter Y.
   4. Press Tab to select Exit and press Enter.
2. Install the useradd and groupadd utilities, used to add the required AWS IoT Greengrass user and group. In production environments, automate adding the user and group to /etc/passwd and /etc/group, respectively.
   1. Select Base system and press Enter.
   2. Select busybox and press Enter.
   3. Select Customize busybox options and press Y.
   4. Select Login/Password Management Utilities and press Enter.
   5. For each of the addgroups, support adding users to groups, adduser, mkpasswd, and chpasswd, press Y.
   6. Press Tab to select Exit and press Enter (three times).
3. Install the ca-certificates package, because it is required for AWS IoT Greengrass to perform the initial TLS 1.2 handshake with the AWS IoT Core gateway:
   1. Select Base system and press Enter.
   2. Select ca-certificates and press Y.
   3. Press Tab to select Exit and press Enter.
4. Press Tab three times to select Save and press Enter. Press Enter again to save the configuration.
5. Press Tab to select Exit and press Enter.

Create a file named LXC_PATCH in the current working directory. Copy and paste the following code block:

cat <<OUT >> .config
CONFIG_KERNEL_BLK_CGROUP=y
CONFIG_KERNEL_CC_STACKPROTECTOR_REGULAR=y
CONFIG_KERNEL_CFQ_GROUP_IOSCHED=y
CONFIG_KERNEL_CGROUPS=y
CONFIG_KERNEL_CGROUP_CPUACCT=y
CONFIG_KERNEL_CGROUP_DEVICE=y
CONFIG_KERNEL_CGROUP_FREEZER=y
CONFIG_KERNEL_CGROUP_SCHED=y
CONFIG_KERNEL_CPUSETS=y
CONFIG_KERNEL_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_KERNEL_FREEZER=y
CONFIG_KERNEL_IOSCHED_DEADLINE=m
CONFIG_KERNEL_IPC_NS=y
CONFIG_KERNEL_LXC_MISC=y
CONFIG_KERNEL_MEMCG=y
CONFIG_KERNEL_MEMCG_SWAP=y
CONFIG_KERNEL_MM_OWNER=y
CONFIG_KERNEL_NAMESPACES=y
CONFIG_KERNEL_NETPRIO_CGROUP=y
CONFIG_KERNEL_NET_CLS_CGROUP=y
CONFIG_KERNEL_NET_NS=y
CONFIG_KERNEL_PID_NS=y
CONFIG_KERNEL_POSIX_MQUEUE=y
CONFIG_KERNEL_RESOURCE_COUNTERS=y
CONFIG_KERNEL_USER_NS=y
CONFIG_KERNEL_UTS_NS=y
CONFIG_KERNEL_SECCOMP_FILTER=y
CONFIG_KERNEL_SECCOMP=y
OUT

Finishing the configuration and compiling

Now that the key changes have been set for the device, you can fill in the remaining baseline configuration by running the defconfig target:

make defconfig

To compile the system, run the following command. Consider how many cores you instantiated for your EC2 instance, add one to it, and apply it to the make command:

make -j X

If you chose a C4.8xlarge distribution earlier, this number would be 37:

make -j 37

If the build breaks, try continuing with X=1, because make dependencies may be broken. In my experience, running at that parallelism under the configuration defined in this post did not break the build. However, if you added other packages, it might happen.

make -j 1

When I built using a C4.8xlarge EC2 instance, the build took less than 14 minutes.

After the system is built, download the image to your workstation using scp.  The images are under the directory:

~/openwrt/bin/targets/brcm2708/bcm2710

Enter the command for downloading:

scp -i ~/.ssh/xxx.pem admin@xx.xxx.xxx.xx:/home/admin/openwrt/bin/targets/brcm2708/bcm2710/openwrt-brcm2708-bcm2710-rpi-3-ext4-factory.img.gz .

The EC2 instance you used to build the OpenWrt system can be paused or deleted to save EC2 billing costs.

Flashing the image to the microSD card

Next, flash your microSD card with the image. This is similar to the process described in the previous section.

sudo dd if=./openwrt-brcm2708-bcm2710-rpi-3-ext4-factory.img of=/dev/disk3 bs=2048
sync
diskutil unmountdisk /dev/disk3

Remove the microSD card from your workstation, insert the microSD to your Raspberry Pi, and power it on.

Configuring the network

For this post, use ethernet connectivity. Edit /etc/config/network to change eth0 to use DHCP. After editing, your network should look similar to the following:

config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd77:891b:69b8::/48'

config interface 'lan'
option type 'bridge'
option ifname 'eth0'
option proto 'dhcp'

After editing the network, reload the network configuration to put the changes into effect:

service network reload

Configure user and group for AWS IoT Greengrass

Add the users to the system:

addgroup ggc_group
adduser -DH -G ggc_group ggc_user

Modify the boot parameter in /boot/cmdline.txt to mount the memory cgroup. Enter vi /boot/cmdline.txt, select the End key, select a, and then enter the following text:

cgroup_enable=memory cgroup_memory=1

Press Esc, then enter :wq! to write the buffer to file and exit vi. Reboot the system by entering reboot and choosing Enter.

Creating your AWS IoT Greengrass group

The next step is to create your Greengrass group. Follow the directions in the Easy group creation wizard, but specify the following:

1. Pick a name for your group, such as openwrt-containermode.
2. Under Choose a root CA, save the Amazon Root CA 1 to the same directory where you saved the tar.gz downloads.
3. Under Choose your platform, download the OpenWrt distribution for Aarch64 to the same known directory.

Starting AWS IoT Greengrass on your device

Now you get to start AWS IoT Greengrass on your physical device.

1. Insert a memory stick and copy all three files to your memory stick:
   • The group configuration tar.gz
   • The AWS IoT Greengrass software tar.gz
   • The root CA
2. Unmount the memory stick and insert it to the Raspberry Pi.
3. To mount the memory stick, switch back to your serial connection window and run the following commands:

lsblk # identifies the disk partition for your memory stick
mkdir /media
mount /dev/sda1 /media

Unpack AWS IoT Greengrass and its configuration to the filesystem.

cd /
tar xzvf /media/[name of greengrass distribution file]
cd greengrass
tar xzvf /media/[name of greengrass configuration file]
cp /media/AmazonRootCA1.pem certs/root.ca.pem

Start AWS IoT Greengrass.

/greengrass/ggc/core/greengrassd start

Deployment

Now you get to deploy the Greengrass group.  AWS IoT Greengrass is now running with your first deployment to an OpenWrt system.

Next, you can begin working through tutorials, such as:

• Lambda Functions on AWS IoT Greengrass Part 1
• Lambda Functions on AWS IoT Greengrass Part 2
• Interacting with Devices in an AWS IoT Greengrass Group
• Interacting with Device Shadows
• Accessing Other AWS Services

You might also want to consider learning more about Hardware Security Integration when designing an AWS IoT Greengrass system that runs production workloads like finished consumer products.

Conclusion

Depending on why you are building a solution on AWS, you may find it beneficial to run AWS IoT Greengrass on the OpenWrt operating system. From basic telematics to performing ML inference, our customers are building AWS IoT Greengrass solutions that innovate and extend the solution value over time.

In this post, you learned how to build the OpenWrt system image that supports AWS IoT Greengrass. Now you can evaluate whether the OpenWrt operating system improves your capabilities, and use AWS IoT Greengrass to continuously improve your edge capabilities.