AWS IoT Device Tester for Amazon FreeRTOS and AWS IoT Device Tester for AWS Greengrass

AWS IoT Device Tester is now available. AWS IoT Device Tester is a Windows/Linux/macOS test automation application that enables silicon vendors and OEMs to easily perform qualification testing to determine if their devices can run Amazon FreeRTOS or AWS IoT Greengrass and interoperate with AWS IoT services. AWS IoT Device Tester produces a test report that serves as qualification evidence for silicon vendors and OEMs who want to list their devices on the AWS Partner Device Catalog. There are two versions of AWS IoT Device Tester: AWS IoT Device Tester for Amazon FreeRTOS and AWS IoT Device Tester for AWS IoT Greengrass. In this blog post, I discuss both versions.

AWS IoT Device Tester for Amazon FreeRTOS

Amazon FreeRTOS provides connectivity and security libraries that make it possible for embedded developers, silicon vendors, and OEMs to develop easily cloud-enabled microcontroller-based devices like fitness trackers, sensors, smart power meters, and the like. Because these devices come with different connectivity (Wi-Fi, Ethernet, BLE) and security capabilities (for example, Secure Element), embedded developers implement device-specific Amazon FreeRTOS libraries. To confirm that their implementation is correct, embedded developers perform qualification tests, which are time-consuming and cumbersome. They have to manually run tests that involve modifying the source code to configure test case parameters, compiling and flashing of the test cases onto the microcontroller for execution. In addition, they have to learn about AWS IoT Core and set up cloud resources to perform end-to-end testing.

With AWS IoT Device Tester for Amazon FreeRTOS, it is easier for embedded developers to test if Amazon FreeRTOS works on their devices. AWS IoT Device Tester automatically executes the right set of tests based on device capabilities and then aggregates and stores the results on the computer where it is running. AWS IoT Device Tester for Amazon FreeRTOS also sets up the AWS resources required for end-to-end tests (such as MQTT, OTA) and automates compiling and flashing of test cases to the microcontroller by integrating with the microcontroller’s development toolchain.

AWS IoT Device Tester for Amazon FreeRTOS in action

To show you how AWS IoT Device Tester works, I will pretend to be an embedded developer and use it to test the STM32L4 Discovery Kit IoT Node. The capabilities of this kit include Wi-Fi connectivity and TCP/TLS processing. The embedded developers at STMicroelectronics saved the day by implementing the required Amazon FreeRTOS libraries (Wi-Fi and TLS/TCP) and making it available on the GitHub for anyone to use. (Other silicon vendors with qualified devices have their libraries in GitHub, too.) All I have to do now is to use AWS IoT Device Tester to test their implementation. My faithful Windows laptop works as my embedded development machine. It is connected to the STM32L4 kit through a USB cable.

 I downloaded Amazon FreeRTOS from GitHub to my laptop. It has all the required libraries, including the qualification tests. To get started, I simply downloaded Device Tester from here to my laptop. I then extracted the downloaded zip to the devicetester_afreertos_win folder and configured Device Tester to test the attached STM32L4 kit.

First, to provide AWS IoT Device Tester with my AWS account information, I edited the .\devicetester_afreertos_win\configs\config.json template file in the configs directory. AWS IoT Device Tester uses this information to set up AWS IoT Core cloud resources required for the tests (for example, creating things, downloading certificates, and so on). I provided my credentials through environment variables, so I set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables in Windows.

Second, to provide capabilities of the STM32L4 kit, I edited the features section of the device.json template file in the configs folder. Device Tester chooses the right set of tests to run based on the features I entered in the template file. I also provided the Windows port that the STM32L4 kit is connected to.

Lastly, to let AWS IoT Device Tester automate the process of building and flashing the test cases onto the STM32L4 kit for execution, I provided the build and flash scripts in the userdata.json template file in the configs folder. I also provided the path to the Amazon FreeRTOS source code on my laptop. If you are using microcontrollers from other vendors like Espressif, NXP, and Microchip, you can configure their build and flash commands in a similar way.Executing tests

AWS IoT Device Tester provides a command line interface to run test cases. On my power shell, I ran the following command from the bin folder to let AWS IoT Device Tester execute the required qualification test cases on STM32L4 kit.

AWS IoT Device Tester for Amazon FreeRTOS also provides an option to run a specific test group instead of the entire qualification test suite.

Viewing results
It took less than 30 mins for AWS IoT Device Tester to run the tests. I browsed to the .\results\<uid_of_test_run>\logs folder and opened the devicetester_report.xml to find the status of the tests. The report shows which tests were run and their status. The test report is in JUnit format. You might find this useful if you want to integrate AWS IoT Device Tester results into your existing CI/CD systems like Jenkins, Bamboo, or Hudson. My test report showed all tests passed. Bingo!

AWS IoT Device Tester for Amazon FreeRTOS is available for download now. To get started, visit here.

AWS IoT Device Tester for AWS IoT Greengrass

AWS IoT Greengrass is software that extends the benefits of AWS IoT Core to the edge. Linux devices that can run AWS IoT Greengrass vary widely in hardware configurations (for example, x86, ARMv7l, ARMv8), Linux versions (Ubuntu, Yocto, and customized Linux to run on appliances), and peripherals (such as Secure Element). OEMs who manufacture these devices wanted an easy and scalable way to verify if AWS IoT Greengrass works on their choice of Linux device.

Like AWS IoT Device Tester for Amazon FreeRTOS, AWS IoT Device Tester for AWS IoT Greengrass is designed to run a set of standardized, automated tests to verify if the Linux device can run AWS IoT Greengrass and interoperate with AWS IoT services. Device Tester for AWS IoT Greengrass sets up the resources required by test cases. This includes creating a Greengrass group, certificates for authentication with AWS IoT Core, and Lambda functions. The Device Tester for AWS IoT Greengrass not only checks if the device hardware and OS meet Greengrass requirements, it also checks if key functionality (such as local resource access, device shadow, and OTA) work correctly on the device.

AWS IoT Device Tester for AWS IoT Greengrass in action

Logic Supply shipped me one of their industrial computers, the ML100G-50, to test whether AWS IoT Greengrass works on the device. The ML100G-50 has a 6th generation “Skylake” Intel Core i5-6300U CPU with 4 GB RAM and 128 GB SSD storage. AWS IoT Greengrass v1.6 was already installed on the device.

After I connected the device to my network, I downloaded Device Tester from here to my Windows laptop and then extracted it to the devicetester_greengrass_win folder. I provided Device Tester with my AWS account and AWS Region information. The mechanics of this step are identical to what I did in the first configuration step for Device Tester for Amazon FreeRTOS.

Second, to enable passwordless SSH access to the Logic Supply device for the Device Tester, I copied the public key of my Windows laptop to the device. Now with Linux utilities built in into Windows 10, I used ssh-keygen from bash shell to generate private/public keys and then used ssh-copy-id to copy the public key over to the Logic Supply device.

Lastly, I edited the devices.json template file in the .\devicetester_greengrass_win\configs folder to provide device CPU architecture, OS, Greengrass root folder location on the device, and connectivity information. The Logic Supply device uses Ubuntu 16.04 OS with x86_64 architecture. Device Tester requires root access on the device under test, so I created a user with root privileges on the device and added the user name in the devices.json file.

Executing tests

Like the Amazon FreeRTOS version, the AWS IoT Device Tester for AWS IoT Greengrass provides a command line interface to run test cases. In PowerShell, I ran the following command from the .\devicetester_greengrass_win\bin folder to let AWS IoT Device Tester execute the qualification test cases on ML100G-50. By default, it runs all of the tests in the Greengrass qualification suite.AWS IoT Device Tester for AWS IoT Greengrass also provides an option to run a specific test group instead of the entire qualification test suite.

Viewing results

It took about 45 mins for AWS IoT Device Tester to run the tests. I browsed to the .\devicetester_greengrass_win\results\GGQ\ folder and opened the devicetester_report.xml to find the status of the tests. The report shows which tests were run and the status. My test report showed that all tests passed. Hurray!AWS IoT Device Tester for AWS IoT Greengrass is available for download now. To get started, visit here. Happy testing!