Perform OTA Updates on Espressif ESP32 using Amazon FreeRTOS Bluetooth Low Energy

Amazon FreeRTOS 2019.06.00 Major now includes Bluetooth Low Energy MQTT proxy support to simplify tasks such as Wi-Fi provisioning and secure connections to AWS IoT services. The Bluetooth Low Energy feature enables you to build low-power devices that can be paired to a customer’s mobile device for connectivity without requiring Wi-Fi. Devices can communicate using MQTT by connecting through Android or iOS Bluetooth Low Energy SDKs that use generic access profile (GAP) and generic attributes (GATT) profiles. You can find details in the blog – Announcing support for Bluetooth Low Energy in Amazon FreeRTOS.

Amazon FreeRTOS already gives you the flexibility to perform over-the-air (OTA) firmware updates. OTA allows you to change the functionality of devices and provide security fixes without user intervention. AWS IoT Device Management allows secure and scalable OTA updates as well as monitoring and reporting tools to simplify the update process. The Amazon FreeRTOS device OTA agent validates the authenticity of the update using X.509 certificates. If an issue is found, it rolls back to the previously functioning firmware.

OTA updates

In this post, I walk you through an update to Espressif ESP32 microcontroller connected to an MQTT Bluetooth Low Energy proxy on an Android device. You update the device using AWS IoT OTA update jobs. The device connects to AWS IoT using Amazon Cognito credentials entered in the Android demo app. An authorized operator initiates the OTA update from the cloud. When the device connects through the Android demo app, the OTA update is initiated and firmware is updated on the device.

Here are the steps to allow OTA updates over Bluetooth Low Energy:

Configure storage: Create an S3 bucket and policies and configure an IAM user that can perform updates.
Create a code-signing certificate: Create a signing certificate and allow the IAM user to sign firmware updates.
Configure Amazon Cognito authentication: Create a credential provider, user pool, and application access to the user pool.
Configure Amazon FreeRTOS: Set up Bluetooth Low Energy, client credentials, and the code-signing public certificate.
Configure an Android app: Set up credential provider, user pool, and deploy the application to an Android device.
Run the OTA update script: Use the OTA update script to initiate an OTA update.

For more information about how the updates work, see Amazon FreeRTOS Over-the-Air Updates. For general information, see the Amazon FreeRTOS User Guide.

For information about how to set up the Bluetooth Low Energy MQTT proxy functionality, see the Using Bluetooth Low Energy with Amazon FreeRTOS on Espressif ESP32 post by Richard Kang.

Prerequisites

To follow along with this solution, you need the following resources:

An ESP32 development board
A MicroUSB to USB A cable
An AWS account (the Free Tier is sufficient)
Sufficient disk space (~500 Mb) for the Xtensa toolchain and Amazon FreeRTOS source code and examples.
An Android phone with Android v 6.0 or later and Bluetooth version 4.2 or later.
Android Studio
The AWS CLI installed
Python3 installed
The boto3 AWS Software Developer Kit (SDK) for Python

This post is written with the assumption that Xtensa toolchain, ESP-IDF, and Amazon FreeRTOS code are installed in the /esp directory in the user’s home directory. You must add ~/esp/xtensa-esp32-elf/bin to your $PATH variable.

Step 1: Configure storage

You can skip these steps by launching the AWS CloudFormation template.

Create an S3 bucket with versioning enabled to hold the firmware images.
Create an OTA update service role and add the following managed policies to the role:
- AWSIotLogging
- AWSIotRuleActions
- AWSIotThingsRegistration
- AWSFreeRTOSOTAUpdate
Add an inline policy to the role to allow it to perform actions on the IoT service and allow access to the S3 bucket that you created.
Create an IAM user that can perform OTA updates. This user can sign and deploy firmware updates to IoT devices in the account, and has access to do OTA updates on all devices. Access should be limited to trusted entities.
Attach permissions with an OTA user policy.
Create an inline policy that allows the IAM user created to sign the firmware.

Step 2: Create the code-signing certificate

Create a code-signing certificate that can be used to sign the firmware. Note the certificate ARN when the certificate is imported.

aws acm import-certificate --profile=ota-update-user --certificate file://ecdsasigner.crt --private-key file://ecdsasigner.key
{
"CertificateArn": "arn:aws:acm:us-east-1:<account>:certificate/<certid>"
}

The ARN will be used later to create a signing profile. If desired, the profile can be created using the following command at this point:

aws signer put-signing-profile --profile=ota-update-user --profile-name esp32Profile --signing-material certificateArn=arn:aws:acm:us-east-1:<account>:certificate/<certid> --platform AmazonFreeRTOS-Default --signing-parameters certname=/cert.pem
{
"arn": "arn:aws:signer::<account>:/signing-profiles/esp32Profile"
}

Step 3: Cognito Authentication Configuration

Follow Steps 1 and 2 in the AWS IoT Configuration section here.
Next, follow Steps 1 through 3 in the Amazon Cognito Configuration section here.

Step 4: Configure Amazon FreeRTOS

To download the latest version of the Amazon FreeRTOS code, use the Amazon FreeRTOS console or Amazon FreeRTOS GitHub repo.
To enable the OTA update demo, follow the steps in the Getting Started with the Espressif ESP32-DevKitC and the ESP-WROVER-KIT Only one demo can be enabled at a time in the Demo Runner.

Additional modifications to be made, by location:

vendors/espressif/boards/esp32/aws_demos/config_files/aws_demo_config.h
- Define CONFIG_OTA_UPDATE_DEMO_ENABLED.
vendors/espressif/boards/esp32/aws_demos/common/config_files/aws_demo_config.h:
- Change democonfigNETWORK_TYPES to AWSIOT_NETWORK_TYPE_BLE.
demos/include/aws_clientcredential.h:
- Adjust the endpoint URL in clientcredentialMQTT_BROKER_ENDPOINT[].
- Adjust the thing name to "esp32-ble" in clientcredentialIOT_THING_NAME.
- Certificates don’t have to be added when you use Amazon Cognito credentials.
vendors/espressif/boards/esp32/aws_demos/common/config_files/aws_iot_network_config.h
- AdjustconfigSUPPORTED_NETWORKS and configENABLED_NETWORKS to only includeAWSIOT_NETWORK_TYPE_BLE.
demos/include/aws_ota_codesigner_certificate.h:
- Adjust signingcredentialSIGNING_CERTIFICATE_PEM with the certificate to be used to sign the firmware binary file.

The application should start up and print the demo version:

11 13498 [iot_thread] [INFO ][DEMO][134980] Successfully initialized the demo. Network type for the demo: 2
12 13498 [iot_thread] [INFO ][MQTT][134980] MQTT library successfully initialized.
13 13498 [iot_thread] OTA demo version 0.9.20
14 13498 [iot_thread] Creating MQTT Client...

Step 5: Configure an Android app

Download the Android Bluetooth Low Energy SDK and a sample app from the amazon-freertos-ble-android-sdk GitHub repo.

The following sections explain the changes to be made.

Modify awsconfiguration.json

This file is located at:

app/src/main/res/raw/awsconfiguration.json

Fill in Pool Id, Region, AppClientId, and AppClientSecret using the instructions in the following sample JSON.

{
  "UserAgent": "MobileHub/1.0",
  "Version": "1.0",
  "CredentialsProvider": {
    "CognitoIdentity": {
      "Default": {
        "PoolId": "Cognito->Manage Identity Pools->Federated Identities->mqtt_proxy_identity_pool->Edit Identity Pool->Identity Pool ID",
        "Region": "Your region (eg. us-east-1)"
      }
    }
  },

  "IdentityManager": {
    "Default": {}
  },

  "CognitoUserPool": {
    "Default": {
      "PoolId": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> PoolId",
      "AppClientId": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> App clients ->Show Details",
      "AppClientSecret": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> App clients ->Show Details",
      "Region": "Your region (eg. us-east-1)"
    }
  }
}

Modify DemoConstants.java

This file is located at:

app/src/main/java/software/amazon/freertos/DemoConstants.java

Specify the policy name (e.g.esp32_mqtt_proxy_iot_policy) created earlier.
Set the Region (e.g. us-east-1).

Build and install the demo app

Follow these steps:

In Android Studio, choose Build, Make Module app.
Choose Run, Run app. You can go to the logcat window pane in Android Studio to monitor log messages.
On the Android device, create an account from the login screen.
Create a user. If a user already exists, enter the credentials.
Allow AmazonFreeRTOS Demo to access the device’s location.
Scan for Bluetooth Low Energy devices,
Move the slider for the device found to the On
Press ‘y’ on the serial port debug console for the ESP32.
Choose Pair & Connect.

The More… link becomes active after the connection. You see the connection state change to BLE_CONNECTED in the Android device logcat when the connection is complete:

2019-06-06 20:11:32.160 23484-23497/software.amazon.freertos.demo I/FRD: BLE connection state changed: 0; new state: BLE_CONNECTED

Before the messages can be transmitted, the Amazon FreeRTOS device and the Android device negotiate the MTU. You should see the following print in logcat:

2019-06-06 20:11:46.720 23484-23497/software.amazon.freertos.demo I/FRD: onMTUChanged : 512 status: Success

The device is connected to the app and starts sending MQTT messages using the MQTT proxy. To confirm that the device can communicate, make sure that you can see the MQTT_CONTROL characteristic data value change to 01:

2019-06-06 20:12:28.752 23484-23496/software.amazon.freertos.demo D/FRD: <-<-<- Writing to characteristic: MQTT_CONTROL with data: 01
2019-06-06 20:12:28.839 23484-23496/software.amazon.freertos.demo D/FRD: onCharacteristicWrite for: MQTT_CONTROL; status: Success; value: 01

Enable BLE on the ESP32 console

You are asked to press ‘y‘ on the ESP32 console when the device is paired with the device. The demo does not function until this step is performed.

E (135538) BT_GATT: GATT_INSUF_AUTHENTICATION: MITM Required
W (135638) BT_L2CAP: l2cble_start_conn_update, the last connection update command still pending.
E (135908) BT_SMP: Value for numeric comparison = 391840
15 13588 [InputTask] Numeric comparison:391840
16 13589 [InputTask] Press 'y' to confirm
17 14078 [InputTask] Key accepted
W (146348) BT_SMP: FOR LE SC LTK IS USED INSTEAD OF STK
18 16298 [iot_thread] Connecting to broker...
19 16298 [iot_thread] [INFO ][MQTT][162980] Establishing new MQTT connection.
20 16298 [iot_thread] [INFO ][MQTT][162980] (MQTT connection 0x3ffd5754, CONNECT operation 0x3ffd586c) Waiting for operation completion.
21 16446 [iot_thread] [INFO ][MQTT][164450] (MQTT connection 0x3ffd5754, CONNECT operation 0x3ffd586c) Wait complete with result SUCCESS.
22 16446 [iot_thread] [INFO ][MQTT][164460] New MQTT connection 0x3ffc0ccc established.
23 16446 [iot_thread] Connected to broker.

Step 6: Run the OTA update script

To install the prerequisites, run the following commands:
```
pip3 install boto3
pip3 install pathlib
```
Download the python script.
Bump up the Amazon FreeRTOS application version in demos/include/aws_application_version.h and build a new .bin file.

To get help, run the following command in a terminal window:

python3 start_ota.py -h

Here’s the help information:

usage: start_ota.py [-h] --profile PROFILE [--region REGION]
                    [--account ACCOUNT] [--devicetype DEVICETYPE] --name NAME
                    --role ROLE --s3bucket S3BUCKET --otasigningprofile
                    OTASIGNINGPROFILE --signingcertificateid
                    SIGNINGCERTIFICATEID [--codelocation CODELOCATION]
Script to start OTA update
optional arguments:
-h, --help            show this help message and exit
--profile PROFILE     Profile name created using aws configure
--region REGION       Region
--account ACCOUNT     Account ID
--devicetype DEVICETYPE thing|group
--name NAME           Name of thing/group
--role ROLE           Role for OTA updates
--s3bucket S3BUCKET   S3 bucket to store firmware updates
--otasigningprofile OTASIGNINGPROFILE
                      Signing profile to be created or used
--signingcertificateid SIGNINGCERTIFICATEID
                      certificate id (not arn) to be used
--codelocation CODELOCATION
                      base folder location (can be relative)

If you used the provided AWS CloudFormation template to create resources, here’s the example execution:

python3 start_ota_stream.py --profile otausercf --name esp32-ble --role ota_ble_iot_role-sample --s3bucket afr-ble-ota-update-bucket-sample --otasigningprofile abcd --signingcertificateid <certificateid>

You should see the update start in the ESP32 debug console:

38 2462 [OTA Task] [prvParseJobDoc] Job was accepted. Attempting to start transfer.
---
49 2867 [OTA Task] [prvIngestDataBlock] Received file block 1, size 1024
50 2867 [OTA Task] [prvIngestDataBlock] Remaining: 1290
51 2894 [OTA Task] [prvIngestDataBlock] Received file block 2, size 1024
52 2894 [OTA Task] [prvIngestDataBlock] Remaining: 1289
53 2921 [OTA Task] [prvIngestDataBlock] Received file block 3, size 1024
54 2921 [OTA Task] [prvIngestDataBlock] Remaining: 1288
55 2952 [OTA Task] [prvIngestDataBlock] Received file block 4, size 1024
56 2953 [OTA Task] [prvIngestDataBlock] Remaining: 1287
57 2959 [iot_thread] State: Active  Received: 5   Queued: 5   Processed: 5   Dropped: 0

When the OTA update is complete, the device restarts as needed by the OTA update process and try to connect with the updated firmware. If the upgrade succeeds, the updated firmware is marked as active and you should see the updated version in the console:

13 13498 [iot_thread] OTA demo version 0.9.21

Conclusion

In this post, I described how to perform OTA updates over Bluetooth Low Energy for devices that support the Amazon FreeRTOS Bluetooth Low Energy MQTT proxy. I explained how to set up storage, certificates for signing, Amazon Cognito for authentication, Amazon FreeRTOS on the device, and a Bluetooth Low Energy app on an Android phone. Finally, I explained how to update the device using a Python script.

The Internet of Things on AWS – Official Blog