How to perform secondary processor over-the-air updates with FreeRTOS

Many embedded architectures include a connectivity processor connected to one or more secondary processors that perform business logic. The ability to perform secondary processor over-the-air (OTA) updates is just as critical as updating the connectivity processor. This is because it allows for low-cost patching of bugs and security vulnerabilities as well as delivering new features to the device.

FreeRTOS is an open source, real-time operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage. AWS IoT Device Management makes it easy to securely register, organize, monitor, and remotely manage IoT devices at scale. AWS IoT Device Management provides an OTA Update Manager service to securely create and manage updates across a fleet of devices. The service works with the FreeRTOS OTA agent library by digitally signing the firmware, converting the file into an MQTT stream using the streaming API, and delivering the firmware to the device using AWS IoT jobs. The OTA agent library allows the reuse of an MQTT connection over TLS to reduce memory consumption on the connectivity processor.

In this post, we show you how to use the fileId parameter to deliver updates to the secondary processor. This post is not specific to any hardware and can be adapted to any system that runs FreeRTOS 201908.00 or later. For additional details on how to set up OTA using FreeRTOS and AWS IoT Device Management, refer to the FreeRTOS OTA tutorial.

Reference architecture overview

The following architecture diagram describes the flow of the secondary processor update from AWS IoT through a connectivity processor. The device runs FreeRTOS on the connectivity processor and has a serial interface such as SPI connected to the secondary processor. An authorized operator securely uploads the firmware to an Amazon S3 bucket and initiates the OTA update. The firmware file is signed using a digital certificate and a stream is created. An AWS IoT job is then created to send the firmware update to the device. The device’s connectivity processor identifies that the firmware update is one destined for the secondary processor and sends the update over the serial interface.

Make firmware changes

FreeRTOS includes code that demonstrates how to perform OTA updates. You can find out details about how the demo works in the OTA updates demo application documentation. You can also download changes to the OTA demo from this code link and follow along after applying the patch.

OTA for a processor is typically handled by a Microcontroller Unit (MCU) vendor using guidelines listed in the OTA porting guide. The MCU vendor implements functions in the Platform Abstraction Layer (PAL) to perform the update. The patch prevents you from disrupting the firmware updates to the connectivity processor and overriding the behavior for secondary processor updates. The patch file previously provided lets you accomplish the following:

1. Provide function overrides to the PAL layer. If the firmware is intended for the connectivity processor, the vendor-supplied PAL functions are called. Otherwise, the function allows you to send the update to the secondary processor using the overrides. These functions have been left empty so that hardware-specific transfers can be performed as needed by your platform.
2. Override the PAL layer to call the function overrides using the internal OTA agent initialization function:

OTA_AgentInit_internal(xConnection.xMqttConnection, (const uint8_t *)(clientcredentialIOT_THING_NAME), &otaCallbacks, (TickType_t)~0);

Here are some items that you might need to adjust in the patch for your application:

• Initialize communication with the secondary processor before initializing the OTA agent.
• Check the file ID in the code to identify which processor is being updated. This file ID must match the ID sent in the script described in the next section. When an OTA update is created from the console, the file ID sent down is 0. Do not use 0 for any secondary processor updates.
• Ensure that each of the callbacks return the appropriate error. For example, in the prvPAL_CreateFileForRx_customer callback, you might want to put the secondary processor into a known state to start receiving updates. If the state change fails, the callback should return an error.
• Ensure that you return back eOTA_PAL_ImageState_Valid as the current platform image state on startup and eOTA_PAL_ImageState_PendingCommit when the platform state is set to eOTA_ImageState_Testing by the OTA agent state machine.
• Ensure that the secondary processor is updated by checking the version number in the self-test routine. There is no explicit check done by the OTA state machine to ensure that the secondary processor updated.

To ensure that you are running the demo correctly, be sure to make the following changes:

1. Configure your AWS environment by setting up storage and code-signing certificate. You can follow Steps 1 and 2 in Perform OTA Updates on Espressif ESP32 using FreeRTOS Bluetooth Low Energy.
2. Locate the aws_demo_config.h for your platform. For example, for ESP32, this file is located in vendors/espressif/boards/esp32/aws_demos/config_files/
   • Define CONFIG_OTA_UPDATE_DEMO_ENABLED and comment out any other demo defines.
3. Modify demos/include/aws_clientcredential.h:
   • Adjust the endpoint url in clientcredentialMQTT_BROKER_ENDPOINT[]
   • Adjust the thing name in clientcredentialIOT_THING_NAME
4. Modify demos/include/aws_clientcredential_keys.h:
   • Add the device certificate to the keyCLIENT_CERTIFICATE_PEM define.
   • Add the device private key to the keyCLIENT_PRIVATE_KEY_PEM define.
5. Modify demos/include/aws_ota_codesigner_certificate.h:
   • Adjust signingcredentialSIGNING_CERTIFICATE_PEM with the certificate that will be used to sign the firmware binary file. If you need more details on how to create the certificate, follow instructions in the first step.

Once the firmware is programmed, the OTA agent should continue to function normally for the connectivity processor. At the same time, it should also allow you to provide the OTA update to your secondary processor. You should see the following prints on the debug console of the connectivity processor:

12 309 [iot_thread] OTA demo version 0.9.2
13 309 [iot_thread] Creating MQTT Client...
----
21 823 [iot_thread] Connected to broker.
22 824 [iot_thread] [OTA_AgentInit_internal] OTA Task is Ready.
23 825 [OTA Agent Task] [prvOTAAgentTask] Called handler. Current State [Ready] Event [Start] New state [RequestingJob]
----
56 924 [iot_thread] State: Ready  Received: 1   Queued: 0   Processed: 0   Dropped: 0
57 1024 [iot_thread] State: WaitingForJob  Received: 1   Queued: 0   Processed: 0   Dropped: 0
58 1124 [iot_thread] State: WaitingForJob  Received: 1   Queued: 0   Processed: 0   Dropped: 0

At this point, your device is ready to receive an OTA update.

Set up the OTA update script

Once you have set up the firmware to allow for secondary processor updates, you must set up the cloud to transmit these updates to the device. The following steps walk you through how to initiate the OTA update to your secondary processor:

1. Install prerequisites:

   pip3 install boto3
   pip3 install pathlib

2. Get the OTA script from this code link.
3. Run the script with a fileId greater than 0 and by providing the file location for the secondary processor binary.
4. Help can be obtained by issuing:

   python3 start_ota_stream.py —h
   usage: start_ota_stream.py [-h] [--fileId FILEID] —profile PROFILE
   [--region REGION] [—account ACCOUNT]
   [--devicetype DEVICETYPE] --name NAME —role ROLE
   --s3bucket S3BUCKET —otasigningprofile
   OTASIGNINGPROFILE —signingcertificateid
   SIGNINGCERTIFICATEID [—codelocation CODELOCATION]
   [—filelocation FILELOCATION]
   
   Script to start OTA update
   
   optional arguments:
   -h, --help show this help message and exit
   --fileId FILEID ID of file being streamed to the device
   --profile PROFILE Profile name created using aws configure
   --region REGION Region
   --account ACCOUNT Account ID
   --devicetype DEVICETYPE
   thing|group
   --name NAME Name of thing/group
   --role ROLE Role for OTA updates
   --s3bucket S3BUCKET S3 bucket to store firmware updates
   --otasigningprofile OTASIGNINGPROFILE
   Signing profile to be created or used
   --signingcertificateid SIGNINGCERTIFICATEID
   certificate id (not arn) to be used
   --codelocation CODELOCATION
   base FreeRTOS folder location (can be relative) when
   fileId is 0
   --filelocation FILELOCATION
   OTA update file location when fileId is greater than 0
   

5. Example execution:

   python3 start_ota_stream.py --profile otausercf --name mythingname --role ota_role --s3bucket ota-update-bucket --otasigningprofile signingprofile --signingcertificateid <certid> --fileId 1 --filelocation update.bin
   
   Certificate ARN: arn:aws:acm:us-east-1:123456789012:certificate/cert-uuid
   Using App Location: update.bin
   Build File Name: update.bin
   Searching for profile signingprofile
   Found Profile signingprofile in account
   Waiting for signing job to completeOTA Update Status: {'ResponseMetadata': {'RequestId': '2c910ef5-1df5-4df6-8fe9-ddc3c46c68d2', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 07 Jan 2020 19:53:48 GMT', 'content-type': 'application/json', 'content-length': '184', '
   connection': 'keep-alive', 'x-amzn-requestid': '2c910ef5-1df5-4df6-8fe9-ddc3c46c68d2', 'access-control-allow-origin': '*', 'x-amz-apigw-id': 'F8g4CFECoAMFz5g=', 'x-amzn-trace-id': 'Root=1-5e14e1cc-1fb61a4d9261e6b0602290c9'}, 'RetryAttem
   pts': 0}, 'otaUpdateId': 'device-8673-0-0-0', 'otaUpdateArn': 'arn:aws:iot:us-east-1:123456789012:otaupdate/device-8673-0-0-0', 'otaUpdateStatus': 'CREATE_PENDING'}
   

6. You should see the update start in the console. Here are some prints you see in the debug console of the device:

   75 2767 [OTA Agent Task] [prvParseJobDoc] Size of OTA_FileContext_t [64]
   76 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ jobId: AFR_OTA-device-58124-0-0-0 ]
   77 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ protocols: ["MQTT"] ]
   78 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ streamname: device-8673-0-0-0 ]
   79 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filepath: update.bin ]
   80 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filesize: 10446 ]
   81 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ fileid: 1 ]
   82 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ certfile: /cert.pem ]
   83 2767 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ sig-sha256-ecdsa: MEUCIQDNRumLRyXqUM3Z2wa71/LV4ufv... ]
   84 2767 [OTA Agent Task] [prvParseJobDoc] Job was accepted. Attempting to start transfer.
   85 2767 [OTA Agent Task] [prvPAL_GetPlatformImageState_customer] OTA Demo for secondary processor.
   86 2767 [OTA Agent Task] [prvPAL_CreateFileForRx_customer] OTA Demo for secondary processor.
   ----
   96 2781 [OTA Agent Task] [prvRequestFileBlock_Mqtt] OK: $aws/things/mythingname/streams/device-8673-0-0-0/get/cbor
   97 2781 [OTA Agent Task] [prvOTAAgentTask] Called handler. Current State [RequestingFileBlock] Event [RequestFileBlock] New state [WaitingForFileBlock]
   98 2805 [OTA Agent Task] [prvIngestDataBlock] Received file block 0, size 4096
   99 2805 [OTA Agent Task] [prvPAL_WriteBlock_customer] OTA Demo for secondary processor.
   ----
   108 2816 [OTA Agent Task] [prvIngestDataBlock] Received final expected block of file.
   109 2816 [OTA Agent Task] [prvStopRequestTimer] Stopping request timer.
   110 2816 [OTA Agent Task] [prvPAL_CloseFile_customer] Received prvPAL_CloseFile_customer inside OTA Demo for secondary processor.
   111 2816 [OTA Agent Task] [prvIngestDataBlock] File receive complete and signature is valid.
   ----
   124 2833 [iot_thread] State: WaitingForJob  Received: 5   Queued: 0   Processed: 0   Dropped: 0
   125 2833 [OTA Agent Task] [prvPAL_ActivateNewImage_customer] OTA Demo for secondary processor.
   

Once the OTA update is complete, the device restarts as needed by the OTA update process and tries to connect with the updated firmware. If the connection succeeds, the updated firmware is marked as active, and you should see the updated version in the console:

58 866 [OTA Task] [prvUpdateJobStatus] Msg: {"status":"SUCCEEDED","statusDetails":{"reason":"accepted v0.9.2"}}

Conclusion

In this blog post, we described how you can perform OTA updates to secondary processors with AWS IoT Device Management and FreeRTOS. This mechanism can be expanded to upgrade any number of processors attached to a connectivity processor using an existing MQTT connection to AWS IoT Core.

We hope that you are able to use the steps provided in this post on your platform. Please visit this link to learn more about FreeRTOS.