Introduction

Developing an IoT device or client can be a significant undertaking. Development efforts, ranging from design, prototyping, testing, quality assurance and more, can take months, if not years. Improving the development velocity and agility is obviously attractive for product cost and time-to-market. However, any acceleration should not compromise on quality, and deliver a device or client that is reliable, performant and secure.

Trend Micro, a multinational cyber security company that offers a wide range of internet security and antivirus software products and solutions, faced similar challenges when developing their Cloud One security platform. Cloud One is a comprehensive SaaS (Software as a Service) solution that helps customers secure their cloud infrastructure by delivering Trend Micro security solutions on a single platform, and provides a seamless cloud journey. The SaaS solution includes an agent that is deployed on a computer to provide application control, anti-malware, firewall protection and more. This agent connects to AWS IoT Core and continuously collects metrics and events for the purposes of threat analytics and management, preventing and responding to cybersecurity incidents immediately.

AWS IoT Core Device Advisor is a cloud-based, fully managed tool for validating IoT devices or clients that connect to AWS IoT Core, such as the Cloud One agent. Trend Micro used pre-built test cases in AWS IoT Core Device Advisor to accelerate the development of their Cloud One platform and to automate regression testing through their in-house Cloud One Continuous Integration/Continuous Deployment (CI/CD) pipeline.

“Before we automated this process, a person would have to dedicate an hour to manually run all the test cases, which made it impractical to test every build. Now, the process runs automatically on every build and completes within half an hour without any human intervention. The feedback is fast, and the manual test effort is greatly saved.” said Shan Rao, Automation Test Engineer at Trend Micro.

Challenges

Before adopting AWS IoT Core for the Cloud One security agent, Trend Micro used a traditional API-based client-and-server architecture. The API-based client-and-server model is a familiar and straightforward way of building an application like Cloud One.

However, as the number of clients grows to hundreds of thousands, or even millions, the solution can be challenging to scale. The costs associated with infrastructure and maintenance can rise quickly, and the architecture design may not easily cope with the number of concurrent connections. Trend Micro faced these issues with their Cloud One product, which required reliable and stable infrastructure to serve the incoming requests from Cloud One agents. Furthermore, Trend Micro aims to rapidly deliver new features and updates to protect their customers against vulnerabilities and unauthorized changes.

Consequently, they chose AWS IoT Core to build their next-generation Cloud One agents because it allows them to connect billions of IoT clients and route trillions of messages to AWS services, without managing infrastructure. With AWS IoT Core, Trend Micro simplified their Cloud One architecture, reduced operational complexity and focused more on product feature development and differentiation.

Figure 1: Simplified Cloud One Agent architecture

Upon selecting AWS IoT Core, Trend Micro then needed tools to improve their development process and verify compatibility between AWS IoT Core and the Cloud One agent. AWS IoT Core Device Advisor is purpose-built for this validation and can be used to validate both physical devices and soft clients. It was therefore a logical choice.

Solution Overview

AWS customers can use AWS IoT Device SDKs to build IoT clients. These SDKs are already qualified against AWS IoT Core Device Advisor, reducing the development burden for customers. Alternatively, customers can use a third-party MQTT client of their choice to connect to AWS IoT Core, or even develop their own client.

Trend Micro elected to develop the Cloud One agent using a custom MQTT client library. To aid the development of the agent, Trend Micro integrated AWS IoT Core Device Advisor as an automation test workflow within the agent’s CI/CD pipeline. This verifies that every build of the agent can securely connect to AWS IoT Core, and can handle retry and back-off scenarios. Moreover, any functional regressions are immediately identified, allowing for fast correction and preventing serious regressions from ever reaching the field.

Test Cases

The CI/CD pipeline implements a test suite that uses a selection of pre-built TLS test cases and MQTT test cases in AWS IoT Core Device Advisor. The test suite includes both happy path and sad path test cases, and can quickly identify common device software issues during the development process.

The following test cases are used to validate that the agent can complete TLS handshake with AWS IoT Core and that the agent presents a valid cipher suite in the TLS Client Hello message:

• TLS Connect
• TLS Support AWS IoT Cipher Suites

The agent should close the connection if the server certificate doesn’t meet requirements. The following test cases present invalid server certificates to the agent, ensuring that the agent only connects to an endpoint that presents a valid certificate:

• TLS Unsecure Server Cert
• TLS Incorrect Subject Name Server Cert

These test cases validate the agent’s MQTT implementation, confirming that the agent can establish a connection with the MQTT broker, and publish a message and subscribe to a topic:

• MQTT Connect
• MQTT Publish
• MQTT Subscribe

The following test cases validate that the agent uses the proper jitter and exponential back-off while connecting with the broker:

• MQTT Connect Jitter Retries
• MQTT Connect Exponential Back-off Retries
• MQTT Reconnect Back-off Retries On Server Disconnect

Some of the test cases run for a long time, but these automated tests provide fast feedback and are more efficient than manual tests.

Workflow

The CI/CD pipeline automated test is based on the AWS IoT Core Device Advisor workflow and can be summarized as follows:

Figure 2: Cloud One Agent automated test workflow.

For each pipeline execution, the automation test job uses the AWS SDK to create an IoT Thing, create an X.509 certificate, attach the certificate to the Thing, and attach an AWS IoT policy. The test job uses the AWS CLI to create test suites, start the suite running, and poll the suite status.

When tests take longer than expected, the test job stops the test suite. The stopped test suites are seen as failed tests and are kept in the AWS management console, along with the failed tests, so that developers can access the Amazon CloudWatch logs for deeper investigation and troubleshooting.

In this particular CI/CD pipeline, design choices were made to automatically delete test suite runs that were successful. Therefore, only the stopped and failed test suite runs are kept for developers to investigate, and hence no successful test runs are seen in the AWS management console:

Figure 3: Failed and stopped tests in the AWS Management Console.

Conclusion

Trend Micro were able to use AWS IoT Device Advisor pre-built test cases as part of their continuous integration and continuous deployment practices to deliver the Trend Micro Cloud One security platform to the market more quickly and without compromising on product quality. They continue to benefit from this as the automated regression testing helps new features and enhancements to be shipped at an accelerated pace with a high degree of confidence.

“The most important thing is that we can integrate Device Advisor into our existing CI/CD pipelines and test every change. This helps give us confidence that a change is good. And if it’s not, it is isolated and we get diagnostics from Device Advisor to help understand what went wrong. This means we can deliver faster without compromising quality.” said Michael Dysart, Senior Staff Software Engineer at Trend Micro.

To get started with AWS IoT Core Device Advisor, please watch “How to Get Started with AWS IoT Core Device Advisor” and read our earlier blog series.

To learn more about AWS IoT services and solutions, please visit AWS IoT or contact us. To learn more about Trend Micro, please visit their website.

About the authors

Greg Breen is a Senior IoT Specialist Solutions Architect at Amazon Web Services. Based in Australia, he helps customers throughout Asia Pacific to build their IoT solutions. With deep experience in embedded systems, he has a particular interest in assisting product development teams to bring their devices to market.

Wayne Huang is a Solutions Architect at Amazon Web Services based in Taiwan. Wayne has in-depth experience in IoT and software development, and is a member of IoT Technical Field Community (TFC) for AWS. He helps our customers design IoT architecture and enable them to build end-to-end IoT solutions from the ground up.

Shan Rao is an Automation Test Engineer at Trend Micro. She is a DevOps, IoT and automation enthusiast. She is passionate about implementing automation tests on AWS to accelerate the release to the market with high quality.