AWS Partner Network (APN) Blog
Nordcloud’s Automated Solution for Computer Vision Applications at the Edge Using AWS Panorama
By Salman Khaleghian and Yue Bai, Data Scientists – Nordcloud
By Stephan Völkl and Subash Prakash, Cloud Data Engineers – Nordcloud
By Ehsan Mohebianfar and Ilias Biris, Tech Leads – Nordcloud
By Phani Kumar Lingamallu, Sr. Partner Solution Architect – AWS
 |
| Nordcloud |
 |
In computer vision applications, the transmission of video data to the cloud for analysis can result in added delays due to various contributing factors such as queuing, propagation, and network latency. These factors make it challenging to fulfill the speed requirement necessary for real-time applications where quick and accurate results are of utmost importance.
Moreover, computer vision projects face numerous difficulties in training models and deploying them to multiple devices in real-world environments. These difficulties significantly impede the progress from proof of concept (PoC) to production-level deployment and maintenance, particularly when the locations are geographically dispersed.
While the machine learning (ML) models are typically industry-specific and developed based on client requirements, the technological skills required to build, deploy, and maintain the models add delays and cost to the overall deployment of the solution.
Maintaining these models in a dynamic and constantly evolving environment can be challenging, requiring ongoing monitoring and updates to ensure model accuracy. These ongoing efforts can make it difficult to sustain the implementation of the solution, even after deployment.
The Nordcloud team, in collaboration with Amazon Web Services (AWS), has designed a “Computer Vision at the Edge” solution based on AWS Panorama. It caters to organizations seeking low-latency decision making without the burden of managing complex technology, and Nordcloud leverages the capabilities of AWS Panorama to execute real-time, hardware-accelerated interference on the edge for industries that demand rapid decision making.
Nordcloud is an AWS Premier Tier Services Partner and Managed Service Provider (MSP) that empowers enterprises’ digital transformation by providing high-quality public cloud services and expertise.
Solution Overview
Nordcloud’s computer vision solution will transform the deployment of computer vision models at the edge, making it a hassle-free process for customers. The solution includes a fully automated CI/CD pipeline and integrates an industry-specific accelerator with just a few clicks.
Furthermore, the streamlined user onboarding process enables customers to simplify their Day 2 management process during production.
The solution consists of four major components to provide your desired results:
- Physical setup of cameras and AWS Panorama appliances.
- CI/CD pipeline to deploy applications on AWS.
- Customizable control panel user interface (UI).
- Deploying computer vision models using deploy wrapper for the AWS Panorama appliance.
The solution provides customers with centralized management of one or multiple models and cameras. As demonstrated in this post, the solution has two models already up and running while the UI allows you to select the appropriate prediction model for different use cases.
The UI is highly customizable, allowing users to fine-tune their settings by choosing the model, adjusting the threshold, and setting the refresh rate according to their specific needs.
Figure 1 displays an example of the control panel UI, which allows users to switch between, in this example, fish detection and bottle detection models. The image also showcases the corresponding real-time stream, along with the model predictions.
Figure 1 – Control panel UI to select models and adjust parameters.
In Figure 2 below, the model for bottle detection has a threshold of 1 and a refresh rate of 5 seconds. This means that if the number of bottles on the shelf falls below 1, an email notification will be sent to the end user, and the monitor screen will refresh every 5 seconds.
This configuration allows for real-time monitoring and timely notification of changes, ensuring users are always up to date and can take necessary actions as needed.
Figure 2 – Existing UI with bottle detection model.
ML Model Training for Targeted Applications
The model in this solution was created through the use of Amazon SageMaker, a fully managed machine learning platform that provides everything required to build, train, and deploy ML models. The training process utilized a custom dataset that was specifically tailored to meet the requirements of the particular use case.
Transfer learning was also utilized, allowing Nordcloud to leverage knowledge gained from previous models to improve the training process and increase the accuracy of the new model. By combining the powerful capabilities of Amazon SageMaker with transfer learning using YOLOv5 and a custom dataset, the team was able to create a highly accurate and effective model for the intended applications.
The trained model is integrated into Nordcloud’s CI/CD pipeline, allowing for seamless deployment to the AWS Panorama platform. The trained model is optimized to meet the requirements of Amazon SageMaker Neo, which is a requirement to deploy the compiled model on AWS Panorama devices.
Now, let’s look into use cases of these models created for specific industry needs. We’re using smart fish farming and personal protection equipment for health, safety, and environment (HSE) systems.
Smart Fish Farming
Automation in fish processing aims to streamline dangerous, difficult, and repetitive tasks such as trimming and sorting of defective fish. To achieve this, the technology must be advanced enough to identify and categorize fish defects and make real-time decisions for trimming, sorting or treatment.
In the graphic below, you can see a sample result from a transfer-learned YOLOv5 model which can detect and count fish in real time for further analysis.
Health, Safety, and Environment (HSE) Systems
HSE systems aim to promote a safe and healthy work environment, preventing accidents and ensuring overall well-being. Artificial intelligence (AI) and computer vision play a crucial role in achieving this goal across various industries, including oil and gas, manufacturing, warehousing, and even public spaces.
For example, the model can detect the presence and use of personal protective equipment (PPE) such as helmets, shoes, gloves, and industrial goggles, and monitor heavy machinery and work zones for safety.
As you can see in the sample result below, the trained model can quickly identify individuals who are not wearing a required helmet, triggering an alert to promote safety for a person entering a construction site.
Next, the sample result depicted below showcases the capabilities of the trained model in detecting individuals working on a construction site from a significant distance. The ability to monitor such a large area using computer vision and AI can save valuable time and resources, while also contributing to the overall safety of the site.
Customer Benefits
With Nordcloud’s “Computer Vision at the Edge” solution, users can achieve the following benefits:
- Real-time decision making: AWS Panorama edge device enables real-time decision making without latency.
- Quick deployment: The AWS Cloud Development Kit (AWS CDK) is utilized for fully automated DevOps pipelines. The preconfigured infrastructure can be deployed in minutes, saving time and money and increasing agility.
- Cost and maintenance reduction: By using AWS-native serverless technologies and DevOps best practices in the solution, maintenance costs are reduced. DevOps best practices such as automation and continuous delivery streamline processes and minimize manual intervention, further reducing maintenance efforts and costs.
- Scalability: The solution supports managing multiple AWS Panorama devices and cameras in a highly scalable manner. Serverless computing enables horizontal scalability of backend services.
- Customizable: The solution allows quick model adaptation for different computer vision use cases. It offers a custom UI that can be tailored according to customer requirements.
- Security: Extensive built-in security features to ensure data privacy.
Solution Architecture
The proposed architecture receives a real-time data stream from a web camera. The AWS Panorama appliance processes the stream using the deployed model to accurately recognize the images.
Once the model makes predictions, the resulting image frame is uploaded to Amazon Simple Storage Service (Amazon S3), and the payload data required by the control panel UI is sent to Amazon Simple Queue Service (SQS). Next, SQS triggers an AWS Lambda function, which retrieves the messages, applies customized business logic, and saves the output information into Amazon DynamoDB.
AWS Lambda sends alerts to the end user via Amazon Simple Notification Service (SNS) based on the payload content. To illustrate, a smart fishing farm could receive alerts when the number of fish arriving in a particular region of interest increases over time, allowing them to determine optimal fishing times.
The UI comprises two components: a monitoring window and a control panel. The UI runs on Amazon Elastic Container Service (Amazon ECS) and AWS Fargate. End users can log in to the UI via Amazon Cognito and monitor the real-time object detection scenes.
To enable customization of the control panel UI, a serverless approach is used in designing the APIs, making it adaptable to any custom changes as per the requirements.
Figure 3 – Computer Vision at the Edge solution architecture.
The architecture in Figure 3 supports multiple camera connections and enables deployment of various models. By leveraging edge computing, the solution enhances data privacy protection and reduces the impact of low network bandwidth.
In order to improve the security, AWS Certificate Manager is employed to facilitate the issuance of certificates that enable the SSL/TLS encryption of data in transit. Additionally, AWS Key Management Service (AWS KMS) is utilized to encrypt S3 storage buckets, thereby ensuring encryption at rest.
The infrastructure is implemented with AWS CDK, which enables straightforward and efficient replication of the entire structure in a new AWS environment.
Automation with AWS CDK
Nordcloud has created a pipeline with AWS CDK to implement the CI/CD and deploy all of the AWS application resources. The whole procedure is managed with AWS CodeBuild projects. To maximize the automation, Nordcloud deploys machine learning applications on AWS Panorama using customized scripts.
In Figure 4, you can see the architecture of the infrastructure deployment steps.
Figure 4 – CI/CD pipelines and model deployment automation architecture.
Nordcloud chose a mono-repo approach in AWS CodeCommit, which allows all pipelines to access the code and can be triggered based on code changes, specifically merged pull requests.
To enhance maintainability and scalability, the solution comprises three distinct pipelines that facilitate the clear separation of the infrastructure deployments:
- Bootstrap pipeline: The foundational components required for the application pipeline to function in a new AWS account, such as AWS Identity and Access Management (IAM) policies, roles, and S3 buckets are deployed through bootstrap pipeline.
- Application CI/CD pipeline: The core application which includes highly available control panel UI and APIs, database, identity services, and required storage services is deployed through application CI/CD pipeline.
- Deploy wrapper: The deploy wrapper facilitates the deployment of ML models on AWS Panorama devices, while also providing insights into the state of the appliance device.
Conclusion
In this post, we have showcased a solution architecture powered by AWS-native serverless technologies to deploy computer vision applications and support machine learning model predictions for industrial use cases.
Nordcloud’s Computer Vision at the Edge solution provides an easy means of deploying and running computer vision models on AWS Panorama devices. Thanks to the solution’s high level of automation and decoupling, it can be quickly adapted and deployed to new computer vision scenarios, reducing maintenance workload significantly.
If you need help developing or deploying computer vision applications on AWS, contact Nordcloud.
.
.
Nordcloud – AWS Partner Spotlight
Nordcloud is an AWS Premier Tier Services Partner and MSP that empowers enterprises’ digital transformation by providing high-quality public cloud services and expertise.