By Chee How Lim, CEO – Tapway
Marouane Sefiani, Sr. Partner Solutions Architect – IoT
Vasileios Vonikakis, Sr. Partner Solutions Architect – AIML

Tapway

Computer Vision (CV) models are algorithms trained to recognize patterns in images and videos. In order for them to maintain high accuracy as visual environments evolve, they need to be updated frequently. License plate recognition (LPR) refers to automated systems that use cameras and CV to read license plate numbers on vehicles. The license plate data gathered can be matched against databases to help identify stolen vehicles, uninsured motorists, suspects in crimes, and more. It is used by law enforcement and toll road operators for access control and enforcement. LPR data and analysis helps automate processes that humans previously had to do manually, thus saving time and labor.

In this blog post, we will see how Tapway managed to keep the CV models up to date, by constantly collecting new data and periodically retraining their models.

Tapway is a leading AI/ML & IoT company, leveraging the power of state-of-the-art CV technology to transform video feeds and images into intelligence and actionable insights. Tapway offers a vehicle profiling platform called VehicleTrack for LPR, vehicle classification and vehicle brand.

Challenges and problem statement

Since LPR systems are based on CV models, they can be as good as their training datasets allow.  As such, when deployed in real-world conditions, they may face the following 2 challenges:

• Unseen edge-cases: In real-life, the initial training set will rarely include the multitude of variations available in the world. there will be edge cases that the model was not trained with. In the case of LPR, this includes rare font types in the license plate, stains, stickers, worn-off ink or weather conditions that compromise image acquisition, like heavy rain, or glare etc.
• Data evolving over time: As time progresses, the underlying patterns and relationships within the data can change, leading to data drift. New patterns emerge, old patterns become obsolete, and the distribution of data shifts. This data drift can significantly impact the performance of ML models, causing model performance drift. In the case of LPR, this could be changes in the type of license plate style, color or fonts.

As such, continuous monitoring and retraining are necessary to address the challenges posed by evolving or rare data, allowing CV models to adapt and provide reliable insights in an ever-changing landscape.

Figure. 1 depicts real-world challenges handled by the solution, which can compromise the performance of LPR systems, resulting in false detections.

Figure 1: License plate with “broken” numbers and different character spacing

Solution Overview 

To address edge-cases issues and data drifting, Tapway has implemented an Incremental Training Pipeline (ITP) for their CV models. The ITP ensures that the models are up-to-date, and their performance is continuously improved overtime. Figure 2 shows the overall architecture of VehicleTrack.

Figure 2:  Overall Architecture

The overall architecture has the following workflow:

1. The Edge device has AWS IoT Greengrass installed. It collects and sends all vehicle images to an Amazon Simple Storage Service (Amazon S3) bucket.
2. Amazon SageMaker  annotates vehicle images, trains and optimizes CV models.
3. AWS IoT Core publishes instruction messages to the edge server to request vehicle-specific images for a particular class.
4. Amazon Kinesis  ingests data from logs data from the Edge server and publishes data to Amazon OpenSearch.
5. Amazon OpenSearch service stores all log data and uses Kibana dashboard for visualization.

Figure 3 shows the architecture of the ITP.

Figure 3:  ITP pipeline Architecture

The ITP architecture has the following sections:

• The edge device runs a local AWS Lambda function to send vehicle images to an Amazon S3 bucket
• Data Lake (based on Amazon S3) stores incoming new images, as well as model predictions.
• Data preparation for annotating new unseen images and triggering the incremental model training.
• ML training job for incrementally retraining the existing CV models on new data.
• Model deployment of newly trained CV models.

The system is designed to address the following scenarios:

• (Known) Minority classes with few initial training examples. The initial training dataset, most of the time, will be imbalanced, with some classes being under-represented, such as rare license plates from a particular province. The system can be triggered to send data for known under-represented cases. Over time, more data can be collected, increasing the number of training images for those under-represented classes and, thus, making the model more robust.
• Unusual predictions & (unknown) edge-cases. License plates follow specific regulations in each country. The system can be configured to periodically send discrepancy cases which do not adhere to these regulations. Once annotated, these cases can be important training examples to fix mistakes or learn new license types.

Since the previous scenarios result in the retrieval of new images, annotations will be needed to compile them into a training dataset that can be used to update the existing CV models in a supervised way. The Data Preparation section of the architecture includes a human annotation pipeline. Once the annotation process is finished, the new labelled images are stored in an Amazon S3 bucket, which will trigger an AWS Lambda function and initiate a new Training Job.

The ML Training Job part of the architecture can be used to either (re)train vehicle detection or car plate digit recognition models, using Amazon SageMaker training jobs. The whole process can be configured to train models from scratch, or incrementally. If the model meets a performance threshold on a standardized holdout test-set, then it will be pushed to production by deploying it to all edge servers.

Benefits & Success Metrics

As more training data for edge-cases and minority classes is collected over time, Tapway’s ITP can train a more accurate model. Tapway highlights that it may take up to 3 months to train a model manually, while a new model trained takes 1–2 weeks using their automated platform.

Tapway achieves an LPR level of accuracy above 95%. Figure 4 shows the number of training examples, and accuracy per digit that Tapway’s LPR system achieves. With more than 7000 labelled car plate images, the system achieves 100% accuracy for 26 out of 34 total digits, with the minimum accuracy being above 97.5% (for ‘M’ character).

Figure 4:  Training examples and accuracy on the hold out test set for each car plate digit

The following figure shows an example of how model performance improved within just one successive iteration, using Tapway’s ITP. The original version of the model (version 1), originally exhibited an Average Accuracy of 0.79 and Mean Average Precision of 0.6, across all digits from the golden test dataset. With the next model iteration (version 2), through the inclusion of more minority class examples and edge-cases, the ITP was able to improve Average Accuracy to 0.96 and Mean Average Precision to 0.83.

Figure 5: Performance of different model versions, after incremental training.

Following are some real-world edge-cases, along with the predictions of the model. It is evident that the resulting model, after the incremental training, is robust enough to handle challenging real-world cases.

License plate	Prediction
	PGL 7522
	PMP 9525
	PKJ 3503
	VDH 2001

Table 1: Real-world edge cases and their predictions

This system has been deployed for PLUS and 4 other highways in Malaysia. Tapway achieved 99.2% accuracy and is continuing to push the envelope by tackling unique plates through the incremental training pipeline.

Lessons learned from the field

After deploying the LPR solution in the field, new challenges were discovered. These included occlusions (e.g. car plate blocked by vehicles in front), close “1” characters, compromised license plate images due to motion blur, less accurate bounding boxes for stretched frames and unconventional character spacing. Nevertheless, Tapway’s ITP was able to address these new challenges and gradually improve model performance, by selectively acquiring more training examples, for each of these edge cases.

To increase the performance of their LPR models, Tapway included additional techniques such as:

• fine-tuning the optical character recognition (OCR) parameters to increase accuracy at certain angles and distances
• implementing a voting algorithm to select the most likely prediction across multiple LPR models
• fine-tuning the camera frame rate (FPS) and voting algorithms, based on how quickly the vehicle is moving

Conclusion

Production-ready CV systems face challenges like unseen edge-cases, variations in lighting/weather, and lack of diverse training data. Tapway addresses these issues using an incremental ML training pipeline that collects new edge cases, potential mistakes, and examples for under-represented classes. Their ITP has enabled their LPR and vehicle detection CV models to become more robust in handling real-world conditions and adapt to changing data distributions over time.

.

Tapway – AWS Partner Spotlight

Tapway is an AWS Technology Partner and AWS Smart City and Machine Learning Competency Partner  that works with businesses to improve productivity and reduce manual labor by transforming video feeds into intelligence and actionable insights. Tapway provides its AI-driven solutions for various industries including Retail, Hospitality, Manufacturing and Agriculture.

Contact Tapway | Partner Overview | AWS Marketplace