Machine Learning Infrastructure for Improved Usability During Vision Screening

Provectus is an AWS Premier Consulting Partner

Challenge

GoCheck was looking to enhance the image classification component of its pediatric photoscreening application through machine learning (ML). Since GCK is a cloud-based system, it is a data-rich environment where it was hypothesized that the application of machine learning could improve the usability of the app by supplementing the image analysis, and improving user actions to help capture the best image possible. GoCheck needed a robust and resilient ML infrastructure to run experiments on a dataset of more than one million images faster and more cost-efficiently, in order to train a highly accurate image classification model for the application.

Solution

GoCheck, in collaboration with Amazon Web Services (AWS), identified Provectus as a partner that had the domain experience necessary to execute the experiment. Working closely with GoCheck, Provectus reviewed their image classification software, prior ML methods and pipelines. As part of the effort to build ML models requested by the Client, Provectus delivered a secure, auditable, and reproducible ML training infrastructure deployed on AWS, including a robust experimentation environment with experiment tracking and model versioning, and active learning pipelines for continuous data re-labeling and re-training.

Outcome

GCK received a new machine learning infrastructure on AWS to support regular model retraining, evaluation, tuning and improvements to ML models as new labeled data and feedback arrive. More efficient data preparation and faster experimentation enabled GCK to increase the recall of ML models by 3X while preserving its precision. This brought in both short and long-term benefits, including improved usability of the app and increased customer satisfaction, by decreasing the final results of “child not looking” by asking the user to retake the image.

300x50_3X-icon

Increase in ML model's recall while preserving the precision

300x50_100-icon

Large scale experiments in three weeks by three ML engineers

300x50_95-icon

ML Engineers' time was dedicated to experimentation

Ensuring Photoscreening Efficiency: From Manual Review to Machine Learning

In the United States, vision impairment ranks as the most prevalent disabling childhood condition, according to the CDC. Almost always treatable if detected early, eye conditions such as amblyopia and strabismus cause vision impairment and blindness. Currently, only 15% of preschool children receive an eye exam and only 20% receive some type of vision screening.¹ Historically, photoscreening has required significant upfront capital, been difficult to scale and hard to operate, which has limited the access to pediatric photoscreening.

To remedy the situation, in 2013 GoCheck developed a next-generation clinically validated, pediatric photoscreening platform. Consisting of a smartphone app to perform vision screening and a backend hosted in the cloud, it enables healthcare practitioners to screen for amblyopia and other vision disorders in children in a fast, easy and affordable way.

To screen a child for vision disorders, a user of the app snaps a photo of the child’s eyes at a fixed distance in a dim environment. The app evaluates the image in real-time to check if it is acceptable for assessing via image processing. If so, the app analyzes the image, processes the findings through proprietary risk calculation algorithms and provides a recommendation of “Risk Factors Identified” or “No Risk Factors Identified”. In 1-2% of the screenings, the result may be inconclusive, or “Not Gradable”. Possible reasons for a not gradable result on an image are: child not looking straight at the camera, exposure not adequate, motion blur, extraneous light sources, etc. If the result is inconclusive the pediatrician can choose to rescreen the child, or refer if warranted. If the app indicates “Risk Factors Identified,” the child is referred to an eye care professional for further diagnosis and treatment. If “No Risk Factors Identified,” the child should be screened again in one year.

"How It Works" workflow of the GoCheck Kids application

For this framework to work as intended, highly accurate image analysis and classification algorithms must be part of the platform’s processing engine.

GCK’s original image classification algorithm performs on par with competing photoscreening technologies. Thanks to the cloud-based infrastructure, GCK has the ability to improve the application on a continual basis, unlike its competitors. GoCheck realized that early notification of a “Child Not Looking” was an opportunity to improve the user experience.

In addition to software review of the image, the images may be reviewed by a team of trained individuals who have the ability to manually review images to support user training and reimbursement coding. During the manual review process the reviewers are able to identify images where children were not looking at the camera in order to segment the data for machine analysis. When a “not gradable” result due to child not looking is identified, the reviewer notifies the provider to rescreen the child, which can create dissatisfaction among users, as well as consume time from the image review team, equating to as much as 20% of the teams overhead cost.

GoCheck wanted to alert the user in real-time to “retake” the image when the image was captured while the child was not looking directly at the camera. The application needed a robust and resilient machine learning infrastructure to prepare data, to run experiments on the dataset of 1M+ images and to build and train new image analysis and classification algorithms faster and more efficiently, as well as fine-tune the existing one.

To design and build such an infrastructure for machine learning, GoCheck teamed up with Provectus, an AWS Premier Consulting Partner with proven expertise in developing ML-powered platforms for eye screening.

Advanced ML Infrastructure for Data Preparation and ML Experimentation

For GoCheck, faster and more efficient data preparation, including data re-labeling and re-training and ML experimentation means more accurate image classification, timely feedback for test administrators and happier customers. This naturally translates into reduced rescreening rates, better user experience for users and lowered operational costs for GoCheck. Provectus took on the challenge to design, build and implement a machine learning infrastructure to help GoCheck achieve these success criteria.

At the preparatory stage, GCK’s datasets for gaze estimation were reviewed. Existing performance and KPIs were analyzed, and desirable metrics were agreed upon. It was critical to assess the quality of the datasets (in total, they analyzed over 1M images and provided a data set that included 150K+ eye screens) and determine how well they were labeled and “baked” into existing algorithms. ML methods and pipelines used in previous projects were reviewed as well.

Architecture diagram of ML Infrastructure

Provectus moved forward to deliver a secure and reproducible experimentation and ML training infrastructure, where pipelines are orchestrated and ML models are trained in Kubeflow. Some of the ML models, as well as analytics, were done in Amazon SageMaker.

The machine learning infrastructure development and implementation aligns with GoCheck’s broader strategy and future product developments. It was crucial to incorporate a feature store, experiment tracking and ML pipelines. ML model re-evaluation pipelines, as well as active learning pipelines for continuous, efficient data re-labeling and re-training were implemented to support ongoing model retraining, evaluation, tuning and improvements as new labeled data and user feedback arrive.

At the final stage, Provectus integrated the machine learning solution into the mobile application, and into GoCheck’s business workflow.

Faster ML Experimentation Means More Accurate Models, Less Manual Review and Happier Customers

GoCheck received a secure, auditable and reproducible machine learning infrastructure deployed on AWS, featuring a robust experimentation environment with experiment tracking and model versioning and active learning pipelines for continuous data re-labeling and re-training.

In just three weeks, more than 100 experiments on the entire subset data of 150K images were conducted, making it possible to not only test various improvement hypotheses but to also re-run every quality hypothesis to determine the best hyperparameters. The experiments produced artifacts such as models, metrics and predictions, to efficiently track and version each of them for further review and re-use.

The machine learning infrastructure helped the Provectus ML team significantly. More efficient data preparation and faster ML experimentation enabled us to train and tune multiple ML models, to select those with the best precision and recall. For instance, GCK’s original classification protocol correctly caught “Child Not Looking” cases with recall of ~25% (50% precision), but Provectus’ new model increased the recall to 91%, while preserving the initial model’s precision.

For GCK, more accurate image analysis in their application means fewer “not gradable” results and less manual messaging, which naturally translates into cost savings and increased user satisfaction.

The robust and resilient infrastructure for machine learning bolsters GoCheck’s quick, sustainable growth in the vision screening market. The GCK application is now equipped with advanced ML algorithms — courtesy of ML infrastructure — that further enable accurate and convenient photoscreening for a wide range of conditions, including amblyopia.

Better access to affordable eye screening with GCK means that millions of children will be diagnosed in time, and will not suffer from vision impairments and blindness as adults.

GoCheck Kids

About GoCheck Kids

GoCheck Kids (GCK), created by GoCheck, is a clinically validated, easy to use, comprehensive photoscreening and visual acuity app that helps to prevent vision impairments, blindness, and vision-related learning challenges in children 1-18. GoCheck Kids enables healthcare practitioners to screen for amblyopia (aka “lazy eye”) risk factors and other vision risks by providing an integrated, affordable solution in a simple smartphone app. The company currently serves over 6,500 pediatric teams in the US and Europe. The app is FDA-registered and CE certified.

About Provectus

Provectus drives AI transformation through machine learning, reimagining the way businesses operate, compete, and deliver customer value. Provectus is a consultancy and solutions provider helping design, architect, migrate or build cloud-native applications on the AWS Cloud and holds AWS Competecies in Machine Learning, Data & Analytics, and DevOps. 

Published October 2020