Empowering wheelchair users with a socially assistive robot running on Amazon Machine Learning

Loro is a socially assistive robot that helps users with physical limitations to more robustly experience their worlds by assisting with seeing, sensing, speaking, and interacting with surroundings.  Loro uses a range of AWS artificial intelligence (AI) and especially machine learning (ML) services to enable its broad range of use cases.

Wheelchair users and others without full physical mobility face more than physical barriers; social interactions and personal health and safety are additional ongoing challenges in their lives. Inspired by their wheelchair-bound friend and mentor Steve Saling, Loro co-founders David Hojah and Johae Song sought to create a socially assistive robot to alleviate these challenges. In CTO Hojah’s words, “We wanted Loro to be a friendly companion on your shoulder like a parrot.”

This “parrot” and its companion app are powered entirely by AWS AI/ML.  Among the services that work in concert to give Loro its assistive abilities are Amazon SageMaker and AWS DeepLens, as well as a wide combination of Amazon Comprehend, Amazon Lex, Amazon Polly, Amazon Rekognition, Amazon Transcribe, Amazon Translate, and Amazon Textract.

Loro itself is about a foot tall and is designed to be affixed to the side of a wheelchair. “We started with just the idea of a camera attached to the wheelchair, to give people a panoramic view so they can navigate easily,” Hojah explained to TechCrunch in a recent interview. “We developed from that idea after talking with mentors and experts; we did a lot of iterations and came up with the idea to be smarter, and now it’s this platform that can do all these things.”

By “all these things,” Hojah is referring to Loro’s constantly expanding set of offerings, which currently includes helping its users to see, sense, listen, speak, interact with surroundings, and access information.  The robot uses a small camera and a built-in video screen to provide users a panoramic view of their surroundings. Using Amazon Rekognition, it can identify the faces of people in its 360-degree field of view and label them to help users keep track of their names. The camera also enables a user to navigate by using gaze-tracking.  Additional features include helpful tools like a flashlight and a laser pointer to assist with gesturing.

“Furthermore, Loro has a ‘Follow Mode’ based on face recognition that allows it to rotate automatically to follow the person who is moving in front of a wheelchair user without any manual input to control the camera view.  It is a wonderful tool to interact with the people who they really care about,” comments Hojah.

Loro also incorporates sentiment analysis, making it able to  analyze the facial expression of the patients, caregivers and the people who are around the end users. With Amazon SageMaker Reinforcement Learning, the bot continuously improves its analysis of facial expressions, especially of people it encounters frequently.  Separately, Loro uses Amazon SageMaker to predict the appearance and interpretation of facial expressions of specific people based on the surrounding context.

Many Loro users are non-verbal, so Loro uses Amazon Lex and Amazon Polly work in conjunction to understand and participate in conversations on their behalf.  It displays the speech it hears as text on its screen. Users can type on the screen to have Polly assist them in speaking their responses.  Additionally, Loro uses Amazon Lex and Amazon SageMaker to understand the sounds in its environment and determine whether it needs to take action based on them.  For example, if it hears a doorbell, it prompts its user with a verbal question and on-screen button allowing them to decide if they want to navigate toward the door.  This functionality also plays a role in helping keep users safe, as it can alert caregivers if a user needs assistance. In the future, Loro will be able to engage seamlessly with a large number of smart home devices (beyond the light switch and temperature control that it already offers).

In addition to providing an intuitive, inclusive, and user-friendly experience, Loro is designed to emphasize privacy. Amazon Cognito ensures that Loro’s user-signup and access control processes are completely secure.  The robot also fully encrypts all data that it sends over the internet and stores in the cloud (using Amazon S3).

The Loro solution and the company of innovators behind it have been recognized across the globe for the compassion-driven product.  Hojah describes the success as “like I dreamed, it’s a testimony to the power of technology mixed with good ideas.”  With AWS as Loro’s exclusive AI/ML platform, the technology-assisted care robot is indeed a dream come true.

About the Author

Marisa Messina is on the AWS ML marketing team, where her job includes identifying the most innovative AWS-using customers and showcasing their inspiring stories. Prior to AWS, she worked on consumer-facing hardware and then university-facing cloud offerings at Microsoft. Outside of work, she enjoys exploring the Pacific Northwest hiking trails, cooking without recipes, and dancing in the rain.