Creating a Hands-Free Voice Assistant at Houston Methodist with AWS

Hospital systems are dynamic, fast-paced, and challenging environments to manage. There are vastly different areas from emergency departments to walk-in clinics, where seamless, secure, and efficient interactions must occur in order to achieve optimal patient satisfaction, care delivery, staff productivity, and the utmost safety. Electronic Health Record (EHR) systems and other Healthcare Information Technology (HIT) solutions that were introduced decades ago to help manage data and care delivery have led to a rise in digital systems throughout hospital systems. The resulting side effect is that clinicians now spend more time interacting with computers and digital devices instead of spending more time with patients, leading to increased reports of burnout and stress.

The Journey to Voice at Houston Methodist

Earlier this year, Houston Methodist began working with Amazon Web Services (AWS) to find opportunities to bring digital transformation to patient care. With over 1.3 million clinic visits and more than 89,000 surgeries per year, Houston Methodist was interested in using automatic speech recognition (ASR) technology to create contact-less solutions to improve patient safety and experiences, while also enabling clinicians to interact with clinical applications. Houston Methodist worked with AWS to understand how ASR services like Amazon Lex could help their hospital system deliver a highly scalable, resilient, and secure way to support providers with voice-enabled technology in two areas of operation — operating rooms and ambulatory patient exam rooms.

At AWS, we work backwards from customer needs to build solutions that solve challenges our customers face. To accelerate Houston Methodist’s vision for voice-enabled technology, AWS started with a workshop that we call “Working Backwards.” This process involved meeting with key stakeholders, from support staff to clinicians, to ensure that we captured a complete understanding of the processes at the Houston Methodist hospital system. This approach provided a collaborative way to identify business challenges and prioritize solutions that would address them. Pariveda Solutions, a member of the AWS Partner Network (APN), was brought in to help with addressing the requirements that would need to be involved with building a device, using AWS services, to support patient and provider interactions through voice-enabled technology.

Overview of the Solution

For their operating rooms, Houston Methodist wanted to build a solution that would allow surgeons and support staff to interact with a digital voice assistant before, during, and after surgeries. The solution needed to be able to query data, such as allergies or current medications, from the patient’s electronic health record. It also needed to allow clinicians to start and stop verbal timers (e.g. “start tourniquet right arm”), as well as allow staff to complete verbal safety checklists for the surgery. The following architecture illustrates what was built at a high level.

Looking at the solution diagram for the operating room voice platform, we can see the following services used: Amazon API Gateway, Amazon S3, AWS Lambda, Amazon Cognito, Amazon DynamoDB, Amazon CloudWatch, Amazon Lex. The solution runs on a computer within the operating rooms that uses local microphones to capture audio. A wake word is used to start the interaction with the solution. The audio is then sent to Amazon Lex – a service for building conversational interfaces into any application using voice and text – so that the clinician’s intents can be fulfilled by Amazon Lambda.

Within patient exam rooms, Houston Methodist wanted to create an ambient listening application that would capture dialog between clinicians and patients in a secure and compliant way. During an exam, the dynamic conversation between patient and provider is transcribed and indexed for clinical relevance. The solution runs on a client device (i.e. an iPhone, iPad, Android or PC) in the exam room using in room microphones that the clinician controls. After receiving patient consent to document the visit, a microphone is activated and an application will capture the visit conversation. Should the patient not consent to the device being used or their discussion being recorded, the clinician does not activate the application or microphone. After the visit, a summary note of the interaction is automatically generated in real-time and emailed to the patient as well as inserted into the clinician’s Electronic Health Record (EHR) inbox as a SOAP (subjective, objective, assessment, and plan) note for review. Furthermore, since clinically relevant data from the exam is now indexed and searchable, Houston Methodist is able to automatically insert relevant data points captured during the visit into discreet fields of their EHR. Houston Methodist owns this data and neither AWS nor Pariveda have access to patient-clinician conversations or content captured by the device.

The following architecture illustrates what was built at a high level.

Looking at the solution diagram, we can see the following services used: Amazon API Gateway, Amazon S3, AWS Lambda, Amazon DynamoDB, Amazon Transcribe Medical, and Amazon Comprehend Medical. Audio is sent to Amazon Transcribe Medical where it is processed and forwarded to AWS Lambda. AWS Lambda uses Amazon Comprehend Medical to parse the medical terminology and then process a note into the EHR system for the clinician to review before it’s then entered.

Both of these solutions use available voice-enabled technologies from AWS, but in different ways. The combination of these ASR services powered by AWS is helping Houston Methodist achieve their goal of improving patient safety and experiences all while capturing essential medical and health care information so the focus can stay on patient care.

Conclusion

Healthcare organizations are always re-imagining how patient care can be delivered and improved. Using technologies like ASR, clinicians have the opportunity to spend more time interacting with patients instead of computers and devices. The emphasis on increasing meaningful communication during patient interactions is a key focus for Houston Methodist as they continue their digital transformation journey that supports both patients and clinicians. Houston Methodist recognized that by building new offerings with the most experienced cloud provider with the greatest breadth and depth of services, they can create solutions tailored for clinicians and patient needs.

We encourage other healthcare customers to start a conversation today with their AWS account team so they can learn more about how they can build solutions using our ever-growing number of HIPAA-eligible, highly secure, and reliable services.

Watch Roberta Schwartz, EVP and Chief Innovation Officer at Houston Methodist, speak about this solution in the HLTH 2020 session, Houston Methodist, AWS & Pariveda Enabling Voice Technology in Hospitals. To learn more about how AWS is transforming the healthcare industry, visit Healthcare & Life Sciences on AWS.