Voice enabling participant diary collection powered by AI on AWS – Part 2

Blog is guest authored by Dr. Isaac R. Rodriguez-Chavez, SVP, Scientific and Clinical Affairs, Head, Decentralized Clinical Trials and Digital Medicine, Drug Development Solutions, ICON plc

Learn how to integrate voice enabled diary collection into patient trials. Doing this could translate into a better participant experience―helping optimize study efficiencies to obtain quality efficacy and safety results of novel medical products.

In the first post of this series, we described the architecture and design considerations for building voice applications in clinical research powered by AI on Amazon Web Services (AWS). In this post, we will walk through the key components of the voice participant’s diary sample application and how these components may be implemented for clinical research. Before we dive in, let’s review some background regarding electronic clinical outcome assessments (eCOAs).

In clinical studies conducted by the Pharmaceutical industry, eCOAs help clinical investigators better understand the efficacy of an experimental medical intervention for study participants. For example, some eCOAs may represent endpoints to assess or complement the assessment of the safety and efficacy of an investigational medical product.

Electronic Patient Reported Outcomes (ePROs)

One type of eCOA measure is ePRO, known as electronic Patient Reported Outcome. This is a health outcome-related survey specifically used for a research study and typically deployed through mobile devices, tablets, computers, or other electronic systems. It is important to assess clinical endpoints as the data produced comes directly from the trial participants enrolled in the study. Investigators can better understand how the participant is feeling, functioning, and surviving an experimental medical intervention. While investigators learn about participant’s symptoms during a clinical study, they correlate this information with other collected data.

Participants’ eDiaries are an electronic collection of clinical events from a study participant with automated data entries on a handheld mobile device, tablet, or computer, or an interactive voice response system. Studies have shown that compared to paper diaries, eDiaries are more effective at improving participant compliance with the research protocol. More compliance, in turn will generate more accurate and reliable data, maintaining its integrity and quality.

An automated approach also enhances study participant engagement, retention, and their overall satisfaction of participating in a clinical study. This translates into better participant’s experience, which helps in optimizing study efficiencies to obtain quality efficacy and safety results of novel medical products.

Although many participants find it convenient to receive reminders and record information using their smartphones, this approach does not work for everyone. Some participants may have medical conditions such as vision impairments or motion impairments (for example, Parkinson’s or ALS), making it difficult to enter data on a phone screen. In other clinical studies, investigators may work with participants without sufficient literacy to read the question prompts.

For these participants, a system that allows outcome reporting through voice offers a valuable alternative. Building blocks from AWS such as Amazon Lex, a fully managed artificial intelligence (AI) service with advanced natural language models to design, build, test, and deploy conversational interfaces in applications, and Amazon Connect, an omnichannel cloud contact center that helps you provide superior customer service at a lower cost, enable clinical investigators to quickly stand up a hands-free voice interface for participant outcome reporting. By leveraging capabilities built into AWS services, you can build natural, conversational participant experiences using an automated voice reporting system.

Deploying the sample application

The sample application code is hosted in this GitHub repo. If you’d like to deploy the application in your AWS account, follow the steps outlined in the README.md of the repo.

Amazon Connect contact flow

Contact flows in Amazon Connect define end-user interactions and are comprised of contact blocks, with each block defining a step or interaction in your contact center. To enable a dynamic and personalized experience for users, you can use a combination of AWS Lambda (Lambda) functions and Amazon Lex chatbots as components in a contact flow. For example, AWS Lambda may be used in a contact flow to query a database of customers based upon the customer’s phone number so that the caller can be greeted by name. Amazon Lex could be used in contact flow to ask the caller to verify a security code after the initial greeting.

In the following high-level contact flow diagram a dynamic reporting flow is built using conditional branching based on participant profile and report status, then parsing participant input using natural language processing.

Contact flow diagram

As the call starts, the contact flow can determine whether this was a call initiated by the participant or the automatic participant outreach scheduler. If the call was initiated by the participant (inbound call), a CheckIncomingNumber Lambda function block is used to look up the incoming phone number. Calls from unregistered numbers are prevented from logging an update as a security precaution. This logic may be extended to invoke additional security questions or services as required for instance using Amazon Connect Voice ID (Voice ID). Voice ID is a near real-time caller authentication and fraud risk detection service using machine learning powered voice analysis.

In the case of the call being an outbound call initiated by an automatic scheduler, the flow will check with the participant on whether they are available for a few minutes to log their participant diary. Their response to the availability question will be interpreted by a ConfirmReportTime Amazon Lex bot. If the participant indicates they are ready to report their diary, the flow continues to the next step. If the participant indicates they are not ready to report, the system will continue to reach out again based upon the configured retry schedule.

To provide a two-factor authentication mechanism, the contact flow asks the participant a security question (for example, asking for their pre-registered authentication code). Handling the participant’s response is again delegated to an Amazon Lex bot, VerifyIdentity. The bot is configured to allow three tries for the security question before marking the authentication a failure.

If the participant answers the security question correctly, the Amazon Lex bot will also do a lookup in the backend system. It finds the next questionnaire the participant needs to complete, and sets the name of the questionnaire in a session attribute called NextBot. The contact flow uses the value of this session attribute to route the user to the corresponding Amazon Lex bot for a given questionnaire, such as logging medication time.

After the bot has collected all the required inputs for the questionnaire it is responsible for, it will update the participant’s diary completion status and set the NextBot attribute to the next questionnaire the participant has yet to enter. This process repeats for each questionnaire until the participant completes all the required questionnaires for that session.

This contact flow design has several advantages. First, by being able to programmatically determine and route the participant to the appropriate questionnaire to complete. This implementation is flexible enough to handle a variety of participant profiles, to whom different questionnaires may be assigned based on the specific clinical trial protocol or treatment they are receiving. Secondly, by encapsulating the logic for each questionnaire into its own Amazon Lex bot, each questionnaire can be developed and tested independently. Having single-purpose Amazon Lex bots reduces the chance of ambiguity and streamlines the backend logic for interpreting the user input.

To learn more on how the example contact flow is implemented, follow the setup instructions to import the contact flow definition file into your Amazon Connect contact center instance. You can use the editor user interface (UI) to make changes to the contact flow.

Editor UI in Amazon Connect regarding contact flow

Amazon Lex bot for symptoms recording

When it is time to capture a research participants’ reported outcomes, the Amazon Connect contact flow will prompt the participant “What symptoms are you experiencing today?”. The participant’s response is then handled by a SymptomReport Amazon Lex bot. This is configured in a “Get customer input” block in the Amazon Connect contact flow. The threshold for contacting the participant provider based upon a pain level 1-10 reported is configurable in the LexBot configuration file during deployment.

“Get customer input” block in Amazon Connect contact flow

When asked “Do you have any symptoms to report today?”, the participant is likely to have three types of answers that we can configure to map to different intents:

Participant confirms they have symptoms to report
1. example, “yes I do” – YesSymptom intent
Participant has no symptoms to report
1. example, “no I don’t” – NoSymptom intent
Participant just starts mentioning symptoms in their response
1. example, “yes I’m having a stomach ache” – ReportSymptom intent

Example symptom gathering flow with localized symptoms

Example symptom gathering flow with pain level follow-up

Example symptom gathering flow that starts with simple confirmation

Example symptom gathering flow with no symptoms to report

For the first scenario, the YesSymptom intent is configured with a few different sample utterances the participant may respond positively to the question prompt.

Sample utterances configured for YesSymptom intent

Below is the key code snippet of the AWS Lambda function in Python that fulfills this intent. The list of symptoms that the participant has already reported so far is stored in a session attribute, used to keep conversation states between the client and the Amazon Lex service. The bot will respond to the participant differently based on if some symptoms, already reported, are stored in the session attribute. This is to differentiate between two different situations where a participant may respond with a simple confirmation:

When answering the initial prompt: “Do you have any symptoms to report today?”
When answering the follow-up: “Do you have any other symptoms to report?”

def yes_symptom(intent_request):
    session_attributes = helper.get_attribute(intent_request, 'sessionAttributes')
    symptoms = helper.get_list_from_session(session_attributes, SYMPTOM_ATTR)
    msg = msg_strings.get('ASK_SYMPTOM_FOLLOW_UP') if symptoms else msg_strings.get('ASK_SYMPTOM_DETAIL')
    return helper.elicit_intent(session_attributes, message_content=msg)

The NoSymptom intent is implemented similarly to the YesSymptom intent. It also handles two situations where the participant may respond with a negative. Either that they had no symptoms to report, or that they have no more symptoms to report (for example, they completed telling all of their symptoms).

Sample utterances configured for NoSymptom intent

The ReportSymptom intent is configured to model when participants describe their symptoms.

Sample utterances configured for ReportSymptom intent

The parameters the bot will capture (for example, symptom, bodyPart, modifier, painLevel) are enclosed inside the {}. In Amazon Lex, these input parameters are called “Slots”. The example bot defines the four different slots we capture.

Input parameter slots for ReportSymptom intent

Each slot has a name and a type, which can be a built-in type (for example, the painLevel slot uses the Amazon.Number built-in type, which represents numbers) or a custom type. For the symptom slot that captures the name of the symptom, we chose to build a custom slot type which restricts resolution to a set of predefined values of synonyms. One benefit of this approach is Amazon Lex will automatically map various synonyms to a single root word. For example, “hurt” and “ache” will both resolve to the value “pain”. This also makes it straightforward to add follow-up logic, such as if the symptom is some kind of pain. You can then ask the participant to rate their pain on a scale of 1–10 (in Amazon Lex terminology, “eliciting the slot value for the painLevel slot”).

Slot Type Synonyms

Handling errors

It is important to gracefully handle errors when designing a Voice User Interface (VUI). The sample application demonstrates two ways to handle inputs that Amazon Lex may have trouble understanding.

Case 1: The bot understands the intent but is unable to resolve the slot value
In the previous section, we discussed defining the “symptoms” slot to accept a set of predefined values of synonyms. What if the participant says a symptom that is not in the predefined list? Here, we can leverage the originalValue field in the event data passed from Amazon Lex to the Lambda code hook. In a normal case, when the bot is able to successfully resolve the slot value, the processing Lambda function can read the parsed values from the “slots” field of the input data. For example:

…
"slots": {
"symptom": "pain",
"modifier": null,
"painLevel": null,
"bodyPart": "chest"
},
…

In an error case, when Amazon Lex can recognize the intent of the user, but could not resolve the slot value, the processing Lambda function can look for the originalValue field under slotdetails. Below is an example of what the input data looks like in this scenario:

…
"slots": {
"symptom": null,
"modifier": null,
"painLevel": null,
"bodyPart": null
},
"slotDetails": {
"symptom": {
"resolutions": [],
"originalValue": "migraine"
},
…

The Lambda function that processes the event can then apply business logic to handle the situation. For example, confirm the user input by repeating the phrase it heard the participant say, and then store it in the reporting database with an “unknown” flag. This can trigger a notification to a human operator to review the record.

The following image shows an example of testing the reporting flow with a symptom name intentionally not included in the list of values supported by the symptom slot. The dialog flow continues and an “unknown symptom” flag is added to the session attributes.

Testing the reporting flow with an unidentified symptom name

Case 2: The bot does not understand the intent of the user
Another graceful way to handle errors is by leveraging the built-in fallback intent from Amazon Lex. As an example, in the SymptomReport bot code included in the project repository, we define a SymptomFallback intent for the SymptomReport Lex Bot. You will find this configuration item on the “Editor” tab of the selected intent.

When the user says something that the bot could not map to any other intent defined in the bot, and a fallback intent is defined, the Amazon Lex service will invoke the Lambda function configured for the fallback intent. As shown in the following code snippet, when the fallback intent is triggered for the symptom collection bot, the Lambda function will retrieve whatever the user said from the inputTranscript field and confirm if it heard it correctly. If the user confirms, their utterance is treated as an “unknown symptom” and logged to the backend database. Similar to Case 1, a notification to a human operator may be triggered to review the record.

if confirmation_status == helper.ConfirmationStatus.NONE.value:
    user_utterance = intent_request["inputTranscript"]
    if user_utterance == "":
        return helper.elicit_slot(
            session_attributes,
            INTENT_REPORT_SYMPTOM,
            slots,
            SLOT_SYMPTOM_ONE,
            msg_strings.get("CLARIFICATION"),
        )
    message = f"your said '{user_utterance}', is that right?"
    slots[SLOT_SYMPTOM_ONE] = user_utterance
    helper.append_session_attr(session_attributes, UNKNOWN_SYMPTOM_ATTR, user_utterance)
    return helper.confirm_intent(
        session_attributes,
        INTENT_REPORT_SYMPTOM,
        slots,
        message_content=message,
        message_type="PlainText",
    )

When using fallback intents, you may also find it helpful to disable clarification prompts, this setting is located on the Editor tab of the intent being edited. By doing so, the bot will trigger the custom logic you define for dealing with fallback intents immediately when it could not understand the user’s input.

Amazon Lex bot for recording medication time

To record the time the participant took their medication, the contact flow will prompt the participant “What time did you take your medication today?” Similar to the symptom report bot, the bot is defined with three intents that capture the participant’s likely response. Below are corresponding utterances for each intent.

NoMedication: the participant responds that they have not taken the medication.

NoMedication intent

MedicationTime: the participant responds with the time medication was taken. If the participant’s answer is ambiguous, the bot asks follow-up questions to confirm the time of the day.

MedicationTime intent

YesMedication: the participant may respond that they took the medication but doesn’t provide a specific time.

YesMedication intent

The main MedicationTime intent that records the time of medication is more streamlined than the symptom gathering bot with only two slots to capture: “med_time” (example, 08:00) and “time_of_day” (AM or PM).

Sample utterances configured for MedicationTime intent

Cleanup

Additional details not covered in this walkthrough include how the initialization and validation code hook can be used for input validation, and using Amazon Pinpoint to send notifications through email and SMS (Short Message Service). Amazon Pinpoint is a managed service that provides marketers and developers one customizable tool to deliver customer communications across channels, segments, and campaigns at scale.

Summary

In this blog post, we described the architecture and design considerations for building voice-based ePRO applications in clinical research powered by AI on AWS. We walked through the key components of the voice of the participant’s diary sample application and how these components are implemented in detail. With this foundation you can quickly build a cost effective, innovative call center that will allow your company to focus on building world class interactions for your customers.

To build additional features for your voice reporting use case, please review the code for the application at this GitHub repo.

To find out more about how AWS or our partners can help you please contact your Representative.

AWS for Industries