AWS Machine Learning Blog
Using Amazon Lex Conversation logs to monitor and improve interactions
July 2024: The solution in this blog post is now obsolete with the release of Amazon Lex V2.
As a product owner for a conversational interface, understanding and improving the user experience without the corresponding visibility or telemetry can feel like driving a car blindfolded. It is important to understand how users are interacting with your bot so that you can continuously improve the bot based on past interactions. You can gain these actionable insights by monitoring bot conversations. To capture user input, you could write custom logic in your application, but building and managing additional code and associated infrastructure is both cumbersome and time-consuming. You would also need to make sure that the custom logic does not increase the latency for end-users.
We are excited to announce Conversation logs for Amazon Lex, which enables you to natively save interactions with your bot. You can configure logging text input to Amazon CloudWatch Logs and audio input to Amazon S3. In addition to the user input and bot responses, the logs contain information such as matched intent and missed utterances. To protect sensitive data captured as slot values, you can enable slot obfuscation to mask those values for logging.
You can use Conversation logs to track utterances that were not mapped to any configured intent. These missed utterances allow you to improve the bot design. Now that conversation transcripts for the entire session are available, you can better analyze the conversation flow, improve conversational design, and increase user engagement.
This post demonstrates how to enable conversation logs, obfuscate sensitive slots, and set up advanced monitoring capabilities for your bot.
Building a bot
This post uses the following conversation to model a bot for an auto loan:
User: I’d like to check the outstanding balance on my car loan.
Agent: Sure, can you provide me your account number?
User: It is 12345678.
Agent: Can you please confirm the last four digits of your SSN for verification?
User: 1234
Agent: Thank you for the information. The balance on your car loan for account 12345678 is $12,345.
User: Ok thanks.
Build the Amazon Lex bot AutoLoanBot
(download) with the following intents:
- ApplyLoan – Elicits necessary information, such as name and SSN, and creates a new request.
- PayInstallment – Captures the user’s account number, the last four digits of the user’s SSN, and payment information, and processes the monthly installment.
- CheckBalance – Elicits the user’s account number and the last four digits of their SSN and provides the outstanding balance.
- Fallback – Captures any input that the bot cannot process by the configured intents.
Create an alias Prod
and publish a version of the bot using this alias.
Enabling Conversation logs
You can create Conversation logs for both text and audio.
Text logs
Before configuring Conversation text logs, create a new log group car-loan-bot-text-logs
in CloudWatch Logs. As an optional step, you can encrypt the text logs. For more information, see Encrypt Log Data in CloudWatch Logs Using AWS KMS. Additionally, create the IAM role LexCarLoanBotRole
to provide write permissions to the car-loan-bot-text-logs
log group.
To set up text logs for a bot alias, complete the following steps:
- On the Amazon Lex console, choose the
AutoLoanBot
. - Choose Settings.
- Choose Conversation logs.
- Choose the Settings gear icon that corresponds to the Prod
- Under Log type, choose Text logs.
- For Log group name, select
car-loan-bot-text-logs
from the dropdown list. - For IAM role, select
LexCarLoanBotRole
from the dropdown list. - Choose Save.
Audio logs
Before configuring Conversation audio logs, create the S3 bucket car-loan-bot-audio-logs
. As an optional step, you can encrypt the audio logs with a KMS customer managed CMK. To do so, create the new KMS key car-loan-bot-audio-logs-kms-key
. Additionally, create the IAM role LexCarLoanBotRole
to provide write permissions to the car-loan-bot-audio-logs
S3 bucket and access permissions to car-loan-bot-audio-logs-kms-key
.
To enable audio logging, complete the following steps:
- On the Amazon Lex console, choose the
AutoLoanBot
. - Choose Settings.
- Choose Conversation logs.
- Choose the Settings gear icon that corresponds to the Prod
- For Log type, select Audio logs.
- For S3 bucket, select
car-loan-bot-audio-logs
from the dropdown menu. - For KMS key, select
car-loan-bot-audio-logs-kms-key
from the dropdown menu. - For IAM role, select
LexCarLoanBotRole
from the dropdown menu. - Choose Save.
You can modify or disable Conversation logs settings for an alias by choosing the Settings gear icon. If you are enabling both text and audio logs, the LexCarLoanBotRole
must have write permissions to car-loan-bot-text-logs
log group and car-loan-bot-audio-logs
S3 bucket.
Marking sensitive slots as obfuscated
To protect sensitive data captured as slot values, you can enable slot obfuscation to mask those values for logging. To mask the SSN or last four digits of the SSN in the Conversation logs, mark them as obfuscated in Applyloan
, PayInstallment
, and CheckBalance
. Complete the following steps:
- On the Amazon Lex console, choose
AutoLoanBot
. - Under Intents, choose CheckBalance.
- Choose the Settings gear icon corresponding to the
SSNFourDigit
slot.
The popup SSNFourDigit settings appears.
- For Slot Obfuscation section, choose Store as {SSNFourDigit}.
- Choose Save.
- Choose Build.
- Choose Publish using the
Prod
alias.
Your bot is now ready to deploy.
Reviewing Conversation logs data
After you enable Conversation logs for the alias Prod
, the user input is saved in CloudWatch Logs log group in the following format, with the SSN slot obfuscated:
Similarly, you can navigate to the S3 bucket path present in s3PathForAudio
to review audio logs. The following screenshot shows the audio files stored in your S3 bucket
Improving bot performance with missed utterances
Now that you have set up Conversation logs and verified that user input is saved, you can use these logs to improve conversational experiences. Conversation logs provide you the details about user inputs that the bot didn’t recognize. These missed utterances can provide useful insights to better train your bot. Also, you can now prioritize new capabilities for your bot.
To generate a list of missed utterances for your AutoLoanBot
, complete the following steps:
- On the AWS Management Console, choose CloudWatch.
- Under Logs, choose Insights.
- From the dropdown list, choose
car-loan-bot-text-logs
. - To extract all the utterances mapped to the Fallback intent, run the following query:
Alternatively, if you do not have a
Fallback
intent configured for your bot, you can query for the fieldmissedUtterance
with the following code:
Diving deep into conversations
With Conversation logs, you have access to the entire interaction for all your sessions. As you go through the interactions, you can identify gaps, make design changes, and deploy better flows.
Amazon Lex uses the sessionId
attribute to log every entry that belongs to the same session. You can use sessionId
to drill down into a single conversation.
To monitor conversations across different sessions, complete the following steps:
- On the console, choose CloudWatch.
- Under Logs, choose Insights.
- From the dropdown list,
choose car-loan-bot-text-logs
. - To generate a list of unique
sessionId
and number of turns per conversation, run the following query:The following screenshot shows the output of this query. There are several
sessionId
queries and the number of utterances logged. - For each listed
sessionId
, view the complete conversation by running the following query:The following screenshot shows the output of this query. It displays both the user and bot interactions.
Conclusion
You can use Conversation logs to capture useful insights from user conversations and use these insights to improve your bot performance for enhancing user experience. You can also use the conversational data for auditing purposes. Get started with Conversation logs today!
About the Authors
Anubhav Mishra is a Product Manager with AWS. He spends his time understanding customers and designing product experiences to address their business challenges.
Hammad Mirza works as a Software Development Engineer at Amazon AI. He works on building and maintaining scalable distributed systems for Amazon Lex. Outside of work, he can be found spending quality time with friends and family.
Goutham Venkatesan works as a Software Development Engineer at Amazon AI. He works on enhancing the Lex customer experience building distributed systems at scale. Outside of work, he can be found traveling to sunny destinations & sipping coconuts on the beach.