Analyzing Amazon Lex conversation log data with Amazon CloudWatch Insights

Conversational interfaces like chatbots have become an important channel for brands to communicate with their customers, partners, and employees. They help with faster service, 24/7 availability, and reduced service costs. By monitoring conversations between your customers and the bot, you can gain insights into user interactions, trends, and missed utterances. The additional insights will help your bot improve continuously. Whether you are a product owner looking for insights into user adoption or a conversation designer wanting to review missed utterances, a conversational analytics dashboard plays a vital role in serving these needs.

Solution architecture

The architecture shows the streaming of conversation logs from Amazon Lex to Amazon CloudWatch. We use Amazon CloudWatch Logs Insights to build the visualizations and get insights into user adoption, missed utterances, and user sentiment. The following diagram illustrates the architecture.

Most of the architecture is auto-deployed using an AWS CloudFormation template. There are a few manual steps, which we explain in the next section.

Implementing the solution

To implement this solution, you can either work with your existing Amazon Lex bot or import the sample OrderFlowers bot used in this post. If you plan to use your own bot, skip the following section and go to Creating the CloudWatch Logs Insights queries and dashboard.

Creating or importing a chatbot and publishing an alias

To use the OrderFlowers bot, you must import it and create an alias. Complete the following steps:

1. Download the OrderFlowers bot definition.
2. In the Amazon Lex console, from the Actions drop-down menu, choose Import.
3. Choose the OrderFlowers.zip file you downloaded.
4. After you import the bot, choose OrderFlowers from the list of bots for your account.
5. Choose the Settings tab for your bot, then set the Sentiment Analysis option to Yes.
6. Choose Build and wait for the process to finish.
7. Choose Publish.
8. Choose Create a new alias.
9. Enter a name for your alias. For this post, we use ver_one.
10. Choose Publish.

Creating the CloudWatch Logs Insights queries and dashboard

The AWS CloudFormation template deploys the following resources:

• An AWS Identity and Access Management (IAM) role to allow Amazon Lex to stream to CloudWatch Logs
• A CloudWatch Logs group
• CloudWatch Logs Insights queries
• A CloudWatch Logs dashboard

To deploy the template, complete the following steps:

1. Choose Launch Stack:  Launch Stack
2. Give your stack a unique name.
3. Enter a name for your IAM role and the CloudWatch Logs group.
4. Deploy the template.
5. This deployment should take approximately one minute to complete.

Enabling conversation logs in your Amazon Lex bot

You must make sure that the CloudFormation deployment is complete before completing this step. Additionally, conversation logs are only generated when communicating with an Amazon Lex bot on an associated alias.

1. In the Amazon Lex console, open your bot page.
2. In the Settings tab, choose Conversation Logs.
3. Locate your bot alias and choose the Settings icon next to it.
4. On the next page, for Log Type, choose Text logs.
5. For Log Group, choose the value you selected for LexAnalyticsLogGroup during CloudFormation set up.
6. For IAM Role, choose the value you selected for LexAnalyticsToCWLRole during CloudFormation set up.

To generate the Conversation Logs you will need to interact with the bot by interacting with a published alias. Conversation logs are not generated when testing the bot in the test console. You can learn about Conversation logs on the documentation page. If you don’t have a client generating data on the Amazon Lex alias assigned in the Conversation Logs step, you can generate some test data by interacting with your bot using a web UI. For instructions, see Deploy a Web UI for Your Chatbot.

Building CloudWatch Logs Insights and dashboard

The CloudFormation template deploys all the CloudWatch Logs Insights queries and a dashboard.

To view the dashboard, in the Amazon CloudWatch console, under Dashboards, choose Conversation-Analytics.

The following screenshot shows the Conversational Analytics dashboard. It comprises various insights like Summary, Sentiment, and Top Intents that help you understand the usage patterns of your chatbot.

In the following sections, we walk through how some of these insights were created and added to the dashboard. For more information about the query syntax, see Analyzing Log Data with CloudWatch Logs Insights.

Creating a Count widget

The Summary View section for your chatbot usage helps you get counts of users, sessions, and utterances for a selected time range.

For this post, we create a widget to see the number of unique users who used your bot in a selected time frame (1 hour).

1. In the CloudWatch console, under Logs, choose Insights.
2. For Select your log group(s), choose flowers-lex-bot-logs.
3. In the query box, enter the following code: FIELDS @timestamp, @message| stats count_distinct(userId) as Count_
4. Choose Run query.
5. You should see the query results in the bottom-right panel.
6. Choose Add to dashboard to add the Count widget to your dashboard.

Adding a visual insight

Some of the insights are better with visuals. For this post, we create a bar graph showing a breakdown of utterances by customer sentiment.

1. In the CloudWatch console, under Logs, choose Insights.
2. For Select your log group(s), choose flowers-lex-bot-logs.
3. In the query box, enter the following code: FIELDS @message| stats count(*) as Count_ by sentimentResponse.sentimentLabel as Sentiment
4. Choose Run query.
5. You should see the query results in the bottom-right panel.
6. Choose Visualization.
7. Choose Bar.
8. Choose Add to dashboard.

The following screenshot shows the bar graph on your dashboard.

To add any new custom widget to this dashboard, complete the same steps.

Additional widgets

In this section, we present a few more examples of the widgets and their corresponding CloudWatch Logs Insights queries.

User metrics

You can query to show user engagement based on number of requests as follows:

FIELDS @message
| stats count_distinct(requestId) as Requests by userId as User
| sort Requests desc
| limit 10

The following screenshot shows your query results.

Intent metrics

Intent metrics help identify the top N intents, which helps in staffing decisions at the contact centers serving phone and chat channels. These metrics help determine the top N intents that bot serves. See the following code:

|FIELDS @message
| filter dialogState like 'Fulfilled'
| stats count(intent) as Count_ by intent as intent
| sort Count_ desc
| limit 5

The following screenshot shows your query results.

Missed utterances

Missed utterance reports helps us identify utterances which the bot was not able to understand. These insights help you teach the bot to learn continuously and get better at answering questions. See the following code:

FIELDS @message
| filter missedUtterance = 1
| display inputTranscript

Cleaning up

To remove all resources created throughout this process and prevent additional costs, complete the following steps:

1. In the AWS CloudFormation console, choose the stack you created.
2. Choose Delete. This removes the resources the CloudFormation template created.
3. In the Amazon Lex console, choose the bot you created.
4. From the Actions drop-down menu, choose Delete. This removes your chat bot.

Conclusion

Whether you are a product owner, conversation designer, developer, or data scientist, conversational analytics are pivotal to understanding user adoption and teaching your bot to learn from its past mistakes. This post covered how to use conversation logs to capture useful insights from user conversations, and CloudWatch Log Insights to visualize them. Get started with CloudWatch Logs Insights and build your own custom analytics on top of your conversation logs.

About the Authors

Shanthan Kesharaju is a Senior Architect in the AWS ProServe team. He helps our customers with their Conversational AI strategy, architecture, and development. Shanthan has an MBA in Marketing from Duke University, MS in Management Information Systems from Oklahoma State University. He is also currently pursuing his third Masters in Analytics from Georgia Tech.

Blake DeLee is a Rochester, NY-based conversational AI consultant with AWS Professional Services. He has spent five years in the field of conversational AI and voice, and has experience bringing innovative solutions to dozens of Fortune 500 businesses. Blake draws on a wide-ranging career in different fields to build exceptional chatbot and voice solutions.