Analyzing Amazon Lex conversation log data with Grafana

To support business and internal processes, organizations are increasing their use of conversational interfaces. They offer opportunities for more availability, improved service levels, and reduced costs. As these conversational services become more important, so, too, does the need to monitor performance and effectiveness of these interfaces with analytics and dashboards. This analysis, in turn, is used to drive continuous improvements in these interfaces.

Although there are tools such as Amazon CloudWatch Logs Insights to provide analytics of Amazon Lex conversation logs, it’s not always possible or practical for users to use these tools due to security or other requirements for console access. For these users, Grafana, with its built-in support for Amazon CloudWatch, offers a great alternative.

Solution architecture

In this post, we share a solution architecture that streams conversation logs from Amazon Lex to Amazon CloudWatch. We will configure and use Grafana to build visualization and reporting insights for missed utterances, user requests, and sentiment metrics.

Figure 1: Solution architecture

Step 1: Create or import a chatbot and publish an alias

In this post, we will use the OrderFlowers bot. You must import it and create an alias.
Complete the following steps:

1. Download the OrderFlowers bot definition.
2. In the Amazon Lex console, from the Actions drop-down menu, choose Import.
3. Choose the zip file you downloaded.
4. After you import the bot, choose OrderFlowers from the list of bots for your account.
5. Choose the Settings tab for your bot, and for Sentiment Analysis, choose Yes.
6. Choose Build.
7. When the build is complete, choose Publish.
8. Choose Create a new alias, and then enter a name for your alias. In this post, we use dev.
9. Choose Publish.

Step 2: Create the CloudWatch logs and role

The AWS CloudFormation template deploys the following resources:

• An AWS Identity and Access Management (IAM) role to allow Amazon Lex to stream to CloudWatch Logs.
• A CloudWatch Logs group, which is used in the following Grafana configuration.

To deploy the template, complete the following steps:

1. Choose 
2. Give your stack a unique name.
3. Enter a name for your IAM role and the CloudWatch Logs group or use the defaults.
4. Deploy the template.

After the deployment is complete, go to the next step. The deployment should take approximately one minute.

Step 3: Enable conversation logs in your Amazon Lex bot

Conversation logs are generated only when communicating with an Amazon Lex bot on an associated alias.

1. In the Amazon Lex console, open your bot page.
2. On the Settings tab, choose Conversation Logs.
3. Locate your bot alias (dev) and choose the Settings icon next to it.
4. On the next page, for Log Type, choose Text logs.
5. For Log Group, choose the value you selected for LexAnalyticsLogGroup during CloudFormation setup.
6. For IAM Role, choose the value you selected for LexAnalyticsToCWLRole during CloudFormation setup.

Figure 2: Resources page

To generate the conversation logs, you need to interact with the bot with a published alias. Conversation logs are not generated when you test the bot in the console. For more information, see Conversation Logs in the Amazon Lex Developer Guide. If you don’t have a client generating data on the Amazon Lex alias assigned in Step 1, you can generate some test data by interacting with your bot using a web UI. For instructions, see the Deploy a Web UI for Your Chatbot blog post

Step 4: Prepare to use Grafana to access CloudWatch logs

You need an access key ID and secret access key to configure Grafana access. If you do not have keys, use the following links to configure Grafana for data source access.

• Using Amazon CloudWatch in Grafana on the Grafana website.
• Creating an IAM user in your AWS account in the AWS Identity and Access Management User Guide.
• Managing access keys for IAM users in the AWS Identity and Access Management User Guide.

Step 5: Install and configure Grafana

1. On the Grafana website, choose Downloads, and then choose the Download
2. Choose the icon for your desktop operating system, and then follow the installation steps.
3. After installation is complete, in a browser, go to http://localhost:3000.
4. Sign in with admin/admin. You’ll be prompted to change the default password.

Step 6: Configure a data source

1. Under Configuration, choose Data Sources, and then choose Add data source.
2. In the search field, enter CloudWatch, and then choose Select
3. Enter the following information:
   • For Authentication Provider, choose Access & secret key from the dropdown.
   • For Access Key ID and Secret Access Key, enter the Access Key and Secret Access Key associated with your AWS account.
   • For Default Region, enter us-east-1.

   

   Figure 3: CloudWatch Details section
   

   The other fields are not required.

4. Choose Save & Test. Look for a success message that says the data source is working.

Step 7: Set up dashboard and panels

Figure 4 shows you the fields to complete.

1. In the left navigation pane, hover over the plus (+) sign, and then choose Dashboard.
2. Choose Add new panel.
3. Choose CloudWatch. For Query Mode, choose CloudWatch Logs. For Log Groups, use orderflowers-lex-bot-logs (created by the CloudFormation template in Step 2).
4. Enter the following query. This query provides unique user ID counts.

FIELDS @message

| stats count_distinct(userId) as Users

5. Choose a visualization for the query. For this first query, we use the Gauge visualization.
6. Choose Save.
7. For the dashboard name, enter OrderFlowers Bot Dashboard and then choose Save.

Figure 4: New Dashboard page in Grafana

At this point, the dashboard has one panel. Choose the Add panel button and then follow the previous steps to add panels for each query and visualization listed here.

Figure 5: Add panel button

Session, request, and transcript counts
Visualization: Gauge

CloudWatch query:

FIELDS @message

| stats count_distinct(sessionId) as Sessions, count_distinct(requestId) as Requests, count(inputTranscript) as Transcripts

 Missed utterance counts
Visualization: Gauge

CloudWatch query:

FIELDS @message

| filter missedUtterance = 1

| stats count(*) as Counts

Counts by intent (fulfilled)
Visualization: Bar gauge (vertical orientation). Explore Thresholds to set colors based on values.

CloudWatch query:

FIELDS @message

| filter dialogState like 'Fulfilled'

| stats count(intent) as Counts by intent

| sort Counts desc

Sentiment counts by label
Visualization: Bar gauge (horizontal orientation). Explore Thresholds to set colors based on values.

CloudWatch query:

FIELDS @message

| stats count(*) as Counts by sentimentResponse.sentimentLabel as Sentiment

| sort Counts desc

Missed utterance transcripts
Visualization: Table

CloudWatch query:

FIELDS @message

| filter missedUtterance = 1

| display inputTranscript

Negative Sentiment Transcripts
Visualization: Table

CloudWatch query:

FIELDS @message

| filter sentimentResponse.sentimentLabel like /NEGATIVE/

| display inputTranscript

After all the panels are configured, you can adjust visualizations, change colors, resize and move panels on the dashboard at any time. After the panels are set up, the dashboard should look like the following. Be sure to save the dashboard.

Figure 6: Completed dashboard

Conclusion

Conversational analytics are critical to evaluating bot performance and identifying opportunities for improvement. With an open source tool like Grafana, you can easily evaluate CloudWatch logs and gain key insights. In this post, we covered how to enable and use conversation logs to provide these insights and use Grafana to visualize and report on key metrics. Start building your dashboards and learn more about CloudWatch Log Insights and Grafana integration and Amazon Managed Service for Grafana.

About the Authors

Shanthan Kesharaju is a Senior Architect in the AWS ProServe team. He helps our customers with their Conversational AI strategy, architecture, and development. Shanthan has an MBA in Marketing from Duke University, MS in Management Information Systems from Oklahoma State University. He is also currently pursuing his third Masters in Analytics from Georgia Tech.

Cecil Patterson is a conversational AI consultant with AWS Professional services based in North Texas. He has many years of experience working with large enterprises to enable and support global infrastructure solutions. Cecil uses his experience and diverse skill set to build exceptional conversational solutions for customers of all types.