Analyzing and optimizing Amazon Lex conversations using Dashbot

This post is co-written by Arte Merritt, co-founder and CEO of Dashbot. In their own words, “Dashbot is an analytics platform for chatbots and voice skills that enables enterprises to increase engagement, satisfaction, and conversions through actionable insights and tools.”

After you have deployed a bot, it is critical to analyze bot interactions, learn from this analysis, and use these learnings to improve the end-user experience. Conversational interfaces are easier to analyze than websites and mobile applications. You can infer user behavior directly from conversations instead of guessing what your users want by stitching together page views and choosing events. Your users are telling what they want in their own words and communicating their needs, aspirations, expectations, and frustrations. It’s up to you to listen, learn, and optimize your application from this valuable data.

We are excited to share that you can now easily integrate your Amazon Lex interface with Dashbot—an analytics and optimization platform built from the ground up to analyze and understand conversational applications. Dashbot provides the reports that you would expect from an analytics platform, such as user counts, retention metrics, and funnel tracking. Additionally, Dashbot provides several solutions built specifically for conversational applications, such as searchable transcripts, conversation paths, and phrase clusters. In this post, we describe how to analyze interactions with an Amazon Lex bot using Dashbot capabilities.

Solution Architecture

You will use an AWS Lambda function to stream conversation log data from Amazon CloudWatch Logs. You will set up an event to trigger a Lambda function from the log group. The Lambda function will post JSON data to your Dashbot account. With these resources in place, you can then create your dashboard in Dashbot.

This solution allows you to use your Amazon Lex conversation logs data to capture bot interactions and analyze them using Dashbot service capabilities. The image below shows the top messages for an intent:

Creating an account on Dashbot

To set up a Dashbot account, complete the following steps:

Navigate to dashbot.io.
Choose Sign Up. Note: You need to provide your email address, company name, and a password.
Choose Add ChatBot.
For Bot name, enter a name for your bot. In this post, we are using the “AutoLoanBot” example from a previous blog post.
For Platform, choose Lex.
You can optionally choose a Category and Production Status.
Choose Register.

The following screenshot illustrates the account setup with Dashbot.

After you register the chatbot, an API key is created. Record this API key to use in a future step.

Enabling conversation logs for Amazon Lex

To send your bot interactions to Dashbot, enable conversation logs text for your Amazon Lex bot. You can use the AutoLoanBot example created in the previous blog post as we are here, or you can use one of your own bots.

First, create a new log group called car-loan-bot-text-logs in Amazon CloudWatch Logs.
Next, create an IAM role called LexCarLoanBotRole to allow Amazon Lex to create log streams and post log events in the car-loan-bot-text-logs log group.
On the Amazon Lex console, choose the AutoLoanBot.
Choose Settings.
Choose Aliases, and make sure you have an alias for your bot. We will create an Alias name test that refers to the Bot version Latest.
If you have not yet published your bot alias, you need to do that now. Choose Publish, choose the alias (for example, test) in the Choose an alias dropdown, and then choose Publish.
Now Choose Conversation logs in the side menu.
Choose the Settings gear icon that corresponds to your bot alias.
Under Log type, choose Text logs.
For Log group name, select car-loan-bot-text-logs from the dropdown list.
For IAM role, select LexCarLoanBotRole from the dropdown list.
Choose Save.

Integrating your Amazon Lex conversation logs with Dashbot

Now that you’re logging your conversations to CloudWatch Logs, configure a subscription to send these messages to Dashbot. Complete the following steps.

Navigate to AWS Lambda service on the Console.
Choose Create function and leave Author from scratch selected.
Enter a Function name, such as lex-dashbot-integration.
Choose the latest supported version of Node.js in the Runtime language.
Choose Create a new role with basic Lambda permissions.
Choose Create function.

In Function code, for Code entry type select “Upload a file from Amazon S3” and enter this URL:

https://aws-ml-blog.s3.amazonaws.com/artifacts/lex-dashbot-integration/8231d7e369435c68383681d23df4011a

Change the handler name to src/index.handler.
In Environment variables, chooses Edit, Add environment variable, and add a variable with Key DASHBOT_API_KEY and for the Value use the Dashbot API Key you obtained above, and choose Save.
Choose Add trigger and select CloudWatch Logs for Trigger configuration.
Select the car-loan-bot-text-logs Log group from the drop-down list.
Enter a Filter name, such as dashbot-filter and choose Add.
Choose Save to save your Lambda function.

Testing your integration

When the setup is complete, you can test your integration by sending messages to your Amazon Lex bot. If you are using the AutoLoanBot example, you can download a test script to generate log data (conversation logs do not log interactions through the AWS Management Console). To download the test script, choose test-bot.zip.

After you send messages to your application, you can see them in the Live Transcripts section of Dashbot. For example, the following screenshot shows a transcript that includes the user’s utterances, the intent to which they were mapped, and the bot’s responses.

Using Dashbot over the lifecycle of your bot

After you have sent data to Dashbot for a few days, you can review the reports. This post shows retention reports, conversation paths, goal conversations reports, and phrase clusters. You can also view metric counts, funnels, and audience segmentation.

Retention reports

A retention report helps you understand how often your customers return to the application. The following screenshot shows the report Existing User Retention.

Conversation paths

Conversation paths let you see the most common paths your users take through your application. You can see where they successfully follow the expected path, and where they exit the flow. The following screenshot shows a conversation path for the AutoLoanBot example.

Goal conversations

Goal conversation tracking helps you understand how many users achieve their goal, such as completing a purchase or escalating a problem. The following screenshot shows an escalation report.

Phrase clusters

You can view your messages grouped by semantic meaning. The following screenshot shows how you can use the list of messages with the NotHandled intent to improve the training data for the ThankYou intent.

Summary

Dashbot provides a comprehensive set of reports you can use to analyze the performance of your Amazon Lex bots. You can quickly deploy an AWS Lambda function to securely send your Amazon Lex conversation logs to your Dashbot account in near real-time, and use Dashbot to optimize your conversational application needs. For more information about how to best analyze and optimize your bot, visit the Dashbot website or contact Dashbot directly.

About the authors

Arte Merritt is the CEO and co-founder of Dashbot – a chatbot analytics platform. Previously, he founded the mobile analytics platform, Motally, which he led to a successful exit via sale to Nokia. He has more than 20 years experience in the analytics space, working with companies like Yahoo Mobile and a wide variety of startups. Arte is an MIT alum.

As a Product Manager on the Amazon Lex team, Harshal Pimpalkhute spends his time trying to get machines to engage (nicely) with humans.

Brian Yost is a Senior Consultant with AWS Professional Services. In his spare time, he enjoys mountain biking, home brewing, and tinkering with technology.

AWS Machine Learning Blog