Testing accuracy and regression with Amazon Connect and Amazon Lex
Contacting customer service can often be a challenging experience. Sometimes, the conversation engagement does not meet the caller’s expectations. They may encounter experiences such as waiting on hold, repeating information from one agent to the next, and spending too much time getting answers to questions. These can all lead to a lengthy and often frustrating customer journey. Today, AI is playing a role in improving this customer experience in call centers to include engagement through chatbots — intelligent, natural language virtual assistants. These chatbots are able to recognize human speech and understand the caller’s intent without requiring the caller to speak in specific phrases. Callers can perform tasks such as changing a password, requesting a balance on an account, or scheduling an appointment. And they can do these tasks without speaking to an agent.
Amazon Lex is a service that allows you to create intelligent conversational chatbots. It lets you turn your Amazon Connect contact center contact flows into natural conversations that provide personalized experiences for your callers. Using the same technology that powers Amazon Alexa, an Amazon Lex chatbot can be attached to your Amazon Connect contact flow to recognize the intent of your caller, ask follow-up questions, and provide answers. Amazon Lex maintains context and manages the dialogue, dynamically adjusting the responses based on the conversation, so your contact center can perform common tasks for callers, to address many customer inquiries through self-service interactions. Additionally, Amazon Lex chatbots support an optimal (8 kHz) telephony audio sampling rate, to provide increased speech recognition accuracy and fidelity for your contact center interactions.
Amazon Lex chatbot development is an iterative process that requires tuning to optimize. As you change your bot configuration and evaluate new use cases, unit testing is recommended to capture performance metrics. Through capturing performance metrics, you can proactively anticipate the accuracy of new use cases and measure changes to existing components.
Missed utterances on the Monitoring tab of your Amazon Lex bot is a good metric to review regularly to ensure that your bot is configured/tuned to recognize customer utterances. Having a low number of missed utterances can mean that you’ve architected the experience in a way that’s intuitive to your users. However, missed utterances are not generally an effective indicator of bot accuracy. Incorrect intent matches and incorrect slot value resolution have a significant impact to the quality of experience. They need to be measured regularly as your bot, customer expectations, and their usage evolve. This blog post contains deployment materials and instructions that you can use to perform expected/actuals unit accuracy testing on Connect/Lex workloads with Amazon Connect, Amazon Lex, Amazon Polly, Amazon DynamoDB, AWS CloudFormation, and AWS Lambda.
The solution outlined in this blog allows you to call into your Amazon Connect contact flow, and provide a value to Amazon Lex. Amazon Polly plays back the value resolved by Amazon Lex to confirm it was detected correctly. If the value wasn’t detected correctly, you can enter an ID value that corresponds to the expected result by using the touch-tone input on your phone. This value is then looked up in DynamoDB to find the expected value. All of the resulting data is stored in DynamoDB for tracking and analysis, including aggregate-level accuracy rates. This allows you to quickly and easily collect expected/actual accuracy data for your Amazon Connect and Amazon Lex workloads.
Implementing the solution
To implement our accuracy and regression testing solution, we must:
- Import two new Amazon Lex bots.
- Use AWS CloudFormation to create the Lambda function and DynamoDB tables.
- Gather the Lambda ARN and add test scenarios.
- Grant Amazon Connect permission to use your Amazon Lex bots.
- Import, edit, save, and publish your contact flow.
- Claim a phone number and assign it to your contact flow.
- Call into your contact flow via touch-tone phone to test.
To follow the steps in this blog and create our own expected/actuals unit testing solution for Amazon Connect and Amazon Lex, we need:
- Account access to the AWS Management Console with sufficient permissions to use the Connect, Amazon Lex, Amazon Polly, DynamoDB, AWS CloudFormation, and Lambda services.
- An Amazon Connect instance.
- Google Chrome or Mozilla Firefox web browser to access Amazon Connect and the AWS Management Console.
Step 1: Import two new Amazon Lex bots
We use Amazon Lex bot import/export functionality for this step.
- Download the getPin Amazon Lex bot to capture the Amazon.Number slot type here.
- Download the yesNo Amazon Lex to capture yes/no confirmation here.
- Open the Amazon Lex console here.
- Choose the Bots tab, then choose Actions, Import.
- Choose Browse, choose the getPin bot that you downloaded earlier, and Import.
- Repeat the Import process for the yesNo bot that you downloaded earlier.
- After both Amazon Lex bots are imported, choose your getPin bot, and choose Build.
- After the build is complete, choose Publish, create a new alias named blog, and choose Publish.
- Repeat the Build and Publish processes for the yesNo bot.
Step 2: Use AWS CloudFormation to create the Lambda function and Amazon DynamoDB tables
In this step, we use AWS CloudFormation to automatically create the AWS Identity and Access Management (IAM) role, Lambda function, and Amazon DynamoDB tables needed. When the resources are created, permissions are granted to Connect to invoke the Lambda function. The function is used to look up expected values from the lexExpected table, store expected/actuals data to the lexExpectedActuals table, and aggregate real-time accuracy data to the lexTestResults table.
To create the Lambda function and DynamoDB tables:
- Open the AWS CloudFormation console here.
- Select the check box for I acknowledge that AWS CloudFormation might create IAM resources with custom names, and choose Create.
- Refresh the page until your stack has a Status of CREATE_COMPLETE.
- In the Outputs section, select and copy the value for LambdaARN. You need this ARN when editing contact flows, described later in this post.
Step 3: Add test scenarios
After the stack has the status of CREATE_COMPLETE, we can view the Lambda function and DynamoDB tables in the AWS Management Console to see what was created.
- In the AWS Management Console, select the Amazon DynamoDB service.
- Choose the lexExpected table.
- The lexExpected table is a list of test scenarios you’d like to cover with your Amazon Lex bot. You are prompted to enter the ID via touch-tone phone if you had an accuracy issue. The expectedValue is the value you provide your bot when you call in. You can reuse this table for other scenarios and slot types. For example, adding the value Mike for testing the AMAZON.US_FIRST_NAME slot type. For now, let’s add a test scenario for the AMAZON.Number slot type.
- Choose the Items tab, and choose Create Item.
- For the ID, enter 1, choose the +, select Append, and choose String.
- Enter expectedValue for the name, 12345 for the value, and choose Save.
- Repeat steps 5 and 6 for each test variation that you want to try.
- Export or take note of the lexExpected table contents to provide the corresponding ID when prompted, if there are any inaccuracies.
Step 4: Grant Amazon Connect permission to use your Amazon Lex bots
In this step, we add our getPin and yesNo Amazon Lex bots to our Connect instance, granting permission for Connect to access them.
- Open the Amazon Connect service here.
- In the left pane, choose Contact flows.
- On the right pane, choose the getPin Lex bot from the drop-down list, and choose + Add Lex bot.
- Select the yesNo Amazon Lex bot from the drop-down list, and choose + Add Lex bot.
- After both bots are added, they display in your instance, under Lex bots.
Step 5: Import, edit, save, and publish your contact flow
Start your contact flow import by downloading the contact flow from this location. This contact flow determines the experience we have when we call into our Amazon Connect instance.
- Open the Amazon Connect service here and select your instance.
- Choose Overview, and choose Login as administrator.
- On the Routing menu on the left side, choose Contact flows to show the list of contact flows.
- Choose Create contact flow.
- Next to the Save button, choose Import flow.
- Choose Select, navigate to the lexUnitTestingFlow file that you downloaded earlier, and choose Import.
- In your contact flow, choose the header of the Get customer Input block calling the getPin bot to edit it. Choose your getPin bot from the pick list, update the alias to blog, and choose Save to refresh the block.
- Repeat this step on the other Get Customer Input block for the yesNo bot.
- Choose the header of the Invoke AWS Lambda function block. Replace the ARN to match the ARN for your lexUnitTesting function that you gathered earlier.
- Choose Save & Publish.
Step 6: Claim a phone number and assign it to your contact flow
Let’s complete the final step necessary before calling into our contact flow to begin unit testing.
- Open the Amazon Connect service here.
- Choose Overview, and choose Login as administrator.
- On the Routing menu on the left side, select Phone numbers.
- Select Claim a number, choose a toll free or DID number, the country code you’d like to dial from your touch-tone phone for your tests, and choose the number you’d like to claim.
- Select your lexUnitTestingFlow Contact flow / IVR from the pick list, and choose Save.
You may need to wait up to 2–3 minutes for your changes to propagate. If you call your number and receive a network announcement, wait a few minutes and try again.
Step 7: Call into your contact flow using a touch-tone phone to test
Before calling in to start our tests, let’s review the steps you can follow to perform expected/actual unit testing. Reviewing this before starting your testing helps to make testing efforts more efficient, and ensure consistent input and execution. When ready, use the following steps:
- Secure a quiet location to perform your tests.
- Make sure that you are using a quality device and avoid using speaker phone as noise cancellation, ambient noise, and line quality can impact accuracy rates.
- Call from your touch-tone phone to the phone number claimed in Step 6.
- You are greeted with an Amazon Polly prompt of “Hello, please say your expected value now.” When prompted, you can respond with the expectedValue configured in your lexExpected DynamoDB table.
- After providing your expectedValue, an Amazon Polly prompt asks you to confirm with “Your value is X. Is this correct?” where X is what the Amazon Lex bot picked up from your utterance. You can then respond to this prompt with a “yes” or a “no.”
- If you respond with “yes,” indicating that the Amazon Lex bot successfully picked up your expectedValue, Amazon Polly plays back your testing statistics and overall accuracy rate before looping you back to step 4, where you can cycle through unit tests indefinitely.
NOTE: We recommend that you test each expectedValue in your lexExpected table at least 3–5 times for any potential variation in detection.
- If you respond with “no,” indicating that the Amazon Lex bot did not accurately detect the expectedValue provided, an Amazon Polly prompt plays “Please enter the ID for your expected value.” Using your touch-tone phone, enter the ID associated with your expectedValue in your lexExpected DynamoDB table.
- Amazon Polly plays back your testing statistics and overall accuracy rate before looping you back to step 4, where you can cycle through unit tests indefinitely.
- Review the contents of your lexTestResults and lexExpectedActuals DynamoDB tables for aggregate and granular level unit testing data.
When unit testing is complete, export the lexTestResults and lexExpectedActuals tables to .csv files from DynamoDB:
- Choose your table, and choose the Items tab.
- Select all items, choose Actions, Export to .csv.
- When exported successfully, delete the items by selecting all items, and choosing Actions, and Delete.
At this point, unit test data is backed up to .csv, DynamoDB tables are cleaned up, and you’re ready for another round of testing. Whenever you’re ready to test a different slot type, you can:
- Create a new custom Amazon Lex bot with similar Intent/Utterance/Slot configuration to getPin and change the Slot type to the one you’d like to test next.
- Build and Publish your Amazon Lex bot.
- Open the Amazon Connect console.
- In the left pane, choose Contact Flows.
- Choose your new Amazon Lex bot from the drop-down list, and choose + Add Lex bot.
- In the left pane, choose Overview, and choose Login as administrator.
- On the Routing menu on the left side, choose Contact flows.
- Select your lexUnitTestingFlow contact flow to edit it.
- Edit the first Get customer input block in your contact flow. Choose the header of the block, and change the configuration from getPin to point to your new Amazon Lex bot
- Select Save & Publish.
- Edit and add IDs and their associated expectedValues to your lexExpected table to test new utterance values.
- Continue testing.
In this post, we showed how to use Amazon Connect, Amazon Lex, Lambda, Amazon DynamoDB, AWS CloudFormation, and Amazon Polly to deploy a configurable solution that you use to measure expected and actual performance data for Amazon Connect and Amazon Lex workloads. With this, you can rapidly iterate and evolve existing Amazon Lex bots, measure regression of Amazon Lex bot architectural changes, determine word accuracy rates for given workloads, and gather data necessary to evaluate new use cases.
In the second part of this blog post, Automating word accuracy tests with Amazon Lex and Amazon Connect, we cover how to fully automate this solution to support scaling across tens of thousands of scenarios while increasing efficiency, reducing cost, and eliminating the need for manual testing.