Creating virtual guided navigation using a Question and Answer Bot with Amazon Lex and Amazon Alexa

If our users are using Question and Answer Bot (QnABot), and they are getting the right answers for their questions, what else can you do for them? Now you can create a bot to guide them through a series of logically connected answers. This means, for example, that the bot can guide them through the steps of a recipe, or the bot can help them navigate through the exhibits in a museum tour. This avoids the problem of giving users too much textual information. Instead, by using the bot, it becomes easier to split the information your users need into smaller navigable steps.

In this blog post, we show you how to use new functionality in the QnABot to create a virtual tour that includes tour stops with descriptive text and images. Your tour can represent physical locations (such as rooms or exhibits), instructions (such as steps in a recipe or tutorial), or abstractions (such as topics or concepts in educational material).

Users can ask QnABot to start a virtual tour using a command like “Start tour,” and they can navigate between tour stops by asking for the next stop, or the previous stop. At each stop, QnABot will provide a summary, potentially with an accompanying image, and allow the user to ask follow-up questions to learn more if they feel so inclined. For example, “Tell me more about that desk” or “Who’s the artist of that painting?”

For this blog post, we’ll create a guided tour of the primary sites of the Prairie Line Trail by using the interactive map created by the City of Tacoma and funded by the Washington State Historical Society Capital Heritage Grant program.

AWS Lambda hooks provide you with the ability to add dynamic responses to your documents. By utilizing a new custom Amazon OpenSearch Service feature, those dynamic responses can be other documents. By leveraging the Lambda hook functionality, the Amazon ES functionality, document fields, and client-side attributes available in the QnABot system, we can add interconnectivity between documents and create modalities such as a guided tour.

Getting started

This blog post uses the Question and Answer Bot (QnABot) originally developed by Bob Strahan and John Calhoun. You should click this link and read the blog post to understand how QnABot works. This post includes a discussion of the new Lambda hook functionality, which we’re using here to implement guided navigation capability.

Deploying QnABot

In the AWS CloudFormation console, use this AWS CloudFormation button to launch QnABot in AWS Region us-east-1:

Note: At the time of writing, only the us-east-1 Region is supported.

The source code is available in our GitHub repository.

In the CloudFormation console, on the CloudFormation Parameters page, enter a valid Email and Username for the QnABot Content Designer admin user.

Choose Next twice to display the Review page.

Select the acknowledgement check box, and choose Create to deploy QnABot. (See the QnABot blog for more information about the resources deployed in your account.)

When the deployment is complete (after about 30 minutes), the master stack Output tab shows the following:

• ContentDesignerURL: URL to launch the Content Designer UI.
• ClientURL: URL to launch the user client webpage.

You will receive an email at the email address you provided with the subject “QnABot Signup Verification Code.” This email contains a generated temporary password that you can use to log in and create your own password. (Your new password must have a length of at least 8 characters, and contain upper-case and lower-case characters, plus numbers and special characters.)

Setting up Guided Navigation

1. In the CloudFormation console, launch the Content Designer webpage using the ContentDesignerURL link from the Outputs tab of the master CloudFormation stack.
   
2. Log in with the administrator username you provided when you launched the stack (default: ‘Admin’), and your new password.
3. If you want to import a full guided navigation example without going through the process of creating the documents, skip to the Full Guided Navigation Example that follows. Otherwise proceed to step 4.
4. In the QnABot Content Designer, open the Tools menu by choosing the ☰ icon in the top-left corner of the Designer and then select the Import submenu item.
   
5. Choose Examples/Demos to open the list of built-in examples. Locate the guided-navigation item and choose the LOAD button.
6. You’ll see an Import Jobs menu that will load the documents required for guided navigation. After a few seconds, you’ll see text that says “guided-navigation.json: Complete” in the Import Jobs This indicates that the import job has succeeded.
7. Navigate to the Tools menu mentioned in Step 3 and choose the Edit submenu to return to the main Content Designer page. You should see that the two documents, “Next” and “Previous”, have been loaded into the Designer. Let’s modify these default documents to work better for our Prairie Trails tour.
   
8. Click the Edit Button (?) located on the same line of the document with Id “Next” to update the “Next” Document. The ‘Questions’ section of this document contains phrases that will trigger navigation to next document in a sequence. The “Answer” field is used to define a message that will be displayed if there is no “Next” document in a sequence:
   1. Change “Next Room” to “Next Stop”
   2. Change “Let’s go the Next Room” to “Let’s go to the Next Stop”
   3. Change the answer “Unable to go to the next room…” to “Unable to go to the next stop….”
   4. Choose the UPDATE button located in the bottom-right corner to update the document.

   
   You may add/remove phrases in the Questions field, and change the default answer as you want. We will use the phrases in the previous screenshot for this blog post.

9. After the updating screen reads “Success!” choose the CLOSE
10. Repeat Steps 7 and 8 for the document with the ID “Previous.”
11. Choose the menu button ( ⋮ ) near the top-right corner of the Designer UI. Then choose the LEX REBUILD submenu item to rebuild and publish the underlying Lex Bot. This allows Amazon Lex to understand the new questions that you have entered. You’ll see a progress bar that displays “Success!” upon completion (this may take a few minutes). This rebuilding process allows Amazon Lex to recognize the new questions that have been added.
    

Creating a simple Guided Navigation sequence

Now that QnABot has the ability to navigate forward and backward using our preferred phrases, we’ll add our first set of documents. We’ll use the first three stops on the trail, Where the Rails Meet the Sails, Pressure + Flow, and Shipment to China. Keep these links open because they will be used in the following sections.

If you want to skip the document creation process detailed in steps 1-8, you can import this JSON document into the Designer by copying the URL, opening the Import submenu from the tools menu, pasting the URL in the “from url” section, and choosing the Import button.

1. Open the Designer UI. You should have the “Next” and “Previous” documents that were added in the previous section.
2. On the top-right side of the Designer UI, choose the ADD The Add New Item form should appear. In the document type section, select the qna document type, if it is not already selected.
   
3. Now add the basic information for each of these stops. When creating a document, we suggest using an Item ID that makes sense for the information in the Document. In our case, we use the stop name as the Item ID. We’ll also add Questions (or phrases) that users might ask, and the answer that will appear when those questions are asked. Both questions and phrases will be put into the Questions field. Use the following values for the form, example pictures are included below.
   1. Item ID: Where the Rails Meet the Sails
   2. Questions:
      1. Where the Rails Meet the Sails
      2. Rails Meet the Sails
      3. Start the tour
      4. Start
      5. Let’s begin
   3. Answer: This paired sculpture and mural features a finger and thumb clasping a steel needle, followed by a trail of stitches. The title takes its inspiration from a late-1800’s promotional slogan for Tacoma, and the mural elaborates on this era in Tacoma’s urban development. Together, these elements tell the story of the Northern Pacific Railroad finding its end at Commencement Bay and the growth of urban Tacoma. Ask for more about the art, the artist, or the medium for additional information.
      
      The previous answer document has a line that reads “Ask for more about the art, the artist, or the medium for additional information.” This line will be relevant in the following section when we create a more complex navigation sequence.
4. Choose the Advanced tab at the bottom of the form (you may need to scroll down on the form to see this tab). This tab contains extra features for your documents. In this basic example, we will only use the Next Document field, although other advanced fields may be used.
5. Fill in the Next Document field with the stop name/Item ID of the next stop on the tour, which is Pressure + Flow. When we create the next document, we will use Pressure + Flow as the Item ID. These fields are space and case sensitive.
   
6. Choose CREATE. Choose CLOSE in the Creating dialog box after it displays “Success!”
7. Choose the REFRESH button on the top-right side of the screen. You should see the new document in the list of documents
8. Repeat steps 2 – 7 for Pressure + Flow and Shipment to China using the basic information provided on the linked pages. The “Pressure + Flow” document should have “Shipment to China” listed in the Next Document field, while “Shipment to China” should not have that field filled in for now. Screenshots that show creating the Documents follow.
   1. Pressure + Flow
      1. Item ID: Pressure + Flow
      2. Questions:
         1. Pressure + Flow
         2. Pressure plus Flow
         3. Pressure and Flow
         4. Flow and Pressure
         5. Flow plus Pressure
      3. Answer: Commissioned by the City of Tacoma. A two-part installation, Pressure + Flow is a reflection on the power of technology and communication to transform a landscape. Pressure is a sculpture inspired by the inner workings of a steam engine, and Flow involves etched writing applied directly to the historic Prairie Line rails, excerpted from historic documents and letters. Ask for more about the art, the artist, or the medium for additional information.
      4. Under the Advanced tab
         1. Next Document: Shipment to China
            
            
   2. Shipment to China
      1. Item ID: Shipment to China
      2. Questions:
         1. Shipment to China
         2. Ship to China
         3. a shipment to China
         4. the shipment to China
      3. Answer: Commissioned by the Chinese Reconciliation Project Foundation in partnership with the City of Tacoma. This piece commemorates the early Chinese in America, their labors to construct the transcontinental railroads railroad construction, and their unjust suffering. It consists of an antique train truck and abstracted representation of 100 ash boxes, a reference to the Chinese workers who died while building the railroads. Ask for more about the art, the artist, or the medium for additional information.
         
         
9. Choose the menu button ( ⋮ ) near the top-right corner of the Designer UI, and then choose the LEX REBUILD submenu item to rebuild and publish the underlying Lex Bot. This allows Amazon Lex to understand the new questions that you have entered. You will see a progress bar that will display “Success!” upon completion (this may take a few minutes). This rebuilding process allows Amazon Lex to recognize the new questions that have been added.
   

Testing our simple guided navigation sequence

Your browser may prompt you to enable microphone access in the client. If you would like the option to use speech, you will need to allow microphone access.

1. Open the Tools menu by choosing the ☰ icon in the top-left corner of the Designer and choose the QnABot Client submenu item. This opens the QnABot Client in another browser tab. We will use it to test our navigation sequence.
   
2. In the client, either type or use the microphone to ask to “Start the tour.” Because we listed that phrase as one of the questions, you should see the answer for Where the Rails Meet the Sails.
3. Ask to go to the “Next Stop.” Since we linked the documents in the Designer, you should see the answer for Pressure + Flow.
4. Ask to go to the “Previous Stop.” You should again see the answer for Where the Rails Meet the Sails.
5. Ask to go to “Shipment to China.” You should see the answer for Shipment to China, the last stop on the tour.
6. Ask to go to the “Next Stop.” Since there is no next stop on the tour, you should see the default response, “Unable to go to the next stop…”
7. Ask to go to the “Previous Stop.” You should again see the answer for Where the Rails Meet the Sails. The Guided Navigation “Previous” functionality goes to the stop you were previously at (like the back button on a web browser), while the Guided Navigation “Next” functionality attempts to go the next stop on the tour by looking for the Document in the “Next” field. As of this blog post, the “Previous” function can store up to the last 10 documents that were returned.

Adding supporting documents to our Guided Navigation sequence

Now that we have basic answers for each of the three stops, we will add additional information about each stop through the use of supporting documents. The first part of this process is to assign a Topic to the document for each stop. If you would like to skip the document creation process detailed in steps 1-8, you may import this JSON document into the designer, by copying the URL, opening the Import submenu from the Tools menu , pasting the URL in the “from url” section, and choosing the Import button.

1. Navigate back to the QnABot Designer tab.
2. Navigate to the Edit Button (?) located on the same line of the document with Id “Where the Rails Meet the Sails” and choose it to update the document.
3. Scroll down on the update form and click to expand the Advanced In the Topic field, write “Where the Rails Meet the Sails”. Supporting documents that relate to “Where the Rails Meet the Sails” will use the same Topic.
   1. Note: You can assign any value to the Topic field as long as supporting documents use the same exact Topic. In this case, the name of the stop made sense for the Topic.
4. Click the UPDATE button in the bottom-right corner to update the document. You may need to scroll down to the bottom of the form to see it.
5. After the updating screen reads “Success!” choose the CLOSE button.

We will now create a supporting document for “Where the Rails Meet the Sails” with information about the artist.

6. On the top-right side of the Designer UI, choose the Add The Add New Item form should appear. In the document type section, select the qna document type if not already selected. We will use the “Meet the artists” section on the Where the Rails Meet the Sails webpage to fill out the form.
7. Use the following values for the form (without the quotation marks), example pictures are included below:
   1. Item ID: Where the Rails Meet the Sails: Meet the artists
   2. Questions:
      1. who made this
      2. who is the artist
      3. artist
      4. artist info
      5. artist information
   3. Answer: Rotator Creative is an agency located in Tacoma, working at the intersection of art, advertising, and community building. Lance Kagey is best known for his Beautiful Angle street-art posters. They use poetry, design, and antique typefaces to celebrate Tacoma, much like his first permanent public art piece. For Kagey, Where the Rails Meet the Sails is a meditation on how the world is an incredibly connected place. Mark Alvis, whose great grandfather worked as an advertiser in Tacoma, marvels at the opportunity to use his own design skills to commemorate the efforts of his ancestor and others like him. Scott Varga designs everything from websites, to hot rods, to industrial sculpture. He sees a direct connection between the coming of the railroad in 1873 and the proudly working-class, global city Tacoma has become. Visit their website at : http://www.rotatorcreative.com/
   4. Under the Advanced tab
      1. Topic: Where the Rails Meet the Sails
         
         
8. Repeat steps 2 – 7 for Pressure + Flow and Shipment to China using the information from the respective webpages and using the correct Topic. Examples follow.
   1. Pressure + Flow: Meet the artist
      1. Item ID: Pressure + Flow: Meet the artist
      2. Questions:
         1. who made this
         2. who is the artist
         3. artist
         4. artist info
         5. artist information
      3. Answer: Matthew Dockery is a Seattle-area industrial artist who works in metal, electronics, wood and textiles. The history of technology provides the inspiration for much of his work, and he has a special love for machines, gadgets, gears that mesh properly, and history. His work has been featured on the Discovery Channel, at Greenwich Observatory in London, at the annual Burning Man festival, and around the Pacific Northwest. Dockery describes his work as “living at the intersection of gears and mad science.” When asked how Pressure + Flow tells the Prairie Line’s story, Dockery replied, “In many ways, the railroad is Tacoma’s story. It was just another small town until the Northern Pacific made it their terminus.” Dockery insists that in many ways society is still moving to the frontier: “There are still such things as real estate bubbles and speculation, risk and reward, boom and bust.” Visit his website at : http://www.attoparsec.com/
      4. Under the Advanced tab
         1. Topic: Pressure + Flow
            
            
   2. Shipment to China: Meet the artist
      1. Item ID: Shipment to China: Meet the artist
      2. Questions:
         1. who made this
         2. who is the artist
         3. artist
         4. artist info
         5. artist information
      3. Answer: A native of China, Haiying Wu was a professional sculptor before emigrating to the US. He created this piece while a graduate student at the University of Washington. A prolific artist, his work can be found in Seattle, Lynnwood, Tacoma, Redmond, and in Chengdu, in the Sichuan province of China. According to Wu, “This piece shows the bitterness of the Chinese experience in America during that time, for the railroad built by their efforts was the same transportation used to carry them out of Tacoma.” Thanks to the Chinese Reconciliation Project Foundation for contributing this important piece for exhibition on the Prairie Line Trail, now displayed on the very tracks that the Chinese worked tirelessly to build. Visit their website at : http://www.tacomachinesepark.org/
      4. Under the Advanced tab
         1. Topic: Shipment to China
            
            

9. Choose the menu button ( ⋮ ) near the top-right corner of the Designer UI and choose the LEX REBUILD submenu item to rebuild and publish the underlying Lex Bot. This allows Amazon Lex to understand the new questions that you have entered. You’ll see a progress bar that displays “Success!” upon completion (this may take a few minutes). This rebuilding process allows Amazon Lex to recognize the new questions that have been added.
   

Testing supporting documents

Your browser may prompt you to enable microphone access in the client. If you want the option to use speech, you will need to allow microphone access.

1. Open the Tools menu by choosing the ☰ icon in the top-left corner of the Designer and choose the QnABot Client submenu item. This opens the QnABot Client in another browser tab. We’ll use it to test our navigation sequence.
   
2. In the client, either type or use the microphone to ask to “Start the tour.” Because we listed that phrase as one of the questions, you should see the answer for Where the Rails Meet the Sails.
3. Ask “who made this?” You should see the answer for Where the Rails Meet the Sails: Meet the artists.
4. Ask to go to the “Next Stop.” Since we linked the documents in the Designer, you should see the answer for Pressure + Flow.
5. Ask “who made this?” You should see the answer for Pressure + Flow: Meet the artist.
6. Ask to go to the “Next Stop.” Since we linked the documents in the Designer, you should see the answer for Shipment to China.
7. Ask “who made this?” You should see the answer for Shipment to China: Meet the artist.
8. Ask to go to the “Previous Stop.” You should again see the answer for Shipment to China.
9. Ask to go to the “Previous Stop.” You should see the answer for Pressure + Flow.
10. Ask to go to the “Previous Stop.” You should see the answer for Where the Rails Meet the Sails.

Adding branching paths to our guided navigation sequence

In some cases, you may want to offer users a choice in where they want to go next. Looking at the Prairie Line trail we can see that there are a few instances where this may be helpful.

Suppose that you are at the Joy Building, you may want to cross Pacific Avenue and go to Union Station. Or you may decide that Union Station is out of the way, and you would rather go directly to the West Coast Grocery instead. Although you can only specify one next stop to go to, you can use an extra document to prompt users to say the name of the stop that they want to go to next. A sample flow chart follows.

This example JSON document contains the previous example of branching. You can import this into the Designer by copying the URL, opening the Import submenu from the tools menu, pasting the URL in the “from url” section, and choosing the Import button. After importing the set of documents, perform a Lex Rebuild. Ask to go to the “Joy Building” to begin this sequence.

Full guided navigation example

A full example of guided navigation is available here. You can import this into the Designer by copying the URL, opening the Import submenu from the tools menu, pasting the URL in the “from url” section, and choosing the Import button. After importing the set of documents perform a Lex Rebuild.

This example uses the navigation techniques described earlier, along with the response card and markdown features described in the original QnABot Blog to combine imagery and text formatting to create a guided navigation experience. Ask to “Start the tour” to begin this sequence.

Guided Navigation deep dive

Guided navigation was built as a set of example Lambda hook functions that can be imported into the QnABot system. The Python code can be easily viewed in the AWS Lambda console for the Next-PY and Previous-PY functions (the function names will also have the Cloud Formation stack ID in the name). The two Python functions work in conjunction with the Query Lambda function to properly perform guided navigation. The code for all three functions can also be viewed/downloaded from our GitHub repository (Previous, Next, Query).

The guided navigation functions set session attributes in the client. In our implementation of guided navigation, we create a “navigation” session attribute in addition to the “topic” and “previous” session attributes that QnABot already keeps track of.

"navigation": {
    "next": "Welcome Figure",
    "previous": [
        "Where the Rails Meet the Sails",
        "Pressure + Flow",
        "Shipment to China"
    ],
    "hasParent": false
}

The “navigation” session attribute keeps track of “next”, “previous”, and “hasParent”.

Next

The next document in the sequence. The information is from the Next Document field on the latest document returned to the client. This field can be filled even if the user did not specify a “Next Document” – this is why the “hasParent” sub-attribute exists.

Previous

The previous documents returned to the client. Amazon Lex session attributes are limited to 12 KB as of the publication of this blog post, so we manage consumption by keeping track of the previous 10 documents returned to the client. Although we have not exceeded this limit yet, as more features are added to QnABot, more space may be used in the session attributes section. If this becomes an issue, Amazon DynamoDB could be used to store longer sequences and more attributes, although realistically users would most likely not backtrack more than 10 steps in a guided sequence.

hasParent

Whether or not a document has a parent, this attribute is opaque to content designers. In a previous section, we added supporting documents to QnABot that are intended to provide additional information about the parent document. Because the supporting documents do not specify a “Next Document”, guided navigation pulls information about the “Next Document” from the previous document returned to the client, and sets the “hasParent” flag to true. This is why users are able to go to the next room from supporting documents. Documents that have “hasParent” set to true are not added onto the stack of documents in the “Previous” field. Users would generally like to go back to the previous stop in the tour when they ask to go back, not to the additional information about that supporting document.

The guided navigation functions also make sure that a document is not listed multiple times in a row on the “Previous” sub-attribute stack. In the section about branching paths, the options document lists itself as its own “Next Document”. If a user were to say “next” multiple times on that document, it would be returned multiple times to the client. The document will only be added once to the stack.

Possibilities

You can integrate the tour feature with the regular features of QnABot or other custom functionalities. You could integrate trivia quizzes into your tour (which will be described in a future blog), and test your knowledge before and after the tour. Or you could implement a feedback system in which users can comment on the information in your bot. Guided navigation is only one of the features that you can add to your bot.

Although we only used the QnABot client in this example, you can also create an Amazon Alexa skill to offer guided tours. It is very easy to make an Alexa skill for your tours. The Tools menu in the Content Designer has an Alexa submenu item that will guide you creating an Alexa skill from your content. Response cards look great on the Amazon Echo Show or on the big screen TV using Alexa on FireTV.

Summary

This blog showcases the new guided navigation feature of QnABot. We provide examples of how to create your own guided navigation sequence, and we illustrate the different scenarios that you can create. We also explain some considerations that went into designing guided navigation.

We thank the City of Tacoma Historic Preservation Office for allowing us to use the Prairie Line Trail as an example for this blog post.

Additional reading

• Create a Question and Answer Bot with Amazon Lex and Amazon Alexa
• Create a Questionnaire bot with Amazon Lex and Amazon Alexa

About the Authors

Arun Donti is a Consultant for AWS Professional Services. Bob Strahan is a Senior Consultant for AWS Professional Services. John Calhoun is an Associate Solutions Architect for the AWS Public Sector Partners team. They work with our customers and partners to provide leadership on a variety of projects, helping them shorten their time to value when using AWS.