AWS Media Blog

How-to: Create a Virtual Trainer With Amazon Sumerian Virtual Reality and Amazon Machine Learning – Part 1

  • Part 1: Setting up key AWS services like  Amazon Lex, AWS Lambda, and Amazon DynamoDB
  • Part 2: Creating and exploring Virtual Reality scenes in Amazon Sumerian

As technology evolved the transition from the web-based technologies to mobile has been transformative: we moved away from “Point-click-tab-type” to the “touch-swipe-talk” paradigm. Now AR/VR takes a further step by enabling experiences to build around natural modes of interactions such as gesture, gaze, posture etc. This shifts our focus from use of glass screen-based systems to more simulated world around us. Hence, we can say “AR/VR” is the new normal.

One of the key challenges that different industries like Education, Energy, Medical & Health, Manufacturing & Industrial, Aerospace & Transportation, Security & Defense will face over the next decade is to train the workforce with computer-savvy/gaming technology. These industries need to inject their workforce knowledge in a more efficient and cost-effective way. Training is a major cost for business, with a need to minimize the time and resources spent on training while improving or maintaining quality. Experiential learning using AR/VR solutions can actively engage learners in fast learning, better retention and improved decision making.

AWS provides a cost-effective way to build AR/VR applications using Amazon Sumerian. With Amazon Sumerian, you can build an immersive and highly engaging virtual trainer experience without the need of any additional device or complex virtual reality platform management.

In this two-part series, you will learn how to build a 3D scene using Amazon Sumerian by building the below scene. In part 1 you will learn to setup key AWS services like  Amazon Lex, AWS Lambda, and Amazon DynamoDB. By the end of part 2, you will have your own virtual trainer ready to teach about the AWS services used in this post. You can interact with the AWS trainer by opening the below Amazon Sumerian scene in the latest version of Chrome or Mozilla browser. You are going to build the below scene throughout this post.

Trainer Screen

Recommended Prerequisites

Before you begin, we recommend you familiarize yourself with following pre-requisite:

Architecture Diagram

This architecture utilizes Amazon Sumerian along with Amazon Lex and Amazon Polly with out-of-box integration provided by an Amazon Sumerian host. Amazon Polly provides text to speech capability and Amazon Lex is used for the questions and answers bot. Questions and Answers are stored in Amazon DynamoDB and Amazon Lex will pull those questions and answers, using Amazon Lambda.

The user will interact with Sumerian Host via voice command to get training and learn about different AWS Services.

AWS CloudFormation for setup

To start, you will have a knowledge base with a list of question and answers in Amazon DynamoDB, which later on Sumerian virtual trainer will use to interact with the audience.

To create required services, Launch AWS CloudFormation template by clicking on Launch stack button.

Make sure to select N. Virginia Region and acknowledge the IAM resource creation as shown in below screenshot, then click on create:

This AWS CloudFormation template will create following resources in your account:

  • An Amazon Cognito ID to integrate Amazon Sumerian scene with AWS services. Take note of it as you will use this later when building your scene.
  • An AWS Lambda function called “VirtualTrainerLambda”
  • An Amazon DynamoDB Table called” VirtualTrainerTable”
  • AWS IAM roles for accessing AWS Lambda function and Amazon DynamoDB Table.

You can click on resource tab and find the details:

Creating A Knowledge Base with Amazon DynamoDB

You need to build a knowledge base which the Amazon Sumerian host can use to answer any query. Here we chose Amazon DynamoDB to store questions and answers data.

When a user asks questions to the Amazon Sumerian host named “Luke”, the questions will be processed by Amazon Lex and, based on the intent, AWS Lambda will get the answers from the Amazon DynamoDB table.

Go to AWS console and Navigate to Amazon DynamoDB . You will find a table named “VirtualTrainerTable” in Amazon DynamoDB like the below screenshot:

 

You can load data into the table using Amazon CLI. If Amazon CLI is not installed on your machine follow these steps to install Amazon CLI. After installation make sure to configure your CLI with Access Key ID and Secret Access Key in account.

Now copy “VirtualTrainerTable.json” file from the StarterPack to your current directory and run the below command from that directory.

aws dynamodb batch-write-item --request-items file://VirtualTrainerTable.json

On successful execution, you will see the below output:

Learn more about loading data into DynamoDB using CLI. You can also load data into DynamoDB table using the console.

Now Explore your VirtualTrainerTable and see all question and answers listed here:

Creating Amazon Lex Bot for Q&A:

This scene uses Amazon Lex intents to parse user requests. Because of Amazon Lex’s natural language understanding capabilities, our Host can understand variants of a similar speech. For example, he can understand “tell me about Sumerian” and “tell me about entities” as the same request.

You can start with Amazon Lex conversation bot, by importing VirtualTrainerBot.zip from the StarterPack, and customize it as needed. Use importing a bot user guide to set up your Virtual Trainer Bot. After import your bot will show up in the Amazon Lex console as below:

Now open your bot by double clicking on “VirtualTrainerBot” hyperlink. Select “Greeting” Intent and Scroll down to the Fulfillment section. In this section,  select AWS Lambda function option and select “VirtualTrainerLambda” from Lambda function drop down and click on Save Intent button at the bottom, as shown in below screenshot:

Repeat above steps for other two intents “Questions” and “ThankYou”.

Now click on Build and publish your bot by clicking on the top left corner of Amazon Lex console screen as shown below. Make sure to give a meaningful alias while publishing the bot, for example,“prod”.

You can add Intents to expand the request types and Sample utterances to edit how your users can ask questions to invoke those intents.

Conclusion

In part 1 of this blog series, you learned how to build the required service in support of the Amazon Sumerian scene using key AWS services such as Amazon Lex, Amazon Cognito, Amazon DynamoDB and AWS Lambda. In part 2, you will learn how to create your virtual trainer using Amazon Sumerian and explore important component like Sumerian entities, State Machine, host behavior, Speech and Script component.

Additional Reading

Continue reading Part 2 of this post – Create Virtual trainer with Amazon Sumerian Virtual reality and Amazon Machine Learning Part-2. Also check out the re:invent 2018 session Create Industrial Digital Twin Using Amazon Sumerian, IoT and Analytics.

Chirag Oswal

Chirag Oswal

Chirag Oswal is a partner solutions architect and AR/VR specialist working with global systems integrators. He works with AWS partners and customers to help them adoption of cloud operating model at a large scale. He enjoys playing sports and travel.

Saurabh Shrivastava

Saurabh Shrivastava

Saurabh Shrivastava is a partner solutions architect and big data specialist working with global systems integrators. He works with AWS partners and customers to provide them with architectural guidance for building scalable architecture in hybrid and AWS environments. He enjoys spending time with his family outdoors and traveling to new destinations to discover new cultures