Activity Tracking with a Voice-Enabled Bot on AWS

Listen to this post

It’s New Year’s Eve. Your friends and loved ones have gone to the party, but you can’t go just yet because you haven’t figured out how to track the key performance indicators for your New Year’s resolution.

You’ve already divided your resolution into categories, and you’ve set personal targets for each category. Now you just need to log your activities and routinely calculate how you’re doing, so you stay on track. But you’ve been down this road before. You know that in a few short days log keeping will become tedious. You’ll start putting it off, and then you’ll forget. Before you know it, your resolution has gone the way of so many resolutions before it.

We’ve all been there. Right?

This year, you want a new way to log your activities. A fun and easy way, that will keep you engaged and prevent the procrastination that has so often proved disastrous. The midnight deadline is approaching – you need to implement this thing quickly so you can get to the party and celebrate the arrival of the New Year, secure in the knowledge that this year will be different!

In this post, we provide a solution to this perennial problem: a sample tracking bot application, called TrackingBot, which lets you log your activities by talking to it. The following flowchart shows the steps required to develop and use TrackingBot:

Although we use the example of tracking progress on a New Year’s resolution to illustrate our approach, you can apply the tools and techniques to many other goal setting and tracking scenarios. For example, you can easily customize the application to track how you’re spending your time, how much you’re exercising, how you’re feeling, what you’re eating, and so on.

To create the voice interface, we use Amazon Lex, an AWS service for building conversational interfaces into applications. We also use AWS Lambda for data validation and processing, Amazon Cognito for application authentication, and Amazon DynamoDB for data storage and retrieval. To orchestrate the build and deployment processes, we use a master AWS CloudFormation template. This allows you to install the complete sample application and “kick the tires” in minutes.

Example: Community involvement resolution

Every year, people resolve to get more involved in their communities. TrackingBot helps you define relevant categories, set targets, and track the activities you do in the service of your community. At the end of the year, you’ll be able to look back on your accomplishments, and experience the double satisfaction that comes from keeping your important resolution and playing with cool technology in the process!

Here are some example categories and targets for community involvement:

Volunteer my time. Target 2 hours each week.
Attend town hall and homeowners association meetings. Target one meeting each month.
Donate to local food banks. Target one donation each week.
Give money to local charities. Target $10 each day.
Participate in fundraising events. Target one event each month.

Activities that fall into the volunteer time category could include:

Mentor at school robotics club.
Coach school soccer team.
Help out during football games.

There are various ways that you might tell the application about your robotics club involvement, for example:

Today I volunteered at the school for 30 minutes.
I spent 30 minutes mentoring at the school today.
I volunteered at the robotics club for three hours yesterday.

We have captured all this information in a customizable JSON model file, which defines the information needed to automatically build and deploy all of the components of the TrackingBot application. Later, we show you how you can customize the application by modifying this model file.

Application architecture

TrackingBot has a serverless architecture that is based on the following AWS services:

Amazon Lex for building the conversational interface.
AWS Lambda to run data validation and processing functions.
Amazon DynamoDB to store and retrieve data.
Amazon Cognito to identify and authenticate end users.
Amazon S3 to host the generated Android application, web dashboard, and aws-lex-web-ui client.
AWS CodeBuild to run the build tools that create the bot, the web UI components, and the Android native mobile application.
Amazon CloudWatch to monitor builds and application usage.
AWS Identity and Access Management (IAM) to provide policies and roles that allow least-privilege access to Amazon services.
AWS CodePipeline to provide a continuous integration/continuous deployment (CI/CD) environment by monitoring the source repository and rebuilding and redeploying the application when the model file or source code is modified.
AWS CloudFormation to orchestrate provisioning of the overall environment.

The following diagram illustrates the application architecture:

Build TrackingBot

TrackingBot is available in GitHub. You use a CloudFormation template to build it.

To build TrackingBot:

Fork the public aws-ai-tracking-bot repository into your own GitHub account. If you don’t already have a GitHub account, you can create one for free.
To launch the CloudFormation template in the US East (N. Virginia) Region in your AWS account, choose Launch Stack:

Note: We tested TrackingBot in the AWS us-east-1 Region. If you choose a Region other than us-east-1, see the Region Table to verify that your region supports Amazon Lex.
Accept the default stack name: TrackingBot. Later, when you create new applications, you can use your own descriptive name.
Type your GitHub username and personal access token. (Generate a token with Repo scope on GitHub).
Choose Next twice to display the Review page.
Select the acknowledgement checkbox, and choose Create.

The CloudFormation stack creates a CI/CD pipeline using AWS CodePipeline. The pipeline accesses your GitHub repo and invokes AWS CodeBuild to process the TrackingBot JSON model file using our Node.js tracking application builder code. The application builder code creates or updates all of the resources required for the sample application, as follows:

The Amazon Lex bot, with all the intents, slots, and sample utterances needed for the sample application
AWS Lambda functions, which are called by the Amazon Lex bot, to validate and store inputs in Amazon DynamoDB
DynamoDB table or tables used to store raw and aggregated and categorized tracking information
The Amazon Cognito identity pool used by the UI application to authenticate and gain access to Amazon Lex and DynamoDB
AWS Identity and Access Management (IAM) roles and policies to grant least-privilege access to users and between services
Amazon S3 buckets, used to host the web dashboard and chat UI, and to store the Android application file
WebUI, which provides browser access to the sample application, hosted on S3
An Android application, which is built and available for download to your device

It can take about 15 minutes for CloudFormation to finish.

When you see CREATE_COMPLETE in the Status column for the main TrackingBot stack, choose Outputs. This tab has links to the AWS CodePipeline console, where you can monitor and troubleshoot your deployments; the downloadable Android application; and the web application dashboard.

Note

When the status of the main TrackingBot stack is CREATE_COMPLETE, the stack has successfully built the pipeline. To check on the status of the pipeline, navigate to the AWS CodePipeline console. The following screenshot shows the stages for a build that’s still in progress.

When the BuildDeploy stage has finished, you can launch the web UI or download the Android APK file for your Android table or smartphone.

Launching TrackingBot

You can interact with TrackingBot with either a web browser on your computer or the provided native application on your Android tablet or smartphone.

Launching TrackingBot in a web browser

You can launch TrackingBot from a Chrome, Firefox, or Microsoft Edge browser on your PC, Mac, Chromebook, or Android tablet.

To launch TrackingBot in a browser:

In the CloudFormation console, select the main TrackingBot stack and choose Output, and then choose the link to the WebAppDashboardURL.
When your browser requests access to the microphone on behalf of the web application, allow it. You’ll see the tracking dashboard and chat window:

You can interact with the bot using either text or voice.

Try it! Choose the microphone icon (on the bottom right) and say, “I gave three pairs of shoes to charity today.” When asked to confirm that your statement was understood correctly say “OK.” The dashboard accounts for your donation:

Downloading, installing, and launching the Android application

The TrackingBot native Android application delivers a simplified experience for your Android smartphone or tablet.

To download, install, and launch the TrackingBot Android application

On your device, open Settings, navigate to Security, and enable Unknown sources:
To download the trackingbot.apk file to your Android device, in the CloudFormation console, select the main TrackingBot stack and choose Output, and then choose the link for the ApkUrl.
Tip: To get the link onto your Android device, either access the CloudFormation console directly from your mobile browser, or email the URL to your Android device from your desktop.
If you see a warning like this, choose OK:
To install the TrackingBot Android application, open your Downloads folder (in Chrome, you can access Downloads from the top right menu), and choose trackingbot.apk:
Install the app, and then choose OPEN:
Give the app access to your device’s microphone by choosing ALLOW:
On the home screen, choose Record – Resolutions:
Choose the microphone icon, and then say “I gave nine dollars to charity today.” When asked to confirm, say “OK.”
From the menu (in the top left), choose Dashboard – Resolutions:
You’ll see that you have achieved 90% of your daily target for giving money:

To toggle between Daily, Weekly, and Monthly target views, use the buttons.
To switch the application between Dashboard view and Record view, use the menu.

Experimenting with TrackingBot

Using either the web UI or the Android app, experiment with reporting some of the following activities:

“I taught for 45 minutes at the school today.”
“I went to one meeting at the community association today.”
“I gave away one box of books.”
“I gave 20 dollars to the university yesterday.”
“I participated in the fundraiser last Sunday.”

As you confirm each activity, the dashboard shows your accumulated totals for the past day, week, or month against the target for the category. You can easily see if you’re on track to meet or exceed the target.

Now, try a variety of statements to see if the bot understands you correctly. You will find that the bot understands only certain activities and sentence structures.

You can delete activities you previously entered by saying phrases like:

“Clear data for today.”
“Clear data for yesterday.”
“Reset all.”

Component tour

Now that you’ve deployed and tried the sample application, let’s see how it works.

The model file

In GitHub, navigate to your aws-ai-tracking-bot repository, open the LexAppBuilder/model folder, and then open TrackingBotModel.json. This is the model definition for TrackingBot. It contains the categories with targets and activity reporting options that we defined.

By examining the contents of this model, you can see why TrackingBot understood some of your statements, but not others. You’ll also see that some categories have daily targets, while other have weekly or monthly targets. This defines the level of tracking that you see on the web UI Dashboard chart for each category.

By reviewing the file, you’ll be able to predict what’s likely to work. You also might think of ways to extend or modify the model to allow the bot to better understand the activities that you want to track.

The Amazon Lex bot

In the Amazon Lex console, explore TrackingBot.

To learn more about how Amazon Lex bots work, and to understand the concepts of intents, slots, prompts, sample utterances, AWS Lambda fulfillment functions, and more, see the Amazon Lex Developer Guide.

TrackingBot was built and deployed automatically by the code pipeline, based on the model file in your GitHub repo. To learn more about how to build your own bots directly in Amazon Lex, try some of the example bot tutorials from the documentation, and review the AWS AI blog post Building Better Bots Using Amazon Lex.

DynamoDB tables

Now open the DynamoDB console, and inspect the list of tables related to TrackingBot. The Raw table holds an item for each reported activity. The Aggregate table holds accumulated totals for each category for each day.

When you log a new activity through the web or Android UI, the Lambda functions that TrackingBot calls update these tables automatically. The data displayed in the UI Dashboard is pulled from these DynamoDB tables.

Amazon Cognito identity pool

Finally, open the Amazon Cognito console. Choose Manage Federated Identities, and choose TrackingBotCognitoPool.

There’s one identity in the pool (or, if you have accessed the application from more than one browser or Android device, more than one). The application uses the Amazon Cognito identity pool to obtain the temporary AWS credentials that you need to connect to Amazon Lex and DynamoDB. We use the Amazon Cognito unauthenticated identity method, which means you can use the application without needing to register and log in as a named user.

If you want to create a multiuser tracking application or access your tracking metrics from multiple devices, enhance the application to provide a registration and login page and use the Amazon Cognito authenticated method instead.

Customizing TrackingBot

You can customize various TrackingBot components to suit your scenario.

Customizing the application model

As you’ve seen, the application model file, TrackingBotModel.json, contains the categories and targets to track, along with the associated activity reporting options. To extend the set of items to track, or to define an entirely new tracking application with its own categories and activities, modify the model file. For more information about how the model is defined, see the LexAppBuilder README file in the GitHub repository.

When you commit changes to the model file and push the changes to your GitHub repo, the code pipeline in your AWS account detects the change and starts a new build. (The pipeline and build processes are defined in the master-cft.yaml CloudFormation template.)

Try customizing the bot:

Edit the model file, ./LexAppBuilder/model/TrackingBotModel.json, and then commit and push the changes to your GitHub repo. Start simple. For example, change the target value for a category, or extend a verb or object list to add a new activity to a category.
Navigate to the AWS CodePipeline console. Within a minute or two, the pipeline activates and starts a new build. TrackingBot is soon replaced by a new version reflecting your changes.

After you have mastered the model syntax, you can define and build entirely new applications. Track how often you walk your dog, how many cups of coffee you drink, how often your annoying headache recurs, and how much time you spend playing with your amazing tracking bot application!

Customizing the code

You can also customize and extend the capabilities of the ModelBuilder.js code, which generates TrackingBot from the model. For example, you could add additional error handling, or extend the voice interface to enable features like search, update, and more. Again, when you commit and push code changes to your repository, the pipeline automatically builds and redeploys the application.

To separate dev, test, and production code in your repository, consider a branching strategy. Use the master-cft.yaml CloudFormation template to set up separate code pipelines to produce different bots for each branch. This allows you to thoroughly test changes before promoting them to production.

Customizing the dashboard

You might want to add handy features to the web UI or mobile app. For example, add the ability to drill down from the dashboard summary charts to see the raw event detail. Or, add the ability to search the raw events to answer questions like “When did I last donate money to the hospital?” and so on.

Adding iOS voice support

You need to compile Apple iOS platform applications on a macOS environment. You can’t automatically build them in AWS. However, if you have iOS development skills and a build environment, you can build your own iOS application to work with TrackingBot. Consider using AWS Mobile Hub to generate source code for a skeleton client application that can interact with our bot.

Adding support for multiple users

To allow multiple users to track their activities using multiple devices, extend TrackingBot to support user authentication with Amazon Cognito. Consider trying the new Amazon Cognito built-in user sign up and sign in feature.

Adding backend analytics

To provide a rich source for individual or population-level analytics, use the DynamoDB tables that contain the raw and aggregate data for users’ tracked activities.

Integrating with other AWS services

Integrate and extend your tracking application, using the features of AWS services. Here are just a few suggestions:

To track usage and performance, integrate with Amazon Mobile Analytics.
To provide reminders and updates on progress, integrate with Amazon SNS mobile push notifications.
To augment voice tracking with the ability to upload and automatically tag images, integrate with Amazon Rekognition.
To publish an Alexa skill to offer Amazon Echo owners a hands-free alternative for voice updates, integrate with Amazon Alexa.

About the Amazon Lex web UI

You might have noticed that TrackingBot uses a git submodule to pull in the files from the aws-lex-web-ui GitHub repository. This repository provides the sample code for the web page IFrame that interacts with the Amazon Lex bot. It also handles audio capture and playback on webRTC-supported web browsers.

You can use the sample Amazon Lex web UI to provide an S3-hosted web browser UI for any Amazon Lex bot, not just our sample tracking bot. For more information, review the README file. Then try it out with one of the sample bots or with your own custom bot. The sample Amazon Lex web UI offers the choice of a standalone web page or an IFrame that you can incorporate into your own web site (as we did with the tracking application dashboard). To install it independently of TrackingBot, choose Launch Amazon Lex Web UI Stack and provide the name of your Amazon Lex bot when prompted.

Clean up

When you’re done experimenting, in the CloudFormation console, choose and delete the master TrackingBot stack. This removes the code pipeline, and all of the AWS resources associated with the bot.

Summary

We described how to use Amazon Lex and other AWS services to build a voice enabled tracking application that you can access from a web browser or Android application. The sample application gives you a chance to kick the tires. Try customizing the sample application to meet your own needs.

If you have questions or suggestions, please leave them as comments. We’d love to hear from you!

Additional Reading

Learn how to capture voice input in a browser and send it to Amazon Lex.

Explore the sample Amazon Lex web interface on Github.

About the Authors

Bob Strahan, Oliver Atoa and Bob Potterveld are Senior Consultants for AWS Professional Services. They work with our customers to provide leadership on a variety of projects, helping them shorten their time to value when using AWS.

BobStrahan_pic_resized

OliverAtoa_pic_resized

AWS Machine Learning Blog