Business Productivity

Analyze skill analytics using Metrics API and Amazon QuickSight

Alexa is probably already in your home, and now you can bring Alexa to your office too. Alexa for Business enables organizations and employees to use Alexa to get more work done. With Alexa for Business, employees can use Alexa as their intelligent assistant to be more productive in meeting rooms, at their desks, and even with the Alexa devices they already use at home or on the go.

All Alexa enabled devices have built-in capabilities like playing music, answering questions, providing weather forecasts, checking the news, etc. With Alexa Skills Kit developers can add new capabilities to Alexa, referred to as ‘skills’. Skills are like apps for Alexa and provide a new channel for your content and services. Organizations and individuals can publish custom skills to reach and delight customers at home and work. Customers can access these new abilities by asking Alexa questions or making requests. For example, you can use Alexa skill to order a car from Uber, order a pizza from Domino’s Pizza, control lights and other devices in the home, etc. You can also use Alexa for Business Blueprints to create a private skill for your workplace without writing a single code line.

Analytics are essential for measuring the success of your Alexa skill. Both enterprise customers who are using Alexa for Business and individual developers can check the key metrics like the number of unique users, sessions, total utterances, etc. in the Amazon developer console. The analytics provided by the developer console is sufficient for most companies. However, with the launch of the GetMetrics API, developers now can access the analytics data for their skill and create better visualization using third-party tools.

GetMetrics API is ideal for enterprises with multiple Alexa skills. They can now access and aggregate the analytics data in their custom data store and simultaneously visualize trends across many skills. Also, some enterprises have standard analytics and data visualization tools for all applications. GetMetrics API can enable them to comply with the company’s standards.

This blog post provides step-by-step guidance on accessing the data using GetMetrics API and creating a visualization using a sample application. To use this code, you need the skill ID from one of your skills. You can find this in the Alexa Skills Developer Console. Under each skill in your skill list, there’s a link to view the skill ID.

Process flows & architecture

As the above architecture diagram highlights, our application uses several AWS services. The Alexa Get Metrics API provides calculated metrics, insights, and advanced analytics reporting for skills usage. The GetMetrics API client is hosted on the AWS Lambda which lets you run code without provisioning or managing servers.  AWS Secrets Manager is used to store the credentials required to connect with the Alexa GetMetrics API. Amazon CloudWatch rule is set up to trigger the Lambda function on a set schedule. The skill metrics data retrieved using the Alexa GetMetrics API is stored on Amazon S3. Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. Amazon Quicksight is used to visualize skill metrics and trends in a dashboard.

Here are some graphs we created using Amazon Quicksight to show how an aggregated data would look for multiple Alexa skills.

 

Deployment steps

This section explains the step by step guide on how to deploy the solution.

Prerequisites

Before you deploy the solution using the AWS CloudFormation template in the upcoming steps, you must do the following:

1) Set up a Security Profile

2) Generate a refresh token

Open the terminal console on your computer. If you are a Windows user, you can launch the command prompt console using CMD or use PowerShell. MAC user, you can use the Terminal window.

  • Type the following command on your terminal window. This command uses the ASK CLI V2.
ask util generate-lwa-tokens --client-id <your client id> --client-confirmation <client secret> --no-browser --scopes alexa::ask:skills:readwrite
  • When prompted, provide the Client ID, and Client Secret you saved during the Security Profile step above. You should see the following JSON output.
{
 "access_token": "Atza|xxx", 
 "refresh_token": "Atzr|xx", 
 "token_type": "bearer",
 "expires_in": 3600,
 "expires_at": "2020-02-02T05:18:25.990Z"
 }

3) Create secrets for the Client ID, Client Secret, and Refresh Token

Next step is to create secrets for the Client ID, Client Secret, and Refresh Token using the AWS Secrets Manager.  You can follow this link for the easy steps.

  • Type key name as token.
  • For value field, enter the Client ID saved during the Setup Security Profile step above and select Next.
  • Type Secret Name as SECRET_CLIENT_ID 

  • Select Next, continue to default values until you click on Store.
  • Repeat the above process for:
    • SECRET_CLIENT_SECRET : This is the Client Secret retrieved while setting up the Security Profile.
    • SECRET_REFRESH_TOKEN : This is the Refresh Token retrieved during the “Generate a refresh token” step.

Deploy code and services

The next step is to deploy the architecture using a CloudFormation template.  Login to your AWS account and click on the Launch Stack button below to provision the resources and deploy the code for this solution in the us-east-1 region, then choose Next.

Note: Please make sure the value in the Stack Name field is in lowercase as the CloudFormation Stack Name is used for the S3 bucket name. You will receive an error if the S3 bucket name doesn’t meet the following requirements. You can use CloudFormation Macros to convert Stack Name to lower case, here is an example template if you choose to implement Macros.

Edit the following parameters in the CloudFormation template

1) Names of the skills for which you require data, you can add multiple skills here:

skillName = [“Skill1″,”Skill2”]  

 

2) Skill IDs of the skills provided in the step 2.

skillID = [“amzn1.ask.skill.xxx”,”amzn1.echo-sdk-ams.app.xxx”] 

 

3) Secret name for Client ID in Secrets Manager

client_id_config = “SECRET_CLIENT_ID” 

 

4)Secret Name for Client Secret in Secrets Manager

 client_secret_config = “SECRET_CLIENT_SECRET” 

 

5) Secret Name for Refresh Token in Secrets Manager

refresh_token_config = “SECRET_REFRESH_TOKEN” 

 

6) Stage of the skill live or development

stage = “live”

 

7) The aggregation period that is used when retrieving the metric. The possible values are SINGLE, PT15M, PT1H, P1D. For this blog post, we set the period to 1 day.

period = “P1D”

Optional parameters:

Here are some additional parameters you can set. If these values are not populated, the code refreshes data in S3 every day from the time Lambda is up and running.

startTime = “” # This should be in UTC Format. For example: 2019-01-01T00:00:00Z

endTime =”” # This should be in UTC Format. For example: 2019-01-06T00:00:00Z

The template creates the following resources:

Creating visualizations in Amazon QuickSight

Amazon QuickSight, a cloud-powered and fully managed service that lets you easily create and publish interactive dashboards, is used to visualize the data and trends. Before completing the following steps, add the “quickSight-analytics-userAWS IAM user to Amazon QuickSight. For instructions, see Managing User Access Inside Amazon QuickSight. For Role, choose AUTHOR.

Create a connection to the data source

1) Create a manifest.json file containing the file location of your S3 bucket skill-metrics-data-bucket.

See the example below for more information please visit here.

{
   "fileLocations":[
      {
         "URIPrefixes":[
            "https://s3.amazonaws.com/<bucket name provisioned by the CloudFormation Template>/"
         ]
      }
   ]
}

2) Log in to Amazon QuickSight as the quickSight-analytics-user IAM user

3) Choose New analysis

4) Choose New dataset

5) For the data source, choose S3

6) For Data Source Name, type in “alexa-skills-metrics

7) Upload the manifest.json that you have created in step 1

8) Click on Connect

9) Click on Visualize to complete the dataset setup.

Creating an analysis in Amazon QuickSight

When you complete the last step in the “Creating a connection to the data source” section above, you are ready to create your analysis. Next, you create a trend analysis of unique users for your Alexa skills, Skill 1 and Skill 2.

1) Click the Line Chart option from the Visual types panel in the bottom left of your screen.

2) Drag the timestamps field from the Fields list panel into the X-axis box in the Field wells section. Choose the aggregate value as Day.

3) Drag the uniqueCustomers field from the Fields list panel into the Value box in the Field wells section. Choose the aggregate value as Sum.

4) Drag the skillName field from the Fields list panel into the Color box in the Field wells section.

5) You should now see a line graph showing daily unique customer trends for both your skills Skill1, and Skill2.

Similarly, you can create visualizations for other trends like skill sessions, total utterances, failed utterances, etc.

Conclusion

In this post, we demonstrated how to access and aggregate Alexa skill analytics to a custom data store and create visualizations trends across multiple skills simultaneously. You can build many more visualizations as per your companies requirements using  Amazon Quicksight, for more information visit https://docs.aws.amazon.com/quicksight/latest/user/working-with-visual-types.html

References

SMAPI – Metrics API

Analyze Your Alexa Skill Metrics

Amazon QuickSight

 

Joyson Neville Lewis

Joyson Neville Lewis

Joyson obtained his Masters in Information Technology from Rutgers University in 2018. He has worked as a Software/Data engineer before diving into the Conversational AI domain in 2019 where he works with companies to connect the dots between business and AI using Voice and Chatbot solutions.

Shanthan Kesharaju

Shanthan Kesharaju

Shanthan Kesharaju is a Senior Product Manager in AWS Kumo organization. Shanthan has over a decade of experience managing diverse teams in both product and engineering. He is an award winning product manager and has built top trending Alexa skills. Shanthan has an MBA in Marketing from Duke University.

Praveen Modi

Praveen Modi

Praveen Modi works as Sr. Consultant with the Conversational AI team at AWS. In his role, he leads the design and development of specialized conversational AI solutions using Amazon Lex, Amazon Connect, and Alexa for AWS customers. Prior to joining Amazon, as an expert in digital marketing technologies, Praveen worked as a consultant for Fortune 500 companies such as Microsoft, Sony, Dell, and Kraft. Praveen's mantra for life is "You are never too old to set another goal or to dream a new dream."