Predictive User Engagement using Amazon Pinpoint and Amazon Personalize

(Edited November 25, 2019)—We’ve updated this post to include a link to the official Predictive User Engagement solution page.

Note: This post was written by John Burry, a Solution Architect on the AWS Customer Engagement team.

Predictive User Engagement (PUE) refers to the integration of machine learning (ML) and customer engagement services. By implementing a PUE solution, you can combine ML-based predictions and recommendations with real-time notifications and analytics, all based on your customers’ behaviors.

This blog post shows you how to set up a PUE solution by using Amazon Pinpoint and Amazon Personalize. Best of all, you can implement this solution even if you don’t have any prior machine learning experience. By completing the steps in this post, you’ll be able to build your own model in Personalize, integrate it with Pinpoint, and start sending personalized campaigns.

Using the AWS Solution

This solution is available as an official AWS Solution. For more information, see the Predictive User Engagement page on the AWS Solutions page.

The CloudFormation template in the Solution doesn’t include the training data. After you deploy the CloudFormation template, you can complete the steps below to create an ML model that you can use for demonstration purposes.

Prerequisites

Before you complete the steps in this post, you need to set up the following:

Create an admin user in Amazon Identity and Account Management (IAM). For more information, see Creating Your First IAM Admin User and Group in the IAM User Guide. You need to specify the credentials of this user when you set up the AWS Command Line Interface.
Install Python 3 and the pip package manager. Python 3 is installed by default on recent versions of Linux and macOS. If it isn’t already installed on your computer, you can download an installer from the Python website.
Use pip to install the following modules:
- awscli
- boto3
- jupyter
- matplotlib
- sklearn
- sagemaker
For more information about installing modules, see Installing Python Modules in the Python 3.X Documentation.
Configure the AWS Command Line Interface (AWS CLI). During the configuration process, you have to specify a default AWS Region. This solution uses Amazon Sagemaker to build a model, so the Region that you specify has to be one that supports Amazon Sagemaker. For a complete list of Regions where Sagemaker is supported, see AWS Service Endpoints in the AWS General Reference. For more information about setting up the AWS CLI, see Configuring the AWS CLI in the AWS Command Line Interface User Guide.
Install Git. Git is installed by default on most versions of Linux and macOS. If Git isn’t already installed on your computer, you can download an installer from the Git website.

Step 1: Create an Amazon Pinpoint Project

In this section, you create and configure a project in Amazon Pinpoint. This project contains all of the customers that we will target, as well as the recommendation data that’s associated with each one. Later, we’ll use this data to create segments and campaigns.

To set up the Amazon Pinpoint project

Sign in to the Amazon Pinpoint console at http://console.aws.amazon.com/pinpoint/.
On the All projects page, choose Create a project. Enter a name for the project, and then choose Create.
On the Configure features page, under SMS and voice, choose Configure.
Under General settings, select Enable the SMS channel for this project, and then choose Save changes.
In the navigation pane, under Settings, choose General settings. In the Project details section, copy the value under Project ID. You’ll need this value later.

Step 2: Create an Endpoint

In Amazon Pinpoint, an endpoint represents a specific method of contacting a customer, such as their email address (for email messages) or their phone number (for SMS messages). Endpoints can also contain custom attributes, and you can associate multiple endpoints with a single user. In this example, we use these attributes to store the recommendation data that we receive from Amazon Personalize.

In this section, we create a new endpoint and user by using the AWS CLI. We’ll use this endpoint to test the SMS channel, and to test the recommendations that we receive from Personalize.

To create an endpoint by using the AWS CLI

At the command line, enter the following command:
```
aws pinpoint update-endpoint --application-id <project-id> \
--endpoint-id 12456 --endpoint-request "Address='<mobile-number>', \
ChannelType='SMS',User={UserAttributes={recommended_items=['none']},UserId='12456'}"
```
In the preceding example, replace <project-id> with the Amazon Pinpoint project ID that you copied in Step 1. Replace <mobile-number> with your phone number, formatted in E.164 format (for example, +12065550142).

Note that this endpoint contains hard-coded UserId and EndpointId values of 12456. These IDs match an ID that we’ll create later when we generate the Personalize data set.

Step 3: Create a Segment and Campaign in Amazon Pinpoint

Now that we have an endpoint, we need to add it to a segment so that we can use it within a campaign. By sending a campaign, we can verify that our Pinpoint project is configured correctly, and that we created the endpoint correctly.

To create the segment and campaign

Open the Pinpoint console at http://console.aws.amazon.com/pinpoint, and then choose the project that you created in Step 1.
In the navigation pane, choose Segments, and then choose Create a segment.
Name the segment “No recommendations”. Under Segment group 1, on the Add a filter menu, choose Filter by user.
On the Choose a user attribute menu, choose recommended-items. Set the value of the filter to “none”.
Confirm that the Segment estimate section shows that there is one eligible endpoint, and then choose Create segment.
In the navigation pane, choose Campaigns, and then choose Create a campaign.
Name the campaign “SMS to users with no recommendations”. Under Choose a channel for this campaign, choose SMS, and then choose Next.
On the Choose a segment page, choose the “No recommendations” segment that you just created, and then choose Next.
In the message editor, type a test message, and then choose Next.
On the Choose when to send the campaign page, keep all of the default values, and then choose Next.
On the Review and launch page, choose Launch campaign. Within a few seconds, you receive a text message at the phone number that you specified when you created the endpoint.

Step 4: Load sample data into Amazon Personalize

At this point, we’ve finished setting up Amazon Pinpoint. Now we can start loading data into Amazon Personalize.

To load the data into Amazon Personalize

At the command line, enter the following command to clone the sample data and Jupyter Notebooks to your computer:
```
git clone https://github.com/markproy/personalize-car-search.git
```
At the command line, change into the directory that contains the data that you just cloned. Enter the following command:
```
jupyter notebook
```
A new window opens in your web browser.
In your web browser, open the first notebook (01_generate_data.ipynb). On the Cell menu, choose Run all. Wait for the commands to finish running.
Open the second notebook (02_make_dataset_group.ipynb). In the first step, replace the value of the account_id variable with the ID of your AWS account. Then, on the Cell menu, choose Run all. This step takes several minutes to complete. Make sure that all of the commands have run successfully before you proceed to the next step.
Open the third notebook (03_make_campaigns.ipynb). In the first step, replace the value of the account_id variable with the ID of your AWS account. Then, on the Cell menu, choose Run all. This step takes several minutes to complete. Make sure that all of the commands have run successfully before you proceed to the next step.
Open the fourth notebook (04_use_the_campaign.ipynb). In the first step, replace the value of the account_id variable with the ID of your AWS account. Then, on the Cell menu, choose Run all. This step takes several minutes to complete.
After the fourth notebook is finished running, choose Quit to terminate the Jupyter Notebook. You don’t need to run the fifth notebook for this example.
Open the Amazon Personalize console at http://console.aws.amazon.com/personalize. Verify that Amazon Personalize contains one dataset group named car-dg.
In the navigation pane, choose Campaigns. Verify that it contains all of the following campaigns, and that the status for each campaign is Active:
- car-popularity-count
- car-personalized-ranking
- car-hrnn-metadata
- car-sims
- car-hrnn

Step 5: Create the Lambda function

We’ve loaded the data into Amazon Personalize, and now we need to create a Lambda function to update the endpoint attributes in Pinpoint with the recommendations provided by Personalize.

The version of the AWS SDK for Python that’s included with Lambda doesn’t include the libraries for Amazon Personalize. For this reason, you need to download these libraries to your computer, put them in a .zip file, and upload the entire package to Lambda.

To create the Lambda function

In a text editor, create a new file. Paste the following code.

# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

AWS_REGION = "<region>"
PROJECT_ID = "<project-id>"
CAMPAIGN_ARN = "<car-hrnn-campaign-arn>"
USER_ID = "12456"
endpoint_id = USER_ID

from datetime import datetime
import json
import boto3
import logging
from botocore.exceptions import ClientError

DATE = datetime.now()

personalize           = boto3.client('personalize')
personalize_runtime   = boto3.client('personalize-runtime')
personalize_events    = boto3.client('personalize-events')
pinpoint              = boto3.client('pinpoint')

def lambda_handler(event, context):
    itemList = get_recommended_items(USER_ID,CAMPAIGN_ARN)
    response = update_pinpoint_endpoint(PROJECT_ID,endpoint_id,itemList)

    return {
        'statusCode': 200,
        'body': json.dumps('Lambda execution completed.')
    }

def get_recommended_items(user_id, campaign_arn):
    response = personalize_runtime.get_recommendations(campaignArn=campaign_arn, 
                                                       userId=str(user_id), 
                                                       numResults=10)
    itemList = response['itemList']
    return itemList

def update_pinpoint_endpoint(project_id,endpoint_id,itemList):
    itemlistStr = []
    
    for item in itemList:
        itemlistStr.append(item['itemId'])

    pinpoint.update_endpoint(
    ApplicationId=project_id,
    EndpointId=endpoint_id,
    EndpointRequest={
                        'User': {
                            'UserAttributes': {
                                'recommended_items': 
                                    itemlistStr
                            }
                        }
                    }
    )    

    return

In the preceding code, make the following changes:

Replace <region> with the name of the AWS Region that you want to use, such as us-east-1.
Replace <project-id> with the ID of the Amazon Pinpoint project that you created earlier.
Replace <car-hrnn-campaign-arn> with the Amazon Resource Name (ARN) of the car-hrnn campaign in Amazon Personalize. You can find this value in the Amazon Personalize console.

Save the file as pue-get-recs.py.
Create and activate a virtual environment. In the virtual environment, use pip to download the latest versions of the boto3 and botocore libraries. For complete procedures, see Updating a Function with Additional Dependencies With a Virtual Environment in the AWS Lambda Developer Guide. Also, add the pue-get-recs.py file to the .zip file that contains the libraries.

Open the IAM console at http://console.aws.amazon.com/iam. Create a new role. Attach the following policy to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "personalize:GetRecommendations",
                "mobiletargeting:GetUserEndpoints",
                "mobiletargeting:GetApp",
                "mobiletargeting:UpdateEndpointsBatch",
                "mobiletargeting:GetApps",
                "mobiletargeting:GetEndpoint",
                "mobiletargeting:GetApplicationSettings",
                "mobiletargeting:UpdateEndpoint"
            ],
            "Resource": "*"
        }
    ]
}

Open the Lambda console at http://console.aws.amazon.com/lambda, and then choose Create function.
Create a new Lambda function from scratch. Choose the Python 3.7 runtime. Under Permissions, choose Use an existing role, and then choose the IAM role that you just created. When you finish, choose Create function.
Upload the .zip file that contains the Lambda function and the boto3 and botocore libraries.
Under Function code, change the Handler value to pue-get-recs.lambda_handler. Save your changes.

When you finish creating the function, you can test it to make sure it was set up correctly.

To test the Lambda function

On the Select a test event menu, choose Configure test events. On the Configure test events window, specify an Event name, and then choose Create.
Choose the Test button to execute the function.
If the function executes successfully, open the Amazon Pinpoint console at http://console.aws.amazon.com/pinpoint.
In the navigation pane, choose Segments, and then choose the “No recommendations” segment that you created earlier. Verify that the number under total endpoints is 0. This is the expected value; the segment is filtered to only include endpoints with no recommendation attributes, but when you ran the Lambda function, it added recommendations to the test endpoint.

Step 7: Create segments and campaigns based on recommended items

In this section, we’ll create a targeted segment based on the recommendation data provided by our Personalize dataset. We’ll then use that segment to create a campaign.

To create a segment and campaign based on personalized recommendations

Open the Amazon Pinpoint console at http://console.aws.amazon.com/pinpoint. On the All projects page, choose the project that you created earlier.
In the navigation pane, choose Segments, and then choose Create a segment. Name the new segment “Recommendations for product 26304”.
Under Segment group 1, on the Add a filter menu, choose Filter by user. On the Choose a user attribute menu, choose recommended-items. Set the value of the filter to “26304”. Confirm that the Segment estimate section shows that there is one eligible endpoint, and then choose Create segment.
In the navigation pane, choose Campaigns, and then choose Create a campaign.
Name the campaign “SMS to users with recommendations for product 26304”. Under Choose a channel for this campaign, choose SMS, and then choose Next.
On the Choose a segment page, choose the “Recommendations for product 26304” segment that you just created, and then choose Next.
In the message editor, type a test message, and then choose Next.
On the Choose when to send the campaign page, keep all of the default values, and then choose Next.
On the Review and launch page, choose Launch campaign. Within a few seconds, you receive a text message at the phone number that you specified when you created the endpoint.

Next steps

Your PUE solution is now ready to use. From here, there are several ways that you can make the solution your own:

Expand your usage: If you plan to continue sending SMS messages, you should request a spending limit increase.
Extend to additional channels: This post showed the process of setting up an SMS campaign. You can add more endpoints—for the email or push notification channels, for example—and associate them with your users. You can then create new segments and new campaigns in those channels.
Build your own model: This post used a sample data set, but Amazon Personalize makes it easy to provide your own data. To start building a model with Personalize, you have to provide a data set that contains information about your users, items, and interactions. To learn more, see Getting Started in the Amazon Personalize Developer Guide.
Optimize your model: You can enrich your model by sending your mobile, web, and campaign engagement data to Amazon Personalize. In Pinpoint, you can use event streaming to move data directly to S3, and then use that data to retrain your Personalize model. To learn more about streaming events, see Streaming App and Campaign Events in the Amazon Pinpoint User Guide.
Update your recommendations on a regular basis: Use the create-campaign API to create a new recurring campaign. Rather than sending messages, include the hook property with a reference to the ARN of the pue-get-recs function. By completing this step, you can configure Pinpoint to retrieve the most up-to-date recommendation data each time the campaign recurs. For more information about using Lambda to modify segments, see Customizing Segments with AWS Lambda in the Amazon Pinpoint Developer Guide.

AWS Messaging & Targeting Blog