Automatically Create Machine Learning Models

To start a new SageMaker Autopilot experiment, click the + icon to access a new launcher window. In the Launcher window, scroll down to ML tasks and components. Click on the + icon for New Autopilot experiment.

Next, you’ll name your experiment. Click in the Experiment name box and type autopilot-experiment as the name.

 

Next, you’ll connect the experiment to data that is staged in S3. Click the box Enter S3 bucket location. In the S3 bucket address box, paste the following S3 path: s3://sagemaker-sample-files/datasets/tabular/uci_bank_marketing/bank-additional-full.csv

Leave the manifest file option set to Off. In the Target dropdown, select y as the target feature which our model will attempt to predict.

In the Output data location (S3 bucket) table, choose your own S3 bucket. In the Dataset directory name field, type sagemaker/tutorial-autopilot/output. This is where the output data will be saved once the experiment completes.

Leave the Auto deploy option on and the Auto deploy endpoint field blank. This will automatically deploy our model as an API endpoint and assign a name.

Next, a number of optional advanced settings allow you to manually set experimental parameters, such as the problem type, how the experiment is run, runtime details, IAM access, encryption, security, and others. Click the runtime button to show the optional settings.

For this tutorial, decrease the number of Max candidates from 250 to 5. This will run fewer models more quickly. A full experiment is the best approach for truly optimizing your model, but it can take hours to complete. For this tutorial, we will keep optional settings as is.

Click the Create Experiment button to start the first stage of the SageMaker Autopilot experiment. SageMaker Autopilot will begin to run through the phases of an experiment. In the experiment window, you can track progress through the phases of preprocessing, candidate definitions, feature engineering, model tuning, explainability, and insights. If you see a popup notification asking "Are you sure you want to deploy the best model?", click yes.

Once the SageMaker Autopilot job is complete, you can access a report that shows the candidate models, candidate model status, objective value, F1 score, and accuracy. SageMaker Autopilot will automatically deploy the endpoint.

Now that the SageMaker Autopilot experiment is complete, you can open up the top ranking model to obtain more details on the model’s performance and metadata. From the list of models, highlight the first one and right click to bring up model options. Click on Open in model details to review the model’s performance statistics.

In the new window, click on Explainability. The first view you see is called Feature Importance and represents the aggregated SHAP value for each feature across each instance in the dataset. The feature importance score is an important part of model explainability because it shows what features tend to influence the predictions the most in the dataset. In this use case, the customer duration or tenure and employment variation rate are the top two fields for driving the model's outcome.

Now, click on the tab Performance. You will find detailed information on the model’s performance, including recall, precision, and accuracy. You can also interpret model performance and decide if additional model tuning is needed.

Next, visualizations are provided to further illustrate model performance. First, look at the confusion matrix. The confusion matrix is commonly used to understand how the model labels are divided among the predicted and true classes. In this case, the diagonal elements show the number of correctly predicted labels and the off-diagonal elements show the misclassified records. A confusion matix is useful for analyzing misclassifications due to false positives and false negatives.

Next, look at the precision versus recall curve. This curve interprets the label as a probability threshold and shows the trade-off that occurs at various probability thresholds for model precision and recall. SageMaker Autopilot automatically optimizes these two parameters to provide the best model.

Next, look at the curve labelled Receiver Operating Characteristic (ROC). This curve shows the relationship between the true positive rate and the false positive rate over a variety of potential probability thresholds. A diagonal line represents a hypothetical model based on random guessing. The more this curve pulls to the upper left of the chart, the better the model will perform.

The dashed line represents a model with 0 predictive value, which is often called the null model. The null model would randomly assign a 0/1 label, and its area under the ROC curve would be 0.5, representing that it would be accurate 50% of the time.

Next, click on the tab Artifacts. You can find the SageMaker Autopilot experiment’s supporting assets, including feature engineering code, input data locations, and explainability artifacts.

Finally, click the Network tab. You will find information about network isolation and container traffic encryption.

Click the + icon to bring up a new Python notebook. Select Python3 as the kernel.

To know where to send the request, look up the model endpoint’s name. On the left pane, click the SageMaker Resources icon. In the SageMaker resources pane, select Endpoints. Click on the endpoint associated with the experiment name you created at the start of this tutorial. This will bring up the Endpoint Details window. Record the endpoint name and navigate back to the Python 3 notebook.

Copy and paste the following code snippet into a cell in the notebook, and press Shift+Enter to run the current cell. This code sets the environment variable ENDPOINT_NAME and runs inference. As the code completes, you will see a result consisting of the model label and associated probability score.

import os
import io
import boto3
import json
import csv

#: Define the endpoint's name.
ENDPOINT_NAME = 'autopilot-experiment-6d00f17b55464fc49c45d74362f284ce'
runtime = boto3.client('runtime.sagemaker')

#: Define a test payload to send to your endpoint.
payload = {
    "data":{
        "features": {
            "values": [45,"blue-collar","married","basic.9y",'unknown',"yes","no","telephone","may","mon",461,1,999,0,"nonexistent",1.1,93.994,-36.4,4.857,5191.0]
        }
    }
}

#: Submit an API request and capture the response object.
response = runtime.invoke_endpoint(
    EndpointName=ENDPOINT_NAME,
    ContentType='text/csv',
    Body=str(payload)
)

#: Print the model endpoint's output.
print(response['Body'].read().decode())

Congratulations! You have learned how to use SageMaker Autopilot to automatically train and deploy a machine learning model.

If you ran the CloudFormation template in Step 1 to create a new SageMaker Studio domain, continue with the following steps to delete the domain, user, and the resources created by the CloudFormation template.

To open the CloudFormation console, enter CloudFormation into the AWS console search bar, and choose CloudFormation from the search results.

In the CloudFormation pane, choose Stacks. From the status drop down list, select Active. Under Stack name, choose CFN-SM-IM-Lambda-catalog to open the stack details page.

On the CFN-SM-IM-Lambda-catalog stack details page, choose Delete to delete the stack along with the resources it created in Step 1.