AWS Machine Learning Blog

Detect anomalies using Amazon Lookout for Metrics and review inference through Amazon A2I

Proactively detecting unusual or unexpected variances in your business metrics and reducing false alarms can help you stay on top of sudden changes and improve your business performance. Accurately identifying the root cause of deviation from normal business metrics and taking immediate steps to remediate an anomaly can not only boost user engagement but also improve customer experience.

As the volume of data monitored by your business grows, detecting anomalies gets challenging. On March 25, 2021, AWS announced the general availability of Amazon Lookout for Metrics, a service that uses machine learning (ML) to automatically detect anomalies that are most important to businesses with greater speed and accuracy, and identifies their root cause.

ML models often need human oversight to retrain and continuously improve model accuracy. In this post, we show how you can set up Lookout for Metrics to train a model to detect anomalies. We then use a human-in-the-loop workflow to review the predictions using Amazon Augmented AI (Amazon A2I), and use the feedback to improve model accuracy.

Solution overview

Lookout for Metrics uses ML to automatically detect and diagnose anomalies (outliers from the norm) in business and operational data, such as a sudden dip in sales revenue or customer acquisition rates. In a couple of clicks, you can connect Lookout for Metrics to popular data stores like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, and Amazon Relational Database Service (Amazon RDS), as well as third-party software as a service (SaaS) applications such as Salesforce, ServiceNow, Zendesk, and Marketo, and start monitoring metrics that are important to your business.

Lookout for Metrics automatically inspects and prepares the data from these sources to detect anomalies with greater speed and accuracy than traditional methods used for anomaly detection. You can also provide feedback on detected anomalies to tune the results and improve accuracy over time. Lookout for Metrics makes it easy to diagnose detected anomalies by grouping together anomalies that are related to the same event and sending an alert that includes a summary of the potential root cause. It also ranks anomalies in order of severity so you can prioritize your attention to what matters most to your business.

Amazon A2I is an ML service that makes it easy to build the workflows required for human review. Amazon A2I brings human review to all developers, removing the undifferentiated heavy lifting associated with building human review systems or managing large numbers of human reviewers, whether running on AWS or not.

To get started with Lookout for Metrics, we create a detector, create and attach a dataset with metrics that you want to train on and monitor, activate the detector, and view anomalies. Following these steps, we show how you can set up a human review workflow using Amazon A2I. Finally, it updates the detector with human review feedback, which helps retrain the model and further improve accuracy.

Architecture overview

The following diagram illustrates the solution architecture.

The solution has the following workflow:

  1. Upload data from your source to Amazon S3.
  2. We run Lookout for Metrics in continuous mode to process data continuously from the Amazon S3 path.
  3. Inference results are stored in Amazon S3
  4. When Lookout for Metrics detects anomalies, the inference input and outputs are presented to the private workforce for validation via Amazon A2I.
  5. A private workforce investigates and validates the detected anomalies and provides feedback.
  6. We update the results with corresponding feedback from human loop through Amazon A2I.
  7. Updated feedback improves accuracy of future trainings.

In the accompanying Jupyter notebook downloadable from GitHub, we walk you through the following steps:

  1. Generate a synthetic dataset.
  2. Create a detector and map measures and dimensions to metrics.
  3. Activate the detector.
  4. Detect anomalies.
  5. Set up Amazon A2I to review predictions from Lookout for Metrics.
  6. Update the model based on output from human loop through Amazon A2I.

Prerequisites

Before you get started, complete the following steps to set up the Jupyter notebook:

  1. Create a notebook instance in Amazon SageMaker.
  2. When the notebook is active, choose Open Jupyter.
  3. On the Jupyter dashboard, choose New, and choose Terminal.
  4. In the terminal, enter the following code:
git clone 
https://github.com/aws-samples/amazon-lookout-for-metrics-a2i-integration.git
  1. Open the notebook for this post from next_steps/A2I_Integration:
Amazon_L4M_Detector_A2I.ipynb

You’re now ready to run the notebook cells. Run the environment setup step to set up the necessary Python SDKs and libraries, S3 bucket, and Region that we use throughout the notebook. Make sure that the SageMaker Studio IAM role has the necessary permissions.

NOTE: The code uses Python 3.7. Please use the Python 3 (Data Science) kernel for this notebook.

Generate synthetic data

In this section, we generate synthetic data for the detector and for predicting anomalies. The metrics data is aggregated on an hourly basis, and the detector runs in continuous mode in real time, every hour. In this example, we use the detector to detect anomalies. Starting from the current date, we generate data for 6 months in the past and 3 days in the future. The historical data is used for training the model, and we use the current and future data for predicting anomalies on an ongoing basis. Note the following configuration:

  • Historical data is created as a CSV file called ./data/ecommerce/backtest/input.csv
  • Hourly data files are stored in the folder ./data/ecommerce/live/<yyyyMMdd>/<HH:mm>/<yyyyMMdd_HH:mm:ss>.csv
  • Complete data along with the anomaly labels is available in ./data/ecommerce/label.csv

Access the notebook section generate synthetic data and run the cells. Then inspect the DataFrame:

backtest_df = pd.read_csv('data/ecommerce/backtest/input.csv')
backtest_df.head()

Data points are generated in a random manner, so your response might look different than the following screenshot.

Save the data to the S3 bucket previously created:

%%time
!aws s3 sync {DIR_PATH}/{DATASET_NAME}/ s3://{bucket_name}/{DATASET_NAME}/ —quiet —delete

Create a detector and map measures and dimensions to metrics

A detector is a Lookout for Metrics resource that monitors a dataset and identifies anomalies. To detect outliers, Lookout for Metrics builds an ML model that is trained with your source data. This model is automatically trained with the ML algorithm that best fits your data and use case. Access the notebook section Create Lookout for Metrics Detector and run the cell to create a detector:

if len(ecom_anomaly_detector_arn) == 0:
    # Detector for ecommerce example does not exists. Create the anomaly detector.
    create_anomaly_detector_response = L4M.create_anomaly_detector( 
        AnomalyDetectorName = ecom_anomaly_detector_name,
        AnomalyDetectorDescription = "Anomaly detection on a sample ecommerce dataset.",
        AnomalyDetectorConfig = {
            "AnomalyDetectorFrequency" : FREQUENCY,   
        },
    )

You get the following response:

Anomaly Detector ARN: arn:aws:lookoutmetrics:[REGION]:[ACCOUNT NUMBER]:AnomalyDetector:ecommerce-continuous-detector

Measures are the primary fields that the detector monitors. You can also configure up to five additional fields as dimensions. Dimensions are secondary fields that create subgroups of measures based on their value.

In this ecommerce example, views and revenue are our measures and platform and marketplace are our dimensions. You may want to monitor your data for anomalies in number of views or revenue for every platform, marketplace, and combination of both.

Each combination of measure and dimension is called a metric. Measures, dimensions, and metrics map to datasets, which also contain the Amazon S3 locations of your source data, an AWS Identity and Access Management (IAM) role that has both read and write permissions to those Amazon S3 locations, and the rate at which data should be ingested from the source location (the upload frequency and data ingestion delay). Run the cells in the Measures and Dimensions section to create a metric set for your detector that points to the live data in Amazon S3:

create_metric_set_response = L4M.create_metric_set( ** params )
ecom_metric_set_arn = create_metric_set_response["MetricSetArn"]

You get the following response:

Metric Set ARN: arn:aws:lookoutmetrics:<REGION>:<ACCOUNT-NUMBER>:MetricSet/ecommerce-continuous-detector/ecommerce-metric-set-1

Activate your detector

Now it’s time to activate your detector. During activation, the model is trained with historical data that was generated in a previous cell and stored in the ./data/ecommerce/backtest folder. Run the cells under Activate the Detector:

if ecom_detector_status in ["INACTIVE", "ACTIVATING"]:
    
    # Activate the detector
    if ecom_detector_status == "INACTIVE":
        L4M.activate_anomaly_detector(AnomalyDetectorArn = ecom_anomaly_detector_arn)
    
        print("\nActivating ecommerce example Detector.")

You get the following response:

Activating ecommerce example Detector.
Detector status: ACTIVATING
...
Detector status: ACTIVATING
...
Detector status: LEARNING
...
Detector status: ACTIVE

You can also check the status of your detector on the Lookout for Metrics console.

Detect anomalies

In this section, you review the anomalies found by the detector. We have created a continuous detector that operates on live data. It expects to receive input data every hour. We already generated some data into the future, which is in the ./data/ecommerce/live folder. Run the cell from the notebook section Fetch Anomalies to detect anomalies:

....
    if next_token:
        params["NextToken"] = next_token

    response = L4M.list_anomaly_group_summaries(**params )
    
    print("Anomaly group summaries:\n {}".format(response))

    anomaly_groups += response["AnomalyGroupSummaryList"]
    print('\ntype of AnomalyGroupSummaryList: {}'.format(type(anomaly_groups)))

You get the following response:

You may have to wait for the detector to run at the top of the hour to detect anomalies. So, if no anomalies are found when running the next cell, you may want to come back later and run it again.

Set up Amazon A2I to review predictions from Lookout for Metrics

In this section, you set up a human review loop for low-confidence detection in Amazon A2I. It includes the following steps:

  1. Create a private workforce.
  2. Create a human task UI.
  3. Create a human task workflow.
  4. Send predictions to Amazon A2I human loops.
  5. Complete your review and check the human loop status.

Create a private workforce

You must create a private workforce on the SageMaker console. For instructions, see Create an Amazon Cognito Workforce Using the Labeling Workforces Page. Once created, note the ARN of the workforce and enter its value in the notebook cell:

WORKTEAM_ARN = 'your private workforce team ARN'

Create a human task UI

You now create a human task UI resource, giving a UI template in liquid HTML. This HTML page is rendered to the human workers whenever a human loop is required. For over 70 pre-built UIs, see the amazon-a2i-sample-task-uis GitHub repo.

Follow the steps provided in the notebook section Create a human task UI to create the web form and initialize the Amazon A2I APIs:

...
if not describe_human_task_ui_response:
    # Create the human task UI
    create_human_task_ui_response = sagemaker_client.create_human_task_ui(
        HumanTaskUiName=l4m_taskUIName,
        UiTemplate={'Content': ecom_a2i_template}) 

    print("\nCreate human task ui response: ")
    pprint.pprint(create_human_task_ui_response, width = 2)
...

You get the following response:

Human task UI ARN: arn:aws:sagemaker:[REGION]:[ACCOUNT NUMBER]:human-task-ui/l4m-ecommerce-ui

Create a human task workflow

Workflow definitions allow you to specify the following:

  • The worker template or human task UI you created in the previous step.
  • The workforce that your tasks are sent to. For this post, it’s the private workforce you created in the prerequisite steps.
  • The instructions that your workforce receives.

This post uses the Create Flow Definition API to create a workflow definition. The results of the human review are stored in an S3 bucket, which can be accessed by the client application. Run the cell Create a Human task Workflow in the notebook:

...
if not describe_flow_definition_response:
    create_workflow_definition_response = sagemaker_client.create_flow_definition(
        FlowDefinitionName = l4m_flowDefinitionName,
        RoleArn=sagemaker_role_arn,
        HumanLoopConfig= {
            "WorkteamArn": workteam_ARN,
            "HumanTaskUiArn": l4m_review_ui_arn,
            "TaskCount": 1,
            "TaskDescription": "Review the anomalies detected by Amazon Lookout for Metrics",
            "TaskTitle": "Ecommerce Anomalies Review"
        },
        OutputConfig={
            "S3OutputPath" : s3_output_path
        }
    )
...

You get the following response:

S3 output path: s3://[ACCOUNT NUMBER]-[REGION]-lookoutmetrics-lab/ecommerce/a2i-results
Flow definition Arn: arn:aws:sagemaker:[REGION]:[ACCOUNT NUMBER]:flow-definition/l4m-ecommerce-workflow

Send predictions to Amazon A2I human loops

Run cells in the start human review loop section to find the URL of a portal for providing feedback on anomalies. Open the URL in a browser and log in with the credentials of the human review worker. You should have sent an invitation email to a worker for joining the work team when you created the work team on the Amazon A2I console. Complete the review from the portal and inspect the output:

....
start_human_loop_response = a2i_client.start_human_loop(
            HumanLoopName=humanLoopName,
            FlowDefinitionArn=flowDefinitionArn,
            HumanLoopInput={
                "InputContent": json.dumps(ip_content)
            }
        )

print("\nStart human loop response: ")
#pprint.pprint(start_human_loop_response, width=2)

Complete the review and check the human loop status

Complete your review and check the human loop status with the following code:

...
try:
    describe_human_loop_response = a2i_client.describe_human_loop(HumanLoopName=humanLoopName)
    print("\nDescribe human loop response: ")
    pprint.pprint(describe_human_loop_response, width=2)
    
    completed_human_loops_s3_output = describe_human_loop_response["HumanLoopOutput"]["OutputS3Uri"]
    print("HumanLoop Status: {}".format(describe_human_loop_response["HumanLoopStatus"]))
except:
    print("Error getting human loop")


#print("\nHumanLoop Name: {}".format(humanLoopName))
#print("HumanLoop Status: {}".format(describe_human_loop_response["HumanLoopStatus"]))
print("\nOutput in S3 at: \n{}".format(describe_human_loop_response["HumanLoopOutput"]["OutputS3Uri"]))

You get the following response:

HumanLoop Status: Completed

Output in S3 at: 
s3://[ACCOUNT NAME]-[REGION]-lookoutmetrics-lab-2/ecommerce/a2i-results/l4m-ecommerce-workflow/2021/06/22/05/35/13/51e352ff-38f3-4154-a162-1fb6661462da/output.json

Update the detector based on human feedback from Amazon A2I

In this section, we review our results and update the detector to improve prediction accuracy. Refer to the accompanying notebook in GitHub for detailed steps to add human loop. We check whether the human review detected an anomaly:

while anomaly_col_name in review_result:
    
    #tseriesid = review_result['tseriesid-' + str(col_name_suffix)]    
    
    print("\n{}".format(str(col_name_suffix)))
    
    is_anomaly = review_result[anomaly_col_name]['on']
    print("Is Anomaly: {}".format(is_anomaly))

If the review results indicated an anomaly, get the corresponding time series ID and anomaly group ID from the DataFrame and update the training set using the put feedback API call:

    row_value = df_anomalies_by_ts.loc[(df_anomalies_by_ts['timestamp'] == timestamp) & (df_anomalies_by_ts['marketplace'] ==marketplace) & (df_anomalies_by_ts['platform'] ==platform)]
    #print("Row :{}".format(row_value))

    ....
    ....
    put_feedback_response = L4M.put_feedback(
            AnomalyDetectorArn=ecom_anomaly_detector_arn,
            AnomalyGroupTimeSeriesFeedback={
                'AnomalyGroupId': anomaly_group_id,
                'TimeSeriesId': tseriesid,
                'IsAnomaly': is_anomaly}
    )

You can now retrain your model using the updated dataset to improve your model accuracy.

Clean up

Run the clean-up resources cell to clean up the resources that you created. Because we created a continuous detector, it continues to run one time every hour and incur charges until it is deleted.

Conclusion

In this post, we walked you through how to use Lookout for Metrics to train a model to detect anomalies, review diagnostics from the trained model, review the predictions from the model with a human in the loop using Amazon A2I, augment our original training dataset, and retrain our model with the feedback from the human reviews.

With Lookout for Metrics and Amazon A2I, you can set up a continuous prediction, review, train, and feedback loop to audit predictions and improve the accuracy of your models. Use the GitHub repo to access the notebook used in this post.

About the Authors

Neel Sendas is a Senior Technical Account Manager at Amazon Web Services. Neel works with enterprise customers to design, deploy, and scale cloud applications to achieve their business goals. He has worked on various ML use cases, ranging from anomaly detection to predictive product quality for manufacturing and logistics optimization. When he isn’t helping customers, he dabbles in golf and salsa dancing.

Ashish Rawat is a Senior Solutions Architect at Amazon Web Services, based in Atlanta, Georgia. Ashish provides architecture guidance to enterprise customers and helps them implement strategic industry solutions on AWS. He is passionate about AI/ML and Internet of Things.