Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”
Sign in
Sign in
or
Create a new account
About
Categories
Delivery Methods
Solutions
Resources
 Your Saved List Become a Channel Partner Sell in AWS Marketplace Amazon Web Services Home Help

Amazon Sagemaker

Amazon SageMaker is a fully-managed platform that enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale. With Amazon SageMaker, all the barriers and complexity that typically slow down developers who want to use machine learning are removed. The service includes models that can be used together or independently to build, train, and deploy your machine learning models.

product logo

Model Performance Estimation - NannyML Free trial

By:
nannyML
Latest Version:
1.0.0
Estimate the performance of your classification and regression models in production, without ground truth
Continue to Subscribe

    Product Overview

    In production ground truth is often delayed or absent. Traditional data drift detection techniques are noisy and do not only alert to changes that impact model performance. Performance estimation allows you to estimate performance metrics (ROC-AUC, F1, RMSE, etc) without ground truth. Giving you a single metric to monitor, optimize and communicate about your models in production. Some specific examples of when you could benefit from estimating your performance include: When predicting loan defaults, to estimate model performance before the end of the repayment periods. In demand forecasting, the ground truth demand will only be known after the forecast window has passed. Esimating performance lets you know how your model is performaning in real time. When performing sentiment analysis, targets may be entirely unavailable without significant human effort, so estimation is the only feasible way to attain metrics.

    Key Data

    Version
    1.0.0
    Usage Instructions
    Release Notes
    Show other versions
    By
    nannyML
    Categories
    ML Solutions
    Monitoring
    Type
    Algorithm
    Fulfillment Methods
    Amazon SageMaker

    Highlights

    • Estimate the performance of machine learning models in production when targets are absent or delayed.

    • NannyML supports Confidence Based Performance Estimation (CBPE) for performance estimation of binary and multiclass classification models.

    • NannyML supports Direct Loss Estimation (DLE) for performance estimation of regression models.

    Not quite sure what you’re looking for? AWS Marketplace can help you find the right solution for your use case. Contact us

    Pricing Information

    Use this tool to estimate the software and infrastructure costs based your configuration choices. Your usage and costs might be different from this estimate. They will be reflected on your monthly AWS billing reports.

    Contact us to request contract pricing for this product.


    Estimating your costs

    Choose your region and launch option to see the pricing details. Then, modify the estimated price by choosing different instance types.

    Version
    Region

    Software Pricing

    Algorithm Training$9/hr

    running on ml.m5.large

    Model Realtime Inference$14.00/hr

    running on ml.m5.large

    Model Batch Transform$14.00/hr

    running on ml.m5.large

    Infrastructure Pricing

    With Amazon SageMaker, you pay only for what you use. Training and inference is billed by the second, with no minimum fees and no upfront commitments. Pricing within Amazon SageMaker is broken down by on-demand ML instances, ML storage, and fees for data processing in notebooks and inference instances.
    Learn more about SageMaker pricing

    SageMaker Algorithm Training$0.115/host/hr

    running on ml.m5.large

    SageMaker Realtime Inference$0.115/host/hr

    running on ml.m5.large

    SageMaker Batch Transform$0.115/host/hr

    running on ml.m5.large

    About Free trial

    Try this product for 7 days. There will be no software charges, but AWS infrastructure charges still apply. Free Trials will automatically convert to a paid subscription upon expiration.

    Algorithm Training

    For algorithm training in Amazon SageMaker, the software is priced based on hourly pricing that can vary by instance type. Additional infrastructure cost, taxes or fees may apply.
    InstanceType
    Algorithm/hr
    ml.m4.4xlarge
    $9.00
    ml.m5.4xlarge
    $9.00
    ml.m5.12xlarge
    $9.00
    ml.m4.16xlarge
    $9.00
    ml.m5.2xlarge
    $9.00
    ml.c4.4xlarge
    $9.00
    ml.m5.xlarge
    $9.00
    ml.c5.9xlarge
    $9.00
    ml.m4.xlarge
    $9.00
    ml.c5.4xlarge
    $9.00
    ml.m4.2xlarge
    $9.00
    ml.c5.2xlarge
    $9.00
    ml.m5.large
    Vendor Recommended
    $9.00
    ml.c4.2xlarge
    $9.00
    ml.c4.8xlarge
    $9.00
    ml.m4.10xlarge
    $9.00
    ml.c4.xlarge
    $9.00
    ml.m5.24xlarge
    $9.00
    ml.c5.18xlarge
    $9.00
    ml.c5.xlarge
    $9.00

    Usage Information

    Training

    The training data should be a single file in csv or parquet format.

    The first csv row should be the name of the columns. The performance estimation algorithms require model inputs, outputs and targets to work. Through it's hyperparameters we specify the data type of each column.

    Look at the NannyML Data Requirements Documentation for more information.

    Channel specification

    Fields marked with * are required

    training

    *
    Input channel that provides training data
    Input modes: File
    Content types: text/csv
    Compression types: None

    Hyperparameters

    Fields marked with * are required

    problem_type

    *
    Model problem type. If problem type is _regression_, the algorithm used is DLE, otherwise CBPE is used.
    Type: Categorical
    Tunable: No

    data_filename

    *
    The file name that contains the training data.
    Type: FreeText
    Tunable: No

    data_type

    *
    The file format of the training data file.
    Type: Categorical
    Tunable: No

    parameters

    *
    Algorithm parameters dict, encoded as JSON string. This parameters are passed as kwargs to the corresponding algorithm depending the problem type.
    Type: FreeText
    Tunable: No

    Model input and output details

    Input

    Summary

    The input should be a CSV file. It should contain the names of the columns in the first row.

    The required columns depend on the "parameters" defined during training. For more information read NannyML Performance Estimation Documentation .

    The required number of rows depend on the chunking method defined during training.

    Limitations for input type
    The first line of the file should be the columns names, and it should contain the columns defined on the "parameters" during training. For realtime, the maximum size of the input data per invocation is 6 MB. For batch, the maximum size of the input data per invocation is 100 MB.
    Input MIME type
    text/csv
    Sample input data
    Expand all input descriptions
    Field name
    y_pred
    Description
    This column type is required on all machine learning problem types. The values are the predicted labels for classification and predicted number for regression. Notice that the column name mapped to this column type on the "parameters" hyperparameter is defined during training, so you can have a different name for it on your CSV file.
    Required
    Yes
    Data type
    FreeText
    Limitations
    For classification data type can be text or integer but for regression it will be continuous.
    Field name
    y_pred_proba
    Description
    This column type is required on classification problem types. The values are the predicted scores or probabilities for a specific class. Notice that the column name mapped to this column type on the "parameters" hyperparameter is defined during training, so you can have a different name for it on your CSV file.
    Required
    No
    Data type
    Continuous (Float)
    Range
    Default value
    No default values
    Field name
    feature_column_names
    Description
    This input refers to a list of required columns when using the performance estimation algorithm on a regression problem. The list should include all the column names that should be consider features of the model whose performance we are predicting.
    Required
    No
    Data type
    FreeText
    Default value
    No default values
    Limitations
    The values are the features of your model. These can be categorical or continuous. NannyML identifies this based on their declared pandas data types.
    Field name
    y_true
    Description
    This column type contains actual model targets and is required on all machine learning problem types for the training data only. Notice that the column name mapped to this column type on the "parameters" hyperparameter is defined during training, so you can have a different name for it on your CSV file. [Optional] If you include this column on your inference data, realized performance will also be calculated separately to estimated performance to facilitate easy comparison.
    Required
    No
    Data type
    FreeText
    Default value
    No default values
    Limitations
    For classification data type can be text or integer but for regression it will be continuous.

    Output

    Summary

    NannyML performance estimation algorithm estimates model performance. However the outputs also contain confience bands for this estimation that account for sampling error effects. They also contain performance thresholds that help identify significant changes based on the user's Model behavior on reference.

    Output MIME type
    text/csv
    Sample output data
    Expand all output descriptions
    Field name
    Estimated Performance Results dataframe
    Description
    The output of the model is a MultiIndex dataframe saved as a CSV file. The first two rows should be used as the head when consuming the result on pandas. There are columns containing metadata information related to chunks as well as columns that describe the results for each selected metric. You can see an example of the outputs in our User Guide: https://github.com/NannyML/sagemaker_perf_est_docs/blob/main/notebooks/Model%20Performance%20Estimation%20-%20NannyML%20-%20Algorithm.ipynb
    Always returned
    Yes
    Data type
    FreeText
    Limitations
    The output type includes the computed values for the training inputs (`chunk.period = "reference"`), so be aware of this when processing the output. Each row correspond to one chunk, and this chunk is aggregated based on the chunking type defined during training.

    Sample notebook

    Additional Resources

    End User License Agreement

    By subscribing to this product you agree to terms and conditions outlined in the product End user License Agreement (EULA)

    Support Information

    Model Performance Estimation - NannyML

    If you have any questions, reach out to support@nannyml.com

    AWS Infrastructure

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Learn More

    Refund Policy

    Given the free trial we do not support refunds, but you can cancel your subscription to the service at any time.

    Customer Reviews

    There are currently no reviews for this product.
    View all