What is Overfitting?

To diagnose a poorly performing model, you can evaluate it along a "model fit" spectrum, with overfitting and underfitting being the two opposite extremes.

Underfitting is another type of error that occurs when the model can't determine a meaningful relationship between the input and output data. You might see an underfit model if it has not been trained for the appropriate length of time on a large number of data points. Other reasons for underfitting include a model that's too simple for the data relationships or not enough features in the model. Underfitting can often be remedied by increasing the number of passes (epochs) the model makes on the existing training dataset.

Underfit models exhibit high bias, meaning they produce inaccurate results on both the training data and the test set. Overfit models exhibit high variance, meaning they perform well on the training set but not on the test set.

For a sufficiently complex model, more model training leads to less bias, but variance can increase. It is important to try to find the balance between underfitting and overfitting when fitting a model. A well-fitted model can quickly establish the dominant trend for both seen and unseen data sets.

Machine learning models will only give accurate predictions if they can generalize to unseen data within their domain. Overfitting occurs when the model can't generalize and instead fits too closely to the training dataset. Overfitting can happen for several reasons:

• The training data size is too small and does not contain enough data samples to represent all possible input values accurately
• The training data contains large amounts of irrelevant information, which is called noisy data
• The model memorizes the training data set, rather than learning its patterns
• The model complexity is high, so it learns the noise within the training data

Consider a use case where a machine learning model has to analyze photos and identify the ones that contain dogs in them. If you are training the machine learning model on a dataset containing mostly photos of dogs outside in parks, it might learn to use grass as a feature for classification and not recognize a dog inside a room.

Another example of overfitting is a machine learning algorithm that predicts a university student's academic performance and graduation outcome, based on their age at admission. However, the training data only includes data from a private university with high-income students. In this example, the prediction accuracy of this overfit algorithm drops for public universities, as their student demographics aren't reflected in the training data.

The best way to detect overfitting is to test machine learning models on more data, with a comprehensive range of possible input values and types. Typically, part of the training data is used as test data to check for overfitting. A high error rate in the testing data and a large gap between accuracy in the training data and test data indicate overfitting.

K-fold cross-validation

Cross-validation is one of the testing methods used to detect overfitting. In this method, you divide the training set into K equally sized subsets, or folds. The training process consists of a series of iterations. During each iteration, the steps are:

1. Keep one subset as the validation data and re-train the machine learning model from scratch on the remaining K-1 subsets.
2. Observe how the model performs on the validation example.
3. Score model performance based on output data quality.

Iterations repeat until you test the model on every sample set. You then average the scores across all iterations to get the final assessment of the predictive model.

When a model performs well on training and poorly on test data, the issue is not always related to model flexibility or complex parameters. Sometimes, the learning algorithm simply did not have enough data to learn the underlying patterns. Before attempting to adjust hyperparameters or alter the model's architecture, it is helpful to rule out basic data volume issues first. You can often improve a model's performance by increasing the volume of training data.

If data quantity is not the issue, the best approach is to adjust model flexibility. Model flexibility refers to the algorithm's capacity to learn complex relationships from the data.

If a model is overfitting, it has memorized the training set, meaning the developer must reduce model flexibility to encourage better generalization. You can reduce flexibility by increasing the amount of regularization used or by using fewer feature combinations (feature selection).

The data science strategies below outline the most common ways to adjust this flexibility, diversify your training dataset, and prevent overfitting.

Early stopping

Early stopping pauses training before the model learns noise in the data. However, getting the timing right is important, otherwise the model will still not give accurate results.

Pruning

You might identify several features or parameters that impact the final prediction when you build a model. Feature selection—or pruning—identifies the most important features within the training set and eliminates irrelevant ones. For example, to predict if an image is an animal or a human, you can look at various input parameters such as face shape, ear position, body structure, etc. You may prioritize face shape and ignore eye shape.

Regularization

Regularization is a collection of training and optimization techniques that seek to reduce overfitting. These methods aim to decrease factors that do not affect prediction outcomes by grading features by importance. For example, mathematical calculations apply a penalty value to features with minimal impact.

• L1 Regularization (Lasso): Penalizes the absolute value of the weights.
• L2 Regularization (Ridge): Penalizes the squared value of the weights.
• Dropout: A technique specifically for artificial neural networks where random neurons are deactivated during training to help prevent the model from relying on specific neurons.

Consider a statistical model attempting to predict the housing prices of a city in 20 years. Regularization would assign a lower penalty to features such as population growth and average annual income, but a higher penalty to the city's average annual temperature.

Ensembling

Ensembling combines predictions from several separate machine learning algorithms. Some models are called weak learners because their results are often inaccurate. Ensemble methods combine multiple weak learners to achieve more accurate results. They use multiple models to analyze sample data and combine predictions to achieve the most accurate outcomes. The two main ensemble methods are bagging and boosting. Boosting trains different machine learning models one after another to get the final result, and bagging trains them in parallel.

Data augmentation

Data augmentation is a machine learning technique that slightly modifies the sample data each time the model processes it. When used in moderation, data augmentation makes the training sets appear unique to the model and prevents it from learning overly specific characteristics. For example, applying transformations such as translations, flips, and rotations to input images.

You can use Amazon SageMaker AI to build, train, and deploy machine learning models for any use case with fully managed infrastructure, tools, and workflows. Amazon SageMaker AI has a built-in feature called Amazon SageMaker Model Training that automatically analyzes data generated during training, such as input, output, and transformations. As a result, it can detect and report overfitting and other inaccuracies without manual intervention.

For example, you can:

• Automatically stop the training process when the desired accuracy is achieved
• Capture training metrics in real-time
• Receive alerts when overfitting is detected
• Reduce the time and cost of training machine learning models

Get started with machine learning on AWS by creating a free account today.