What Is Random Forest Regression?

Random forest regression is a machine learning algorithm that eliminates data points based on different if-then criteria to make fairly accurate predictions for a given input. It combines the outputs from multiple decision tree algorithms that run in parallel to reach the final prediction more efficiently. The random forest model requires labeled data for initial training as it is a classification algorithm.

Decision trees are the basic machine learning algorithms a random forest regression model uses to make highly accurate predictions. They are a supervised learning algorithm commonly used for classification and regression tasks. Data scientists use decision trees to determine which category an object belongs to or predict the outcomes of future events based on historical data. When applied, a decision tree algorithm learns from training data to make an if-then prediction. After several decisions, the algorithm provides a conclusive outcome.

You can imagine decision trees as trees planted invertedly, expanding their branches downwards. The tree graph branches or divides into two from the root as it answers a yes-no question. These assessments take place in the decision nodes. The process continues until it reaches a certain depth, ending at the leaf nodes, which produce the final prediction.

Example

For example, the decision tree algorithm evaluates whether a person’s cholesterol level is above a certain threshold. If not, it proceeds to the following criteria until it can confidently determine if the person is healthy. Otherwise, the algorithm assesses other health factors to calculate the person’s health risk.

Challenges

Decision trees are fast and helpful in determining non-linear relationships between input features. However, they are prone to overfitting, particularly if the branch extends to a certain depth. Overfitting is a phenomenon in which the machine learning model performs accurately on training data but not unfamiliar real-world information. Decision trees are also highly susceptible to data variation and noise. If there is a slight change in the training data, the algorithm might produce a significantly different result.

The random forest regression model combines the average output of multiple decision trees. It first calculates the prediction of individual trees. Then, it chooses the prediction that occurs most frequently. Instead of relying on a single prediction, it considers the majority votes of all decision trees and predicts the final outcome.

Ensemble learning

Random forest uses ensemble learning to consolidate results from multiple decision trees. There are two types of ensemble learning — bagging and boosting.

• Bagging ensemble creates several independent machine-learning models and trains them with subsets of the same training data.

• Boosting ensemble combines several machine-learning models in series to generate a strong prediction.

Both types of ensemble learning train the models with the same data. The random forest regressor applies the bagging ensemble method to make more accurate predictions and prevent overfitting.

Bagging

Bagging, or bootstrap aggregation, is an ensemble learning method for classification and regression models. There are two stages in bootstrap aggregation.

• Bootstrapping uses a replacement sampling method to ensure multiple random samples are created from the same dataset.

• Aggregation allows the machine learning algorithm to consolidate the outcome of all bootstrapping models involved in the predictive task.

Bootstrap aggregation reduces dependency between samples. Because the samples were created from the same training dataset, the model doesn’t experience high variance.

Data scientists train a random forest regression with bagging. The training process is as follows.

Training individual decision trees

Data scientists randomly select dataset parts with bootstrap sampling or row sampling. For each sample, they draw from a small number of data from the same training dataset, a process we call replacement. Then, data scientists create separate decision trees for each subpart of the dataset. They can create multiple decision trees but must ensure they are not interdependent. Because each sample was drawn from the same dataset, it might contain repetitive data.

Feature sampling

Data scientists apply feature sampling to ensure that each decision tree is truly independent. Feature sampling extracts specific columns or features of the training dataset from the samples without replacement. This way, decision trees can make predictions based on different features without worrying about overlapping.

Data scientists evaluate the random forest model’s accuracy by computing its out-of-bag (OOB) and validation scores.

Out-of-bag score

Out-of-bag (OOB) refers to rows of data not selected when creating subsets of training samples with the replacement method. These leftover data rows are considered out-of-bag because they are not used to train the random forest model. However, OOB data can help validate the model’s accuracy after training. To do that, data scientists feed the random forest model with the OOB data and evaluate its prediction. They calculate how many accurate predictions the model made, which gives the OOB score.

Validation score

The validation score is another metric that measures the model’s accuracy, similar to the out-of-bag score. However, the dataset used to calculate the validation score is intentionally set aside and not used for sampling.

The random forest algorithm improves upon decision trees as follows.

Accuracy

Aggregation overcomes the decision tree model’s overfitting issue. It also allows random forests to be less affected by dataset variations. A few inaccurate predictions by a single decision tree don’t affect the final result, making the random forest model more accurate.

Computing speed

Like the decision tree model, random forests are excellent in analyzing non-linear input features. They can also make predictions even if the training sample has missing data. However, decision trees in random forests don’t train from the entire dataset. Instead, it learns only from a subpart of the training dataset. For example, you’ll need to train a credit-scoring decision tree model with all the customer data. However, each decision tree in the random forest only uses data from some customers. This improves computing speed because several decision trees can predict in parallel and consolidate their outcomes later.

Random forest regression overcomes overfitting issues that decision trees suffer with random feature sampling and bootstrapping aggregation. It is one of the more accurate classification algorithms that data scientists apply in various applications. However, the random forest model has some limitations.

Less interpretability

Random forest regression models are less interpretable when the decision trees increase in depth. Each tree branches into several layers of decision nodes, making it incredibly difficult to determine why a specific prediction was made.

Slow training

Training and generalizing with the random forest model is slower than other similar models. Despite computing the outcome with several decision trees, implementing the ensemble learning method, which involves row and feature sampling, will consume more computing resources.

Limited extrapolation

Random forest cannot extrapolate results beyond values seen in the training dataset for a specific target variable. This means you cannot use random forests for time-series predictions, such as predicting the cost of living for the next decades. That’s because the random forest model will limit its prediction to the maximum and minimum values in the training data.

Random cut forest (RCF) regression is a special machine learning algorithm developed by AWS for detecting outliers in large datasets. Data anomalies can unnecessarily complicate machine learning tasks. RCF identifies abnormal data in time series data and assigns an anomaly score to different data points. This way, data scientists can remove data that diverge significantly from the general representation. To do that, RFC randomly selects several groups of data points. Then, it cuts them so each has the same number of points. Then, the algorithm creates decision trees for each data point group.

Unlike random forests, the trees in RCF allow incremental updates. It’s also suitable for handling high dimensional and streaming data in unsupervised learning applications.

Amazon SageMaker provides a suite of built-in algorithms, pre-trained models, and pre-built solution templates to help data scientists and machine learning practitioners quickly start training and deploying machine learning models. Amazon SageMaker Random Cut Forest (RCF) is a built-in, unsupervised algorithm for detecting anomalous data points within the data set.

Similarly, Amazon SageMaker Canvas is a no-code machine learning service that supports the entire ML workflow, including data preparation, model building and training, generating predictions, and deploying the models to production. In Ensembling mode, Canvas supports random forest regression among several other models.

Get started with random forest regression on AWS by creating a free account today.