Adaptive Boosting vs Gradient Boosting

Adaptive Boosting (AdaBoost)

• Alters the distribution of the training dataset to increase weights on sample observations that are difficult to classify.

• The final prediction is based on a majority vote of the weak learner's predictions weighted by their individual accuracy.

Weak learners applied are decision stumps (i.e., decision trees with a single split or one-level).

The observations in the dataset are weighted such that a greater weight on observations that are misclassified (difficult to classify) versus the observations that have been classified correctly by the model. New weak learners are added sequentially to focus training on the difficult observations. In this manner, observations that are difficult to classify are weighted higher to the point that the algorithm identifies a model that accurately classifies these observations. Adaptive boosting changes the sample distribution being used for training.

Intuitively, Adaboost is known as a step-wise additive model. Adaboost initially specifies a set of weak learners and the boosting process is to learn how to add the weights of these weak learners to be a strong learner. The weight of each learner is based on whether it predicts each observation accurately or not. If the learner mispredicts an observation, its weight is reduced. This process is repeated until convergence.

Predictions are made by using a majority vote of the weak learner's predictions (i.e., using a set of weak learners and seeing which classification outcome is voted for the most) weighted by their individual accuracy.

Gradient Boosted Trees

• The approach trains learners based upon minimizing the loss function of a strong learner (i.e., training on the residuals of the strong model) as an alternative means to focus on training upon misclassified observations.

• The contribution of each weak learner to the final prediction is based on a gradient optimization process to minimize the overall error of the strong learner.

Weak learners are decision trees constructed in a greedy manner with split points based on purity scores (i.e., Gini, minimize loss) thus larger trees can be used around 4 to 8 levels. Learners should still remain weak and so they should be constrained (i.e., maximum number of layers, nodes, splits, leaf nodes)

Gradient boosting re-defines boosting as a numerical optimization problem where the objective is to minimize the loss function of the model by adding weak learners using a gradient-descent like procedure. As gradient boosting is based on minimizing a loss function, different types of loss functions can be used resulting in a flexible technique that can be applied to regression, multi-class classification, etc.

Intuitively, gradient boosting is a stage-wise additive model that generates learners during the learning process (i.e., trees are added one at a time, and existing trees in the model are not changed). It builds a first learner to predict the observations in the training dataset, and then it calculates the loss (i.e., the value between the first learner's outcomes and the actual values). It will build a second learner that is fitted/trained on the residual error produced by the first learner to predict the loss after the first step and continue to do so until it reachest a threshold (i.e., residuals are zero). By training the second learner on the gradient of the error with respect to the loss predictions of the first learner, the second learner is being trained to correct the mistakes of the first model. This is the core of gradient boosting, and allows many simple learners to compensate for each other’s weaknesses to better fit the data.

Gradient boosting does not modify the sample distribution as weak learners train on the remaining residual errors of a strong learner (i.e, pseudo-residuals). By training on the residuals of the model, this is an alternative means to give more importance to misclassified observations. Intuitively, new weak learners are being added to concentrate on the areas where the existing learners are performing poorly.

The contribution of the weak learner to the ensemble is based on a gradient descent optimization process. The calculated contribution of each tree is based on minimizing the overall error of the strong learner.