Bagging & Boosting

Random Forest and gradient boosting are the two strongest families of tabular models. Gradient boosting is the engine behind XGBoost, LightGBM, and CatBoost. Both are ensembles of decision trees, and they reach the same ceiling by opposite paths.

• Bagging trains many trees in parallel on resampled data and averages them. It reduces variance.
• Boosting trains many trees sequentially, each correcting the previous ones. It reduces bias.

Every code block runs in a container in the companion repository, so the numbers and figures are real. They use the breast cancer dataset.

Why ensemble at all

A single decision tree is greedy and high-variance. Small changes to the data shift the splits and give different predictions, and a deep tree will happily memorize noise until it fits the training set almost perfectly and then generalizes badly on anything new. Two strategies fix this without changing the model family. Train many trees on slightly different views of the data and average them, or train them one after another so each new tree corrects the mistakes the ensemble has made so far.

Bagging

Bagging is bootstrap aggregating: draw a bootstrap sample, train a tree, repeat, and average the predictions.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag = BaggingClassifier(DecisionTreeClassifier(random_state=42),
                        n_estimators=100, random_state=42, n_jobs=-1)

Train a single tree across 10 seeds, then bagging ensembles of growing size:

  n=  1  single 0.9234 +/- 0.0061   bagging 0.9164 +/- 0.0211
  n=  5  single 0.9234 +/- 0.0061   bagging 0.9357 +/- 0.0157
  n= 25  single 0.9234 +/- 0.0061   bagging 0.9421 +/- 0.0099
  n=100  single 0.9234 +/- 0.0061   bagging 0.9444 +/- 0.0047
  n=200  single 0.9234 +/- 0.0061   bagging 0.9439 +/- 0.0039

The single tree’s mean stays at 0.92 no matter how many seeds you average. Bagging climbs from 0.92 to 0.94. Its standard deviation drops from 0.021 to 0.004. That falling spread is variance reduction in action.

Random Forest adds one trick to bagging. At every split it samples a random subset of features (max_features="sqrt" by default). That decorrelates the trees and cuts variance further.

Boosting

Boosting builds the ensemble sequentially, each tree fitting the residuals of the previous ensemble, scaled by a learning rate.

from sklearn.ensemble import GradientBoostingClassifier

gbm = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                 max_depth=3, random_state=42)

Track the log-loss on train and test after each tree:

  Training loss at iter 1:   0.5718
  Training loss at iter 50:  0.0134
  Training loss at iter 200: 0.0001
  Test loss at iter 1:       0.5852
  Test loss at iter 50:      0.1530
  Test loss at iter 200:     0.1906
  Best test iter:            60

Training loss falls almost to zero. Test loss bottoms out at iteration 60, then drifts up. That is the boosting overfitting signature. It is why early stopping matters: watch a validation set and stop when it stops improving.

Algorithm	Update rule
AdaBoost	Reweights misclassified samples so the next stump focuses on them
Gradient boosting	Fits each tree to the gradient of the loss
XGBoost / LightGBM / CatBoost	Modern gradient boosting with regularization, second-order gradients, and engineering tricks

Cross-validated comparison

5-fold accuracy on the same dataset:

                        model  cv_accuracy      std
        AdaBoost (100 stumps)     0.9772 0.0105
Gradient Boosting (100 trees)     0.9631 0.0210
          Bagging (100 trees)     0.9579 0.0382
    Random Forest (100 trees)     0.9561 0.0228
         Single decision tree     0.9173 0.0242

Every ensemble beats the single tree by 4 to 6 points. The exact ranking is dataset-dependent, so the honest answer is: try both with cross-validation.

When to reach for which

Use case	Pick
Quick strong baseline, no tuning	Random Forest
Highest accuracy on tabular data	Gradient boosting (XGBoost, LightGBM, CatBoost)
Many categorical features	CatBoost
Very large dataset	LightGBM (fastest)
Small dataset	Random Forest

Next steps

• XGBoost, the canonical gradient boosting.
• LightGBM, faster on large data.
• CatBoost, native categoricals.
• Stacking and out-of-fold ensembling, a layer above bagging and boosting.