OOF Blending & Hill Climbing

Ensembling combines predictions from several models to beat any single one. The key is doing it correctly with out-of-fold predictions rather than a blind average.

Every code block runs in a container in the companion repository, so the numbers and figures are real. The dataset has 3000 samples and four base models.

Blind vs OOF blending

A naive ensemble averages model predictions on the test set. It can help. But you have no reliable way to measure how much it helps, or to find good weights. Out-of-fold predictions fix this: all models share the same folds, and each one predicts only on the fold it never trained on, which gives you unbiased predictions for every training sample. Use those to tune the ensemble before you ever touch the test set.

Generating OOF predictions

Loop over shared folds, train on the training part, predict the held-out part, and average the test predictions across folds.

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in base_models.items():
    oof = np.zeros(len(X_train))
    test_fold = np.zeros((len(X_test), cv.n_splits))
    for fold, (tr, va) in enumerate(cv.split(X_train, y_train)):
        model.fit(X_train[tr], y_train[tr])
        oof[va] = model.predict_proba(X_train[va])[:, 1]
        test_fold[:, fold] = model.predict_proba(X_test)[:, 1]
    oof_preds[name] = oof
    test_preds[name] = test_fold.mean(axis=1)

Logistic Regression : OOF AUC = 0.8446  |  Test AUC = 0.8123
Random Forest       : OOF AUC = 0.9031  |  Test AUC = 0.8754
XGBoost             : OOF AUC = 0.9000  |  Test AUC = 0.8821
LightGBM            : OOF AUC = 0.9029  |  Test AUC = 0.8823

Averaging test predictions across the five fold models reduces variance versus a single final model.

Blind average

blind_oof = np.mean(list(oof_preds.values()), axis=0)
blind_test = np.mean(list(test_preds.values()), axis=0)

Blind average: OOF AUC = 0.9044  |  Test AUC = 0.8776

The blind average already matches the best single model, but it includes the weak logistic regression. Can we find better weights?

Hill climbing

Hill climbing greedily builds a “bag” of models. At each step it adds whichever model most improves the OOF AUC. A model can be added many times, so its final weight is how often it was chosen.

def hill_climb(oof_dict, y_true, n_iter=100):
    best = max(oof_dict, key=lambda n: roc_auc_score(y_true, oof_dict[n]))
    bag, cur = [best], roc_auc_score(y_true, oof_dict[best])
    for _ in range(n_iter - 1):
        pick = None
        for n in oof_dict:
            c = Counter(bag + [n])
            blend = sum((c[k] / sum(c.values())) * oof_dict[k] for k in c)
            s = roc_auc_score(y_true, blend)
            if s > cur:
                cur, pick = s, n
        if pick is None:
            break
        bag.append(pick)
    c = Counter(bag)
    return {n: c[n] / len(bag) for n in c}

Hill climbing weights:
  Random Forest : 0.625
  LightGBM      : 0.250
  XGBoost       : 0.125
Hill climbing: OOF AUC = 0.9078  |  Test AUC = 0.8817

Hill climbing excluded logistic regression entirely and mixed the three tree models. Its OOF score (0.9078) beats the blind average (0.9044), so the weights are well-calibrated.

OOF vs test

The OOF score tracks the test score across methods. That is the whole point: OOF lets you tune the ensemble without touching the test set.

When to use it

Scenario	Approach
2 to 3 similar models	Simple average is fine
4+ diverse models	Hill climbing finds better weights
Weak models in the pool	Hill climbing drops them automatically
Many models (10+)	Hill climbing or stacking

Next steps

The next page, Stacking, replaces the weighted average with a trained meta-model on the OOF predictions.