Stacking

Stacking (stacked generalization) trains a meta-model on the out-of-fold predictions from base models. Instead of a fixed blend weight, the meta-model learns which base models to trust and how to combine them.

Every code block runs in a container in the companion repository, so the numbers and figure are real. It uses the same four base models and OOF setup as the previous page.

How stacking works

Base models → OOF predictions → meta-model → final prediction

1. Train each base model with cross-validation and collect OOF predictions.
2. Stack the OOF columns into a new feature matrix.
3. Train a meta-model on that matrix with the true labels.
4. For the test set, average each base model’s fold predictions, then pass them through the meta-model.

The meta-feature matrix

oof_matrix = np.column_stack(list(oof_preds.values()))
test_matrix = np.column_stack(list(test_preds.values()))
print(oof_matrix.shape, test_matrix.shape)

OOF matrix shape: (2400, 4)   Test matrix shape: (600, 4)

Each column is one base model’s OOF prediction. The meta-model sees a 4-column matrix, one row per training sample.

Training the meta-model

Logistic regression is the common choice: fast, interpretable, well-calibrated.

meta_model = LogisticRegression(C=1.0, max_iter=1000, random_state=42)
meta_model.fit(oof_matrix, y_train)
print(roc_auc_score(y_test, meta_model.predict_proba(test_matrix)[:, 1]))

Stacking (LR meta): OOF AUC = 0.9077  |  Test AUC = 0.8823

OOF correlation and meta-model weights

The tree models (Random Forest, XGBoost, LightGBM) predict similarly to each other, while logistic regression is less correlated. The meta-model coefficients on the right show it learned to weight Random Forest most and to downweight logistic regression to a small negative coefficient, automatically.

Comparing all methods

All three ensemble methods beat the best single model. Hill climbing and stacking are close here; with more diverse models, stacking tends to pull ahead.

scikit-learn StackingClassifier

scikit-learn wraps the whole thing.

from sklearn.ensemble import StackingClassifier

stack = StackingClassifier(
    estimators=list(base_models.items()),
    final_estimator=LogisticRegression(C=1.0, max_iter=1000, random_state=42),
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42), n_jobs=-1)
stack.fit(X_train, y_train)
print(roc_auc_score(y_test, stack.predict_proba(X_test)[:, 1]))

StackingClassifier test AUC: 0.8853

Set passthrough=True to also feed the original features to the meta-model, which can help when the base models miss some signal.

Do not score the meta-model on its own training OOF

The OOF matrix trained the meta-model, so predict_proba(oof_matrix) is optimistic. Always evaluate on the test set or a held-out fold.

Meta-model choices

Meta-model	When to use
Logistic regression	Default: fast, interpretable, 4 to 10 base models
Ridge classifier	When base predictions are correlated
LightGBM / XGBoost	Many base models, or nonlinear combinations
Simple average	When models are similar and you want zero overfitting risk

Summary

Technique	How it works	Best for
Blind average	Equal weights, no OOF	Quick baseline
Hill climbing	Greedy OOF weight search	4 to 10 diverse models
Stacking	Meta-model on OOF features	Many models, diverse errors