LightGBM

LightGBM is a gradient boosting framework optimized for speed. It grows trees leaf-wise instead of the level-wise approach XGBoost uses, which makes it faster and often more accurate on large datasets.

Every code block runs in a container in the companion repository, so the numbers are real.

Train a basic model

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.metrics import classification_report, roc_auc_score

X, y = make_classification(n_samples=2000, n_features=20, n_informative=10,
                           n_redundant=3, weights=[0.7, 0.3], flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

model = lgb.LGBMClassifier(n_estimators=100, max_depth=4, learning_rate=0.1,
                           num_leaves=31, random_state=42, verbose=-1)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
print(cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc").mean())

Cross-validated ROC-AUC: 0.9275 (+/- 0.0230)
Test ROC-AUC: 0.9480
              precision    recall  f1-score   support
    Negative       0.92      0.94      0.93       276
    Positive       0.86      0.81      0.83       124
    accuracy                           0.90       400

Leaf-wise vs level-wise growth

XGBoost grows level by level: every node at one depth is split before the next depth. LightGBM grows leaf-wise, always splitting the leaf with the highest loss reduction.

Level-wise (XGBoost):        Leaf-wise (LightGBM):
      [root]                       [root]
      /    \                       /    \
    [L1]   [L1]                 [L1]   [L1]
   /  \    /  \                /  \
 [L2][L2][L2][L2]           [L2] [L2]
                             /  \
                          [L3] [L3]

Leaf-wise growth yields deeper trees with fewer leaves. It often improves accuracy but can overfit on small data. num_leaves is the primary complexity control: set it lower (15 to 31) for small datasets.

Native categorical features

LightGBM handles categorical features directly, without one-hot encoding. Mark columns as category dtype and pass them to fit via categorical_feature.

cat_features = ["smoking", "exercise", "family_history"]
model_cat = lgb.LGBMClassifier(n_estimators=200, num_leaves=31, learning_rate=0.1,
                               random_state=42, verbose=-1)
model_cat.fit(X_train_cat, y_train_cat, categorical_feature=cat_features)
print(f"ROC-AUC: {roc_auc_score(y_test_cat, model_cat.predict_proba(X_test_cat)[:, 1]):.4f}")

ROC-AUC with native categoricals: 0.7846

LightGBM finds optimal splits by grouping category values rather than expanding them into one-hot columns, which is far more efficient for high-cardinality categoricals.

Comparing configurations

for cfg in configs:
    m = lgb.LGBMClassifier(random_state=42, verbose=-1, **cfg)
    print(cross_val_score(m, X_train, y_train, cv=cv, scoring="roc_auc").mean())

  n_est=100, leaves=31, lr=0.10 -> ROC-AUC=0.9349
  n_est=200, leaves=31, lr=0.10 -> ROC-AUC=0.9366
  n_est=200, leaves=63, lr=0.10 -> ROC-AUC=0.9351
  n_est=200, leaves=31, lr=0.05 -> ROC-AUC=0.9335
  n_est=300, leaves=31, lr=0.05 -> ROC-AUC=0.9365
  n_est=300, leaves=63, lr=0.05 -> ROC-AUC=0.9340

The default num_leaves=31 wins here (best ROC-AUC 0.9366, tuned test 0.9554). Increasing to 63 does not help, because the dataset is not large enough to need more complex trees.

Feature importance

pd.DataFrame({"feature": feature_names,
             "importance": best_model.feature_importances_}).sort_values(
    "importance", ascending=False).head(6)

   feature  importance
 feature_3         550
 feature_0         466
feature_14         431
feature_16         422
 feature_4         419
 feature_9         393

LightGBM reports importance as the split count (how often a feature is used), which differs from XGBoost’s gain-based importance. Both are valid, but they measure different things.

Early stopping

model_es = lgb.LGBMClassifier(n_estimators=1000, num_leaves=31, learning_rate=0.05,
                              random_state=42, verbose=-1)
model_es.fit(X_train_es, y_train_es, eval_set=[(X_val, y_val)],
             callbacks=[lgb.early_stopping(20, verbose=False), lgb.log_evaluation(0)])
print(f"Best iteration: {model_es.best_iteration_}")

Best iteration: 157
Best validation logloss: 0.2164

LightGBM uses callbacks: lgb.early_stopping(20) stops when the validation metric does not improve for 20 rounds, lgb.log_evaluation(0) silences the logs.

Speed

LightGBM’s leaf-wise growth and histogram-based splitting make it faster than XGBoost, and the gap widens with dataset size.

import time

X_large, y_large = make_classification(n_samples=50000, n_features=50,
                                       n_informative=20, random_state=42)
# time XGBClassifier(n_estimators=200, max_depth=6) vs the same LGBMClassifier

On this dataset LightGBM trained several times faster than XGBoost. The exact ratio depends on the hardware and dataset, but the direction is consistent: for large data, LightGBM is the faster choice.

Choosing between them

Feature	LightGBM	XGBoost
Tree growth	Leaf-wise	Level-wise
Complexity control	num_leaves	max_depth
Categoricals	Native	Manual (one-hot)
Speed	Very fast	Fast
Overfitting on small data	Higher risk	Lower risk

Next steps

The next page covers CatBoost, which takes a different approach to categorical features and uses ordered boosting to reduce overfitting.