LightGBM

LightGBM is a gradient boosting framework optimized for speed. It grows trees leaf-wise instead of the level-wise approach XGBoost uses, which makes it faster and often more accurate on large datasets.

Every code block runs in a container in the companion repository, so the numbers are real.

Train a basic model

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.metrics import classification_report, roc_auc_score

features, y = make_classification(n_samples=2000,
                                  n_features=20,
                                  n_informative=10,
                                  n_redundant=3,
                                  weights=[0.7, 0.3],
                                  flip_y=0.05,
                                  random_state=42)
x_train, x_test, y_train, y_test = train_test_split(features,
                                                    y,
                                                    test_size=0.2,
                                                    random_state=42,
                                                    stratify=y)

model = lgb.LGBMClassifier(n_estimators=100,
                           max_depth=4,
                           learning_rate=0.1,
                           num_leaves=31,
                           random_state=42,
                           verbose=-1)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
print(cross_val_score(model, x_train, y_train, cv=cv, scoring="roc_auc").mean())

Cross-validated ROC-AUC: 0.9275 (+/- 0.0230)
Test ROC-AUC: 0.9480
              precision    recall  f1-score   support
    Negative       0.92      0.94      0.93       276
    Positive       0.86      0.81      0.83       124
    accuracy                           0.90       400

Leaf-wise vs level-wise growth

XGBoost grows level by level: every node at one depth is split before the next depth. LightGBM grows leaf-wise, always splitting the leaf with the highest loss reduction.

Level-wise (XGBoost):        Leaf-wise (LightGBM):
      [root]                       [root]
      /    \                       /    \
    [L1]   [L1]                 [L1]   [L1]
   /  \    /  \                /  \
 [L2][L2][L2][L2]           [L2] [L2]
                             /  \
                          [L3] [L3]

Leaf-wise growth yields deeper trees with fewer leaves. It often improves accuracy but can overfit on small data. num_leaves is the primary complexity control: set it lower (15 to 31) for small datasets.

Native categorical features

LightGBM handles categorical features directly, without one-hot encoding. Mark columns as category dtype and pass them to fit via categorical_feature.

cat_features = ["smoking", "exercise", "family_history"]
model_cat = lgb.LGBMClassifier(n_estimators=200,
                               num_leaves=31,
                               learning_rate=0.1,
                               random_state=42,
                               verbose=-1)
model_cat.fit(x_train_cat, y_train_cat, categorical_feature=cat_features)
predicted_probability = model_cat.predict_proba(x_test_cat)[:, 1]
roc_auc = roc_auc_score(y_test_cat, predicted_probability)
print(f"ROC-AUC: {roc_auc:.4f}")

ROC-AUC with native categoricals: 0.7846

LightGBM finds optimal splits by grouping category values rather than expanding them into one-hot columns, which is far more efficient for high-cardinality categoricals.

Comparing configurations

for cfg in configs:
    m = lgb.LGBMClassifier(random_state=42, verbose=-1, **cfg)
    print(cross_val_score(m, x_train, y_train, cv=cv, scoring="roc_auc").mean())

  n_est=100, leaves=31, lr=0.10 -> ROC-AUC=0.9349
  n_est=200, leaves=31, lr=0.10 -> ROC-AUC=0.9366
  n_est=200, leaves=63, lr=0.10 -> ROC-AUC=0.9351
  n_est=200, leaves=31, lr=0.05 -> ROC-AUC=0.9335
  n_est=300, leaves=31, lr=0.05 -> ROC-AUC=0.9365
  n_est=300, leaves=63, lr=0.05 -> ROC-AUC=0.9340

The default num_leaves=31 wins here (best ROC-AUC 0.9366, tuned test 0.9554). Increasing to 63 does not help, because the dataset is not large enough to need more complex trees.

Feature importance

pd.DataFrame({
    "feature": feature_names,
    "importance": best_model.feature_importances_
}).sort_values("importance", ascending=False).head(6)

   feature  importance
 feature_3         550
 feature_0         466
feature_14         431
feature_16         422
 feature_4         419
 feature_9         393

LightGBM reports importance as the split count (how often a feature is used), which differs from XGBoost’s gain-based importance. Both are valid, but they measure different things.

Early stopping

model_es = lgb.LGBMClassifier(n_estimators=1000,
                              num_leaves=31,
                              learning_rate=0.05,
                              random_state=42,
                              verbose=-1)
model_es.fit(
    x_train_es,
    y_train_es,
    eval_set=[(x_validation, y_val)],
    callbacks=[lgb.early_stopping(20, verbose=False),
               lgb.log_evaluation(0)])
print(f"Best iteration: {model_es.best_iteration_}")

Best iteration: 157
Best validation logloss: 0.2164

LightGBM uses callbacks: lgb.early_stopping(20) stops when the validation metric does not improve for 20 rounds, lgb.log_evaluation(0) silences the logs.

Speed

LightGBM’s leaf-wise growth and histogram-based splitting make it faster than XGBoost, and the gap widens with dataset size.

import time

x_large, y_large = make_classification(n_samples=50000,
                                       n_features=50,
                                       n_informative=20,
                                       random_state=42)
# time XGBClassifier(n_estimators=200, max_depth=6) vs the same LGBMClassifier

On this dataset LightGBM trained several times faster than XGBoost. The exact ratio depends on the hardware and dataset, but the direction is consistent: for large data, LightGBM is the faster choice.

Choosing between them

Feature	LightGBM	XGBoost
Tree growth	Leaf-wise	Level-wise
Complexity control	num_leaves	max_depth
Categoricals	Native	Manual (one-hot)
Speed	Very fast	Fast
Overfitting on small data	Higher risk	Lower risk

Next steps

The next page covers CatBoost, which takes a different approach to categorical features and uses ordered boosting to reduce overfitting.