CatBoost

CatBoost handles categorical and text features natively, with no manual encoding. It uses ordered boosting, which reduces the target leakage that plagues ordinary target encoding. This page walks a full workflow on a mixed-type clinical dataset.

Every code block runs in a container in the companion repository, so the numbers and figure are real.

The data

A dataset with numeric, categorical, and free-text (diagnosis) columns, and a target that depends on age, BMI, glucose, smoking, exercise, and diagnosis keywords.

from catboost import CatBoostClassifier, Pool, cv as catboost_cv

cat_features = ["smoking", "exercise", "family_history", "department"]
X = df.drop(columns=["target", "diagnosis"])  # hold text out for now
y = df["target"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

Native categorical features

Just name the categorical columns; no one-hot or label encoding. Use CatBoost’s own cross-validation with Pool for correct categorical handling.

train_pool = Pool(X_train, y_train, cat_features=cat_features)
params = {"iterations": 200, "depth": 6, "learning_rate": 0.1,
          "loss_function": "Logloss", "eval_metric": "AUC", "random_seed": 42, "verbose": 0}
cvr = catboost_cv(train_pool, params, fold_count=5, stratified=True, seed=42, verbose=False)
print(cvr["test-AUC-mean"].iloc[-1])

Cross-validated ROC-AUC: 0.8580 (+/- 0.0147)
Test ROC-AUC: 0.8618
              precision    recall  f1-score   support
    Negative       0.57      0.44      0.50        62
    Positive       0.90      0.94      0.92       338
    accuracy                           0.86       400

Internally CatBoost uses ordered target encoding: for each sample it computes the category’s mean target from earlier samples only, in a randomized order. This avoids both the leakage of ordinary target encoding and the width of one-hot encoding.

Native text features

CatBoost tokenizes text columns and builds bag-of-words features automatically.

model_text = CatBoostClassifier(iterations=200, depth=6, learning_rate=0.1,
                                cat_features=cat_features, text_features=["diagnosis"],
                                random_seed=42, verbose=0)
model_text.fit(X_train_full, y_train_full)
print(roc_auc_score(y_test_full, model_text.predict_proba(X_test_full)[:, 1]))

ROC-AUC with text features: 0.8694

Adding the diagnosis text lifted ROC-AUC from 0.862 to 0.869, because the target partly depends on diagnosis keywords.

When to use text features

CatBoost text features work well for short fields like diagnosis codes, drug names, or brief notes. For long documents, use dedicated NLP models. But for a tabular dataset with a text column, this saves building a separate TF-IDF pipeline.

Hyperparameter tuning

CatBoost has strong defaults; usually you only tune depth, learning_rate, and l2_leaf_reg. Use its native cv so categoricals are handled correctly.

best_score, best_params = 0, {}
for depth in [4, 6, 8]:
    for lr in [0.05, 0.1]:
        for l2 in [1, 3, 5]:
            p = {**params, "depth": depth, "learning_rate": lr, "l2_leaf_reg": l2}
            auc = catboost_cv(train_pool, p, fold_count=5, stratified=True,
                              seed=42, verbose=False)["test-AUC-mean"].iloc[-1]
            if auc > best_score:
                best_score, best_params = auc, {"depth": depth, "learning_rate": lr, "l2_leaf_reg": l2}

Best ROC-AUC: 0.8664
Best parameters: {'depth': 4, 'learning_rate': 0.05, 'l2_leaf_reg': 3}
Tuned Test ROC-AUC: 0.8743

Feature importance

pd.DataFrame({"feature": X_train.columns.tolist(),
             "importance": best_model.feature_importances_}).sort_values(
    "importance", ascending=False)

       feature  importance
           bmi   19.932746
       glucose   19.755517
           age   19.272631
   cholesterol   14.189801
       smoking   13.797002
      exercise    6.259204

BMI, glucose, and age lead, matching how the target was built. The categorical smoking ranks fifth, so CatBoost extracts real signal from the categorical columns.

CatBoost vs XGBoost on categorical data

XGBoost needs one-hot encoding; CatBoost handles categoricals natively.

# XGBoost with a OneHotEncoder ColumnTransformer, vs CatBoost native

XGBoost (one-hot):      ROC-AUC = 0.8394
CatBoost (native cats): ROC-AUC = 0.8743
CatBoost (cats + text): ROC-AUC = 0.8694

CatBoost beats one-hot XGBoost by about 3 points here. Its ordered target encoding extracts more from categoricals than one-hot does.

Early stopping

model_es = CatBoostClassifier(iterations=1000, depth=6, learning_rate=0.05,
                              cat_features=cat_features, random_seed=42, verbose=0,
                              early_stopping_rounds=20)
model_es.fit(X_train_es, y_train_es, eval_set=Pool(X_val, y_val, cat_features=cat_features))
print(f"Best iteration: {model_es.best_iteration_}")

Best iteration: 260
Test ROC-AUC with early stopping: 0.8799

Choosing between them

Feature	CatBoost	XGBoost	LightGBM
Categoricals	Native (ordered)	Manual (one-hot)	Native (integer)
Text	Native	Manual (TF-IDF)	Manual (TF-IDF)
Defaults	Strong	Needs tuning	Needs tuning
Speed	Medium	Fast	Very fast

Reach for CatBoost when you have many categorical or text features and want strong results with little preprocessing.

Next steps

The next pages cover ensembling, where you combine several models to beat any single one.