Cross-Validation

A single train/test split gives a noisy estimate of model performance. The score depends on which samples happened to land in the test set. Cross-validation averages over multiple splits to produce a more stable estimate.

Every code block on this page runs in a container in the companion repository, so the numbers are real. They all use one seeded, imbalanced synthetic dataset.

The data

import numpy as np
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20, n_informative=5,
                           weights=[0.85, 0.15], random_state=42)

About 15% of the samples are positive, the kind of class imbalance common in biomarker and disease-status studies.

K-Fold cross-validation

K-Fold splits the data into K equal parts. It trains K models, each time using K-1 parts for training and 1 part for validation. The final score is the average across all K folds.

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=42)
for fold, (train_idx, val_idx) in enumerate(kf.split(X), 1):
    print(f"Fold {fold}: train={len(train_idx)}, val={len(val_idx)}")

Fold 1: train=800, val=200
Fold 2: train=800, val=200
Fold 3: train=800, val=200
Fold 4: train=800, val=200
Fold 5: train=800, val=200

Always set shuffle=True to randomize the fold assignment, and random_state for reproducibility.

Stratified K-Fold

Standard K-Fold does not guarantee that each fold has the same class distribution. With imbalanced data, some folds may have very few minority-class samples. Stratified K-Fold preserves the class proportions in every fold.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for fold, (train_idx, val_idx) in enumerate(skf.split(X, y), 1):
    val_dist = np.bincount(y[val_idx])
    print(f"Fold {fold}: val size={len(val_idx)}, class 0={val_dist[0]}, class 1={val_dist[1]}")

Fold 1: val size=200, class 0=170, class 1=30
Fold 2: val size=200, class 0=169, class 1=31
Fold 3: val size=200, class 0=169, class 1=31
Fold 4: val size=200, class 0=169, class 1=31
Fold 5: val size=200, class 0=169, class 1=31

Each fold keeps the same 85/15 ratio as the full dataset. Use stratified K-Fold for classification. It is the default in scikit-learn’s cross_val_score when the estimator is a classifier.

Using cross_val_score

Scikit-learn’s cross_val_score handles the loop for you. Pass a Pipeline so preprocessing is fit inside each fold, never on the validation data.

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

scores = cross_val_score(
    pipeline, X, y,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring="accuracy",
)
print(f"Fold scores: {scores}")
print(f"Mean accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

Fold scores: [0.86  0.845 0.87  0.85  0.85 ]
Mean accuracy: 0.8550 (+/- 0.0089)

Compare several metrics to get the full picture:

for metric in ["accuracy", "roc_auc", "f1", "precision", "recall"]:
    scores = cross_val_score(
        pipeline, X, y,
        cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
        scoring=metric,
    )
    print(f"  {metric:>12}: {scores.mean():.4f} (+/- {scores.std():.4f})")

      accuracy: 0.8550 (+/- 0.0089)
       roc_auc: 0.8337 (+/- 0.0388)
            f1: 0.3418 (+/- 0.0838)
     precision: 0.5704 (+/- 0.0512)
        recall: 0.2529 (+/- 0.0817)

Notice the gap between accuracy (0.855) and recall (0.25). The model is good at overall accuracy but misses three quarters of the minority class. This is why accuracy alone is misleading with imbalanced data. The Model Evaluation page covers these metrics in detail.

Group K-Fold

When samples are not independent, standard cross-validation gives inflated scores. This happens with multiple measurements per patient, multiple cells per sample, or repeated measurements over time. Group K-Fold keeps all samples from the same group in the same fold.

from sklearn.model_selection import GroupKFold

# Simulate patient groups (5 samples per patient)
groups = np.repeat(np.arange(200), 5)

gkf = GroupKFold(n_splits=5)
for fold, (train_idx, val_idx) in enumerate(gkf.split(X, y, groups), 1):
    overlap = set(groups[train_idx]) & set(groups[val_idx])
    print(f"Fold {fold}: overlap={len(overlap)}")

Fold 1: overlap=0
Fold 2: overlap=0
Fold 3: overlap=0
Fold 4: overlap=0
Fold 5: overlap=0

Zero overlap in every fold. No patient appears in both training and validation.

When to use GroupKFold

Use GroupKFold whenever your samples have a natural grouping: multiple biopsies from one patient, multiple time points from one subject, multiple cells from one donor, or technical replicates from one biological sample. Ignore the groups and the model memorizes patient-specific patterns, then appears to generalize when it does not.

Hyperparameter tuning with GridSearchCV

Cross-validation also finds the best hyperparameters. GridSearchCV tries every combination in a grid and keeps the one with the best cross-validated score.

from sklearn.model_selection import GridSearchCV

grid_search = GridSearchCV(
    pipeline,
    param_grid={"classifier__C": [0.001, 0.01, 0.1, 1, 10, 100]},
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring="roc_auc",
    n_jobs=-1,
)
grid_search.fit(X, y)
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best ROC-AUC: {grid_search.best_score_:.4f}")

Best parameters: {'classifier__C': 0.01}
Best ROC-AUC: 0.8384

Use estimator__param syntax for pipeline parameters, scoring="roc_auc" for imbalanced classification, and n_jobs=-1 to use all cores.

RandomizedSearchCV

Grid search tests every combination, which grows exponentially with the number of hyperparameters. Randomized search samples a fixed number of random combinations, which is far more efficient for large spaces.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

random_search = RandomizedSearchCV(
    pipeline,
    param_distributions={"classifier__C": loguniform(0.001, 100)},
    n_iter=20,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring="roc_auc", random_state=42, n_jobs=-1,
)
random_search.fit(X, y)
print(f"Best ROC-AUC: {random_search.best_score_:.4f}")

Best ROC-AUC: 0.8387

Use loguniform for parameters that span orders of magnitude, like regularization strength. n_iter sets how many combinations to try: 20 to 50 for a quick search, 100 or more for a thorough one.

Nested cross-validation

When you use cross-validation for both tuning and performance estimation, you need nested cross-validation. The inner loop tunes hyperparameters; the outer loop estimates generalization. Without it, you evaluate the model on the same data used to pick its hyperparameters, which is optimistic.

inner_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
gs = GridSearchCV(pipeline, {"classifier__C": [0.01, 0.1, 1, 10]},
                  cv=inner_cv, scoring="roc_auc", n_jobs=-1)

outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
nested = cross_val_score(gs, X, y, cv=outer_cv, scoring="roc_auc")
print(f"Nested CV ROC-AUC: {nested.mean():.4f} (+/- {nested.std():.4f})")

Nested CV ROC-AUC: 0.8342 (+/- 0.0277)

Choosing a strategy

Method	Use when
StratifiedKFold	Classification, the default choice
GroupKFold	Samples are grouped (patients, subjects, donors)
KFold	Regression
GridSearchCV	Few hyperparameters, small grid
RandomizedSearchCV	Many hyperparameters, large search space
Nested CV	An unbiased performance estimate alongside tuning

Five-fold is the most common choice. Use ten-fold for small datasets where you need more training data per fold.

Next steps

Cross-validation tells you how good your model is, but which metric should you optimize? The next page, Model Evaluation, covers the full set of classification and regression metrics.