AutoML with auto-sklearn

auto-sklearn is the academic ancestor of modern AutoML. It came out of the AutoML group at the University of Freiburg and was the first system to win the ChaLearn AutoML Challenge series. Its design choices, Bayesian hyperparameter optimisation, meta-learning warm starts, and ensemble selection over the full search trajectory, shaped most AutoML libraries that followed.

Reference only

This page is not re-run in the benchtoinsights companion container. auto-sklearn’s last release (0.15.0) is pinned to scikit-learn 0.24 and Python 3.8, which cannot coexist with this repo’s modern stack, so it runs in its own official container. The code and outputs come from such a run. It uses the same Wisconsin Breast Cancer split as the AutoGluon and FLAML pages.

Run auto-sklearn via a container

podman pull docker.io/mfeurer/auto-sklearn:master

The image is about 1.7 GB, considerably smaller than the AutoGluon container. It bundles Python 3.8, scikit-learn 0.24, SMAC for Bayesian optimisation, and auto-sklearn 0.15.0.

podman run --rm \
  -v $(pwd)/scripts:/workspace:Z \
  -v $(pwd)/output:/output:Z \
  -w /workspace \
  docker.io/mfeurer/auto-sklearn:master \
  python3 autosklearn_demo.py

The image’s entrypoint expects python3, not python.

What auto-sklearn does

auto-sklearn searches the joint space of preprocessing pipelines and scikit-learn classifiers. The search is run by SMAC, a sequential model-based optimiser that fits a probabilistic surrogate of (configuration → score) and proposes new configurations.

1. Meta-learning warm start. Pull the 25 best configurations seen on similar OpenML datasets and evaluate them first.
2. Bayesian search. SMAC proposes new pipelines and updates the surrogate model.
3. Ensemble selection. A greedy pass over all evaluated models picks a small weighted combination that maximises validation score.

Compared to AutoGluon, the model stable is older (no LightGBM, XGBoost, or CatBoost in this image). Compared to FLAML, the search is more expensive per evaluation but smarter about which configuration to try next.

Worked example: classifying breast tumours

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import autosklearn.classification

data = load_breast_cancer(as_frame=True)
df = data.frame.copy().rename(columns={"target": "diagnosis"})
df.columns = [c.replace(" ", "_") for c in df.columns]

train, test = train_test_split(
    df, test_size=0.20, stratify=df["diagnosis"], random_state=42)
x_train, y_train = train.drop(columns=["diagnosis"]), train["diagnosis"]
x_test, y_test = test.drop(columns=["diagnosis"]), test["diagnosis"]

Fit auto-sklearn with a 120-second budget.

automl = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=120, per_run_time_limit=30,
    metric=autosklearn.metrics.roc_auc, seed=42,
    tmp_folder="/output/autosklearn_tmp",
    delete_tmp_folder_after_terminate=True, n_jobs=1, memory_limit=None)
automl.fit(x_train, y_train)

time_left_for_this_task is the whole-search budget. per_run_time_limit caps a single fit so a slow learner cannot eat the whole budget. memory_limit=None disables the resource limiter, a long-standing source of headaches on container overlay filesystems.

Search statistics

print(automl.sprint_statistics())

auto-sklearn results:
  Metric: roc_auc
  Best validation score: 0.998684
  Number of target algorithm runs: 67
  Number of successful target algorithm runs: 63
  Number of crashed target algorithm runs: 3

In 120 seconds auto-sklearn evaluated 67 candidate pipelines. Single-digit crash counts are normal: SMAC samples aggressively from the tail of the configuration space.

The final ensemble

leaderboard = automl.leaderboard(detailed=True, ensemble_only=True)
print(leaderboard[["rank", "ensemble_weight", "type", "cost"]].to_string())

     rank  ensemble_weight                type      cost
39      1             0.18  passive_aggressive  0.001316
23      6             0.20                 mlp  0.010150
44      8             0.26                 sgd  0.012312

cost is 1 - roc_auc on the validation fold. The ensemble is dominated by linear models (passive aggressive, SGD, liblinear SVC) and a single MLP. No tree-based models survived the selection cut: the 30-second per-run cap favours fast linear learners on this small dataset.

Test set performance

y_pred = automl.predict(x_test)
y_proba = automl.predict_proba(x_test)[:, 1]

  roc_auc            0.9940
  accuracy           0.9386
  mcc                0.8713
  precision          0.9710
  recall             0.9306

Test ROC-AUC 0.99 places auto-sklearn between AutoGluon and FLAML on the same split. Accuracy is lower because the linear ensemble misclassifies a handful of cases the LightGBM-led models get right.

When to reach for auto-sklearn

• You want a research-grade, reproducible AutoML baseline with a well-understood algorithm (SMAC + ensemble selection).
• You are comparing AutoML methods in a paper and need the field’s reference implementation.
• Your problem fits the scikit-learn 0.24 model zoo: linear models, SVMs, MLPs, shallow trees.

Skip it when you need state-of-the-art tabular accuracy (use AutoGluon), fast laptop iteration (use FLAML), or modern Python.

Summary

auto-sklearn is the historical anchor of AutoML. It pioneered the search-plus-ensemble formula that AutoGluon and FLAML refined with newer learners. Today it is most useful as a canonical, reproducible reference baseline, run inside its container.