AutoML with AutoGluon

AutoML promises a strong tabular model with one line of code. You hand it a dataframe and a target column. It trains many models, tunes them, ensembles them, and gives you a predictor. AutoGluon (from AWS) is the most production-ready option for tabular AutoML in Python today.

Reference only

Unlike the rest of this section, this page is not re-run in the benchtoinsights companion container. AutoGluon ships a ~7 GB image whose pinned dependencies conflict with the other ML libraries used here, so it runs in AutoGluon’s own official container. The code and outputs below come from such a run.

Run AutoGluon via a container

AutoGluon has many heavy dependencies (PyTorch, LightGBM, XGBoost, CatBoost, FastAI). Installing it next to other ML libraries often hits version pin conflicts. The clean answer is the same one the Reproducibility section recommends: use the official container.

podman pull docker.io/autogluon/autogluon:1.3.1-cpu-framework-ubuntu22.04-py3.11

The image is about 7 GB unpacked. Pull once, run forever. Run a script inside it with two volume mounts: one for your script, one for the output directory.

podman run --rm \
  -v $(pwd)/scripts:/workspace:Z \
  -v $(pwd)/output:/output:Z \
  -w /workspace \
  docker.io/autogluon/autogluon:1.3.1-cpu-framework-ubuntu22.04-py3.11 \
  python autogluon_demo.py

Your local Python environment is untouched. A GPU image is also published (1.3.1-cuda12.4-framework-ubuntu22.04-py3.11) for the neural-network components.

What AutoML does

A good AutoML system:

1. Trains many models from different families: linear models, tree ensembles, neural networks.
2. Tunes each model with sensible defaults or a search budget.
3. Ensembles the models, usually with stacking or weighted averaging.
4. Selects the best ensemble by cross-validation.
5. Returns a single predictor you can call .predict() on.

The goal is a strong baseline fast, not a replacement for careful modelling.

The AutoGluon approach

AutoGluon trains a stable of models and stacks them. The default tabular flow:

• LightGBM, XGBoost, CatBoost
• Random Forest and Extra Trees
• A KNN baseline
• A multi-layer perceptron and a FastAI tabular network
• A weighted ensemble combining the best layer-1 models

Higher-quality presets add k-fold bagging and a second stacking layer.

Worked example: classifying breast tumours

We use the Wisconsin Breast Cancer dataset, a real biomedical benchmark built into scikit-learn: 569 patients, 30 features from images of fine-needle aspirates (radius, texture, perimeter, area, and so on). The target is malignant vs benign.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from autogluon.tabular import TabularPredictor

data = load_breast_cancer(as_frame=True)
df = data.frame.copy().rename(columns={"target": "diagnosis"})
df.columns = [c.replace(" ", "_") for c in df.columns]

train, test = train_test_split(
    df, test_size=0.20, stratify=df["diagnosis"], random_state=42)
print(f"Train shape: {train.shape}")
print(f"Class balance (train, 0=malignant, 1=benign): "
      f"{train['diagnosis'].value_counts().to_dict()}")

Train shape: (455, 31)
Test shape:  (114, 31)
Class balance (train, 0=malignant, 1=benign): {1: 286, 0: 169}

Fit AutoGluon with a 120-second budget. ROC-AUC is the standard metric for binary medical classification.

predictor = TabularPredictor(
    label="diagnosis", path="/output/autogluon_breast_cancer",
    eval_metric="roc_auc",
).fit(train_data=train, time_limit=120, presets="medium_quality")

The training output prints each model and its validation ROC-AUC, then the final ensemble:

Fitting model: LightGBMXT ... 0.9974  = Validation score (roc_auc)
Fitting model: CatBoost ... 0.9954  = Validation score (roc_auc)
Fitting model: NeuralNetFastAI ... 0.9974  = Validation score (roc_auc)
Fitting model: XGBoost ... 0.9964  = Validation score (roc_auc)
Fitting model: NeuralNetTorch ... 0.9995  = Validation score (roc_auc)
Fitting model: WeightedEnsemble_L2 ... 1.0000  = Validation score (roc_auc)
        Ensemble Weights: {'NeuralNetTorch': 0.8, 'XGBoost': 0.2}
AutoGluon training complete, total runtime = 6.81s

Eleven models trained in under 7 seconds. The weighted ensemble combines NeuralNetTorch (80%) and XGBoost (20%).

The leaderboard

leaderboard() evaluates every fitted model on a held-out test set and ranks them.

leaderboard = predictor.leaderboard(test, silent=True)
print(leaderboard[["model", "score_test", "score_val", "fit_time"]].head(6).to_string(index=False))

              model  score_test  score_val  fit_time
    NeuralNetFastAI    0.997024   0.997420  0.587639
     NeuralNetTorch    0.995040   0.999484  1.705164
     ExtraTreesGini    0.994213   0.993292  0.297936
         LightGBMXT    0.993717   0.998452  0.308771
WeightedEnsemble_L2    0.993386   1.000000  1.835659
            XGBoost    0.992725   0.996388  0.117231

Read both score_test and score_val. The ensemble has the highest val score but a slightly lower test score than the best individual model, a small case of ensemble overfit on validation.

Best model performance

perf = predictor.evaluate(test, silent=True)

  roc_auc:           0.9934
  accuracy:          0.9561
  balanced_accuracy: 0.9603
  mcc:               0.9085
  precision:         0.9855
  recall:            0.9444

Precision is higher than recall, the safer error profile for a cancer screen: when the model says “malignant” it is almost always right.

Feature importance

AutoGluon estimates importance by permutation: shuffle each feature and measure the score drop.

importance = predictor.feature_importance(test, silent=True)
print(importance.head(5).to_string())

                     importance    stddev   p_value
worst_texture          0.008598  0.006688  0.022626
mean_concave_points    0.003638  0.003899  0.052637
worst_smoothness       0.001521  0.000862  0.008448
worst_area             0.001323  0.002786  0.174153
compactness_error      0.000794  0.000725  0.035242

Top features are worst_texture, mean_concave_points, and worst_smoothness, which match the cytology literature: texture irregularity and pronounced nuclear concavities are classical malignancy markers.

When AutoML helps, and when to skip it

Reach for AutoML when you have a small or medium tabular dataset, limited time, and want a strong baseline before investing in custom modelling, for example a clinical biomarker analysis or a competition with a deadline.

Skip it when you need an interpretable model, must explain feature importance with confidence, use an unusual target metric, or have unusual data structure (time series, panel, hierarchical). In those cases, fit and tune a single LightGBM.

Comparison

Tool	Strengths	Best for
AutoGluon	Best benchmark accuracy, strong stacking	Top accuracy when you have a few minutes
FLAML	Light, fast, cost-aware tuning	Quick baselines and tight compute
auto-sklearn	Canonical Bayesian search + ensemble	Reference baseline for AutoML comparisons

Summary

AutoML gives you a competitive model in minutes when the alternative is days of trial and error. Use AutoGluon for the highest accuracy on tabular data, run via its official Podman container so your environment stays clean.