AutoML with FLAML

FLAML is the lightweight AutoML library from Microsoft Research. Where AutoGluon trains a large stable of models and stacks them, FLAML runs a smarter, cheaper hyperparameter search over a smaller set of learners. The goal is a strong model in seconds to a few minutes on a laptop, not the absolute best score after an hour of GPU time.

Reference only

This page is not re-run in the benchtoinsights companion container. FLAML installs cleanly with uv, but version 2.1.2 has known compatibility caveats with pandas 3 and XGBoost 3 (noted below) that make it fragile against this repo’s newer pins. The code and outputs come from a run in a compatible environment. It runs on the same Wisconsin Breast Cancer split as the AutoGluon page.

Install FLAML with uv

uv add "flaml[automl]"

The [automl] extra pulls in the search backends. Compared to the 7 GB AutoGluon container, FLAML adds about 30 MB.

A compatibility note: FLAML 2.1.2 predates pandas 3 making StringDtype the default for new column indexes. On pandas 3, add this near the top of your script:

import pandas as pd
pd.options.future.infer_string = False

FLAML 2.1.2 also forwards a callbacks keyword to learner .fit() calls that XGBoost 3 no longer accepts. The simplest workaround is to limit FLAML to the learners that still work: estimator_list=["lgbm", "rf", "extra_tree", "lrl1", "kneighbor"].

What FLAML does differently

FLAML treats AutoML as a cost-aware optimisation problem. Each model fit costs time. FLAML spends more of your budget on configurations that look promising and avoids configurations that are both expensive and unlikely to help.

There is no stacking by default. The output is one model, which is easier to interpret and faster to deploy than a stacked ensemble.

Worked example: classifying breast tumours

import pandas as pd
pd.options.future.infer_string = False

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from flaml import AutoML

data = load_breast_cancer(as_frame=True)
df = data.frame.copy().rename(columns={"target": "diagnosis"})
df.columns = [c.replace(" ", "_") for c in df.columns]

train, test = train_test_split(
    df, test_size=0.20, stratify=df["diagnosis"], random_state=42)
x_train, y_train = train.drop(columns=["diagnosis"]), train["diagnosis"]
x_test, y_test = test.drop(columns=["diagnosis"]), test["diagnosis"]

Fit FLAML with a 120-second budget to match the AutoGluon page.

automl = AutoML()
automl.fit(
    X_train=x_train, y_train=y_train,
    task="classification", metric="roc_auc", time_budget=120,
    estimator_list=["lgbm", "rf", "extra_tree", "lrl1", "kneighbor"], seed=42)

Best model

Unlike AutoGluon, FLAML returns a single best model instead of an ensemble.

print(f"Best learner:            {automl.best_estimator}")
print(f"Best validation roc_auc: {1 - automl.best_loss:.4f}")
print(f"Best config:             {automl.best_config}")

Best learner:            lgbm
Best validation roc_auc: 0.9978
Best config:             {'n_estimators': 135, 'num_leaves': 4, 'min_child_samples': 79,
                          'learning_rate': 1.0, 'colsample_bytree': 0.380,
                          'reg_alpha': 0.086, 'reg_lambda': 9.974}

LightGBM wins, with a small tree (4 leaves) trained for many rounds and a heavy L2 regulariser. FLAML found this by sampling against the cost model, not a grid search.

Per-learner leaderboard

for learner in automl.estimator_list:
    best_loss = automl.best_loss_per_estimator.get(learner)
    print(f"  {learner:12s}  roc_auc = {1 - best_loss:.4f}")

  lgbm          roc_auc = 0.9978
  rf            roc_auc = 0.9912
  extra_tree    roc_auc = 0.9905
  lrl1          roc_auc = 0.9414
  kneighbor     roc_auc = 0.9794

The tree-based learners are within a percentage point of each other. L1-regularised logistic regression is meaningfully worse, consistent with the features being non-linear with respect to diagnosis.

Test set performance

y_pred = automl.predict(x_test)
y_proba = automl.predict_proba(x_test)[:, 1]

  roc_auc            0.9831
  accuracy           0.9649
  mcc                0.9245
  precision          0.9595
  recall             0.9861

Recall is higher than precision here, the opposite of AutoGluon’s profile on the same data: FLAML’s single LightGBM catches almost every malignant case but flags a few benign ones for follow-up.

Feature importance

FLAML exposes the underlying model through automl.model.estimator. For LightGBM, that gives split-count importance directly.

import pandas as pd
model = automl.model.estimator
importances = pd.Series(model.feature_importances_, index=x_train.columns).sort_values(ascending=False)
print(importances.head(5).to_string())

worst_area          36
area_error          35
mean_texture        27
worst_concavity     24
worst_texture       21

These are split counts, not permutation importances. A high count means the feature was useful for many decisions.

When to reach for FLAML

• You want a strong baseline in seconds to minutes, not minutes to hours.
• You are on a laptop or CPU-only machine.
• You want a single deployable model, not a stack.
• You need to fit many models in a loop (one per gene, one per patient subgroup). FLAML’s per-fit overhead is much lower than AutoGluon’s.

Summary

FLAML is the right AutoML tool for the common case: a tabular dataset, modest compute, and a need for a strong model fast. Use AutoGluon when you want the best score and have a few minutes plus a fat container. Use FLAML when you want most of that score in a fraction of the time.