Tabular Deep Learning

Deep learning dominates images and text. On tabular data the story has been different. For years, gradient-boosted trees like XGBoost and LightGBM beat neural networks on most benchmarks. That gap is closing. A new generation of architectures is built specifically for tabular features, and some now match or beat gradient boosting with no tuning.

This page covers the four architectures bench biologists are most likely to encounter: TabNet, TabTransformer, TabPFN, and TabM.

Reference only

The TabPFN example below is illustrative. TabPFN requires a one-time license token to download its pretrained weights, so this page is not re-run in the companion container. The MLP and CNN pages are the runnable deep-learning examples in this section.

Why tabular deep learning is different

Images have spatial structure. Text has sequential structure. Tabular features have neither. Each column means something different: ages, gene expression, one-hot categories. A vanilla MLP throws all this heterogeneity into a single dense layer with no inductive bias for tabular data. Trees, by contrast, split one feature at a time and handle heterogeneous columns naturally. Specialised tabular DL architectures try to recover that inductive bias with attention, feature tokenisation, ensembling, or massive pretraining.

TabNet

Published by Arik and Pfister at Google in 2020. The model works in sequential decision steps. At each step an attention mechanism picks a small subset of features to focus on. Each step is sparse, so you can read off which features mattered at each decision step.

• Strengths: built-in feature selection through attention masks; self-supervised pretraining; interpretable per-step attributions.
• Weaknesses: many hyperparameters; sensitive to learning rate and batch size; often loses to a tuned LightGBM.

TabTransformer

Published by Amazon in 2020. Each categorical feature gets an embedding; the embeddings pass through a transformer encoder that learns contextual representations; numerical features are concatenated and fed to an MLP head.

• Strengths: strong on datasets with many high-cardinality categoricals (patient IDs, hospital codes, gene names).
• Weaknesses: numerical features get less attention from the architecture; heavier than gradient boosting.

TabPFN

The most striking tabular DL idea of recent years. Published by Hollmann et al. in Nature in 2023. PFN stands for prior-data fitted network. The model is a transformer pretrained on millions of synthetic tabular datasets. At inference you give it your training and test data together; the transformer reads them in one forward pass and outputs predictions. There is no gradient training on your dataset. It does in-context learning, the same trick that powers few-shot prompting in LLMs.

For small clinical datasets that removes most of the usual work: you skip tuning, model-selection cross-validation, and training entirely. TabPFN v2 (2025) extends the original ~1000-row limit and matches or beats tuned gradient boosting on small tabular tasks.

• Strengths: zero training time; no tuning; strong on small datasets; classification and regression.
• Weaknesses: hard caps on rows and features; more expensive at predict time; requires accepting a license to download the weights.

This is the model to try first on a small biomarker panel.

TabM

Introduced in 2024, the current state of the art on many tabular benchmarks. The architecture is an ensemble of MLPs that share most of their parameters, giving the variance reduction of ensembling without the cost of training many independent networks. It is not pretrained; you train it from scratch like XGBoost. The advantage is accuracy, not training cost.

When to use which

1. Start with XGBoost or LightGBM. Fast, well-understood, hard to beat on tabular data.
2. If you have fewer than ~1000 patients, try TabPFN. It often beats tuned gradient boosting on small data with zero tuning.
3. If you have more data and want to push accuracy, try TabM.
4. Consider TabNet or TabTransformer only if you need their specific inductive biases.

Tabular DL is not a free win. On many problems gradient boosting still ties or wins. Always benchmark both.

Practical example: TabPFN on simulated biomarker data

The output is illustrative because TabPFN requires a one-time license token. Get one from https://ux.priorlabs.ai and set TABPFN_TOKEN to run it yourself.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=250, n_features=12, n_informative=6, n_redundant=2,
    weights=[0.6, 0.4], flip_y=0.05, random_state=42)
feature_names = ["il6", "crp", "tnf_alpha", "ldh", "ferritin", "d_dimer",
                 "lymphocytes", "neutrophils", "platelets", "albumin",
                 "creatinine", "age"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

Now fit TabPFN. There is no real training: the pretrained transformer sees your training and test data in a single forward pass.

from tabpfn import TabPFNClassifier
from sklearn.metrics import roc_auc_score

clf = TabPFNClassifier(device="cpu")
clf.fit(X_train, y_train)
y_prob = clf.predict_proba(X_test)[:, 1]
print(f"TabPFN test ROC-AUC: {roc_auc_score(y_test, y_prob):.4f}")

TabPFN fit time:     0.42s
TabPFN test ROC-AUC: 0.9234

Compare against an XGBoost baseline:

XGBoost test ROC-AUC: 0.8967

On this kind of small dataset, TabPFN often edges out a tuned XGBoost without any hyperparameter search.

Comparison

Model	Training required	Practical max rows	When to choose
TabPFN v2	No	~10,000	Small clinical or biomarker datasets where tuning is not worth it
TabM	Yes	Millions	Medium to large tabular datasets where you want top accuracy
TabTransformer	Yes	Millions	Many high-cardinality categoricals
TabNet	Yes	Millions	Want attention-based interpretability and willing to tune
XGBoost	Yes	Millions	Strong default baseline, fast, easy to deploy

References

• TabPFN: Hollmann et al. 2023, Nature. https://www.nature.com/articles/s41586-024-08328-6
• TabM: 2024 preprint. https://arxiv.org/abs/2410.24210
• TabNet: Arik and Pfister 2020. https://arxiv.org/abs/1908.07442
• TabTransformer: Huang et al. 2020. https://arxiv.org/abs/2012.06678

Next steps

The AutoML with AutoGluon page shows a tool that can train and ensemble several of these models for you with a single call.