Algorithms Overview

This page gives a brief explanation of each major algorithm for tabular data. For a deeper understanding, each section embeds a StatQuest video that walks through the math and intuition. You do not need to master every algorithm before starting. Begin with logistic regression and gradient boosting, and add others as needed.

Linear regression

Linear regression fits a line (or hyperplane) through the data, predicting a continuous output as a weighted sum of the features plus a bias.

Strengths: fast, interpretable, coefficients show feature direction and importance. Weaknesses: assumes a linear relationship, sensitive to outliers, cannot capture interactions without manual feature engineering. When to use: a good regression baseline, and when you need to explain the model.

Logistic regression

Logistic regression is linear regression with a sigmoid applied to the output, which squashes predictions into the 0 to 1 range as probabilities. Despite the name, it is a classification algorithm.

Strengths: fast, interpretable, well-calibrated probabilities, works well with regularization. Weaknesses: assumes a linear decision boundary. When to use: a binary classification baseline, and when you need interpretable coefficients (which genes raise the probability of response). Often surprisingly competitive on tabular data.

Ridge, LASSO, and elastic net

These are regularized versions of linear and logistic regression. Regularization adds a penalty that discourages large coefficients, reducing overfitting.

Ridge (L2) shrinks all coefficients toward zero but never to zero. LASSO (L1) can shrink coefficients to exactly zero, removing features, so it doubles as feature selection. Elastic net combines L1 and L2 via the l1_ratio parameter.

Decision trees

A decision tree splits the data recursively on feature values. At each node it picks the feature and threshold that best separates the classes, giving a flowchart-like structure.

Strengths: easy to visualize, no scaling needed, handles mixed types, captures nonlinearity. Weaknesses: high variance, prone to overfitting. When to use: rarely alone; they are the building block for random forests and gradient boosting.

Random forest

A random forest trains many trees on random subsets of the data and features, then votes (classification) or averages (regression) across them.

Strengths: robust, hard to overfit, gives feature importances, no scaling needed. Weaknesses: slower, less interpretable, struggles with very high-dimensional sparse data. When to use: a strong default for tabular data with little tuning.

Gradient boosting

Gradient boosting builds trees sequentially, each new tree correcting the errors of the previous ones. This is the concept behind XGBoost, LightGBM, and CatBoost.

XGBoost

XGBoost is an optimized gradient-boosting implementation with regularization, missing- value handling, and parallel training. State of the art on tabular data. Detailed walkthrough on the XGBoost page.

LightGBM

LightGBM grows trees leaf-wise instead of level-wise, which makes it fast on large data with native categorical support. Detailed walkthrough on the LightGBM page.

CatBoost

CatBoost handles categorical features natively without manual encoding, using ordered boosting to avoid target leakage. Detailed walkthrough on the CatBoost page.

AdaBoost

AdaBoost trains weak learners sequentially, up-weighting misclassified samples so later learners focus on hard cases. Simple, but outperformed by modern gradient boosting. Understanding it builds intuition for boosting in general.

Support vector machines

SVMs find the hyperplane that maximizes the margin between classes. The kernel trick handles nonlinear boundaries by mapping data into higher dimensions.

Strengths: effective in high dimensions, versatile kernels. Weaknesses: slow on large datasets, sensitive to scaling, hard to interpret. When to use: small to medium datasets with many features, still used in bioinformatics for gene-expression classification.

K-nearest neighbors

KNN classifies a sample by majority vote of its K closest neighbors in feature space, and averages them for regression.

Strengths: simple, no training phase. Weaknesses: slow at prediction, sensitive to scaling, struggles in high dimensions. When to use: small datasets and quick baselines.

Naive Bayes

Naive Bayes applies Bayes’ theorem with the assumption that features are independent. Despite that oversimplification it works well for many problems.

Strengths: very fast, works with small data, good for text, calibrated probabilities. Weaknesses: the independence assumption limits accuracy on complex problems. When to use: text classification and quick baselines.

Comparison

Algorithm	Scaling needed	Handles categoricals	Handles missing	Speed	Interpretability
Linear / Logistic	Yes	No (encode)	No	Fast	High
Decision Tree	No	No (encode)	Some	Fast	High
Random Forest	No	No (encode)	Some	Medium	Medium
XGBoost	No	No (encode)	Yes	Fast	Low
LightGBM	No	Native	Yes	Very fast	Low
CatBoost	No	Native	Yes	Medium	Low
SVM	Yes	No (encode)	No	Slow	Low
KNN	Yes	No (encode)	No	Slow predict	Low
Naive Bayes	No	Yes	No	Very fast	Medium

What to try first

1. Logistic regression as a fast, interpretable baseline.
2. XGBoost or LightGBM as a strong default on tabular data.
3. CatBoost if you have many categorical features.

Compare them with cross-validation and pick the best.

Next steps

The next page covers Feature Selection: identifying the most important features and removing noise.