Feature Engineering

Feature engineering creates new variables from existing data. Good features give models more signal, and on tabular data they often matter more than the choice of algorithm. This page uses a synthetic clinical dataset with numeric, date, text, and categorical columns.

Every code block runs in a container in the companion repository, so the numbers are real.

Setup

import numpy as np
import pandas as pd

np.random.seed(42)
n = 500
df = pd.DataFrame({
    "admission_date": pd.date_range("2020-01-01", periods=n, freq="2D"),
    "discharge_date": pd.date_range("2020-01-03", periods=n, freq="2D")
        + pd.to_timedelta(np.random.randint(1, 14, n), unit="D"),
    "age": np.random.normal(55, 15, n).astype(int),
    "weight_kg": np.random.normal(75, 15, n).round(1),
    "height_cm": np.random.normal(170, 10, n).round(1),
    "systolic_bp": np.random.normal(130, 20, n).astype(int),
    "diastolic_bp": np.random.normal(80, 12, n).astype(int),
    "glucose_mg_dl": np.random.lognormal(4.6, 0.3, n).round(1),
    "diagnosis_text": np.random.choice([
        "Type 2 diabetes with neuropathy", "Hypertension stage 2",
        "Chronic kidney disease stage 3", "Acute myocardial infarction",
        "Heart failure with reduced ejection fraction"], n),
    "medication": np.random.choice(
        ["metformin", "lisinopril", "amlodipine", "insulin", "aspirin"], n),
    "lab_result": np.random.choice(["normal", "low", "high", "critical"], n),
})

Numeric transformations

The most valuable features come from domain knowledge. Combine columns into quantities with known clinical meaning.

df["bmi"] = df["weight_kg"] / (df["height_cm"] / 100) ** 2
df["pulse_pressure"] = df["systolic_bp"] - df["diastolic_bp"]
df["map_pressure"] = df["diastolic_bp"] + df["pulse_pressure"] / 3
df[["weight_kg", "height_cm", "bmi"]].head()

 weight_kg  height_cm       bmi
      74.9      176.8 23.961682
      95.4      167.3 34.084466
      72.8      162.5 27.569231
      64.8      178.7 20.292049
     102.2      181.3 31.092475

BMI, pulse pressure, and mean arterial pressure carry more meaning than the raw inputs.

Many biological measurements are right-skewed. A log transform brings them closer to normal, which helps linear models. Use np.log1p to handle zeros.

print(f"Glucose skewness before log: {df['glucose_mg_dl'].skew():.2f}")
df["log_glucose"] = np.log1p(df["glucose_mg_dl"])
print(f"Glucose skewness after log:  {df['log_glucose'].skew():.2f}")

Glucose skewness before log: 0.86
Glucose skewness after log:  0.11

Binning turns a continuous variable into categories, useful when the effect changes at known thresholds.

df["age_group"] = pd.cut(df["age"], bins=[0, 40, 55, 65, 100],
                         labels=["young", "middle", "senior", "elderly"])
df["age_group"].value_counts().sort_index()

young       91
middle     161
senior     113
elderly    134

When not to bin

Binning discards information: a 41-year-old and a 54-year-old become identical in the “middle” bin. Tree models find their own splits, so binning rarely helps them. It is most useful for linear models with a suspected step-function relationship.

Date features

Dates are not directly usable. Extract numeric components and durations.

df["admission_month"] = df["admission_date"].dt.month
df["admission_dayofweek"] = df["admission_date"].dt.dayofweek
df["length_of_stay"] = (df["discharge_date"] - df["admission_date"]).dt.days
df["is_weekend_admission"] = df["admission_dayofweek"].isin([5, 6]).astype(int)

admission_date  admission_month  admission_dayofweek  length_of_stay  is_weekend_admission
    2020-01-01                1                    2               9                     0
    2020-01-03                1                    4               6                     0
    2020-01-05                1                    6              15                     1
    2020-01-07                1                    1              13                     0
    2020-01-09                1                    3              10                     0

Month captures seasonality, day of week captures weekend effects, and time differences capture duration.

Text features

Free-text fields hold signal the structured columns miss. Start with keyword flags.

df["diagnosis_word_count"] = df["diagnosis_text"].str.split().str.len()
df["has_diabetes"] = df["diagnosis_text"].str.contains("diabetes", case=False).astype(int)
df["has_heart"] = df["diagnosis_text"].str.contains("heart|myocardial|cardiac", case=False).astype(int)
df["has_kidney"] = df["diagnosis_text"].str.contains("kidney|renal", case=False).astype(int)

 diagnosis_word_count  has_diabetes  has_heart  has_kidney
                    6             0          1           0
                    3             0          1           0
                    3             0          1           0
                    6             0          1           0
                    6             0          1           0

For more general text, TF-IDF turns text into weighted numeric vectors.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=10, stop_words="english")
mat = tfidf.fit_transform(df["diagnosis_text"])
pd.DataFrame(mat.toarray(),
             columns=[f"tfidf_{w}" for w in tfidf.get_feature_names_out()]).head(3).round(3)

 tfidf_acute  tfidf_ejection  tfidf_heart  tfidf_infarction  tfidf_myocardial  tfidf_reduced
       0.000           0.577        0.577             0.000             0.000          0.577
       0.577           0.000        0.000             0.577             0.577          0.000
       0.577           0.000        0.000             0.577             0.577          0.000

CatBoost handles text natively

CatBoost can process text columns directly without manual TF-IDF. See the CatBoost page.

Encoding categorical features

For ordered categories, map to integers that reflect the order:

df["lab_result_ordinal"] = df["lab_result"].map({"normal": 0, "low": 1, "high": 2, "critical": 3})

For high-cardinality categoricals, frequency encoding replaces each category with its prevalence, avoiding a wide one-hot expansion:

med_freq = df["medication"].value_counts(normalize=True)
df["medication_freq"] = df["medication"].map(med_freq)
med_freq.round(3)

metformin     0.228
lisinopril    0.226
aspirin       0.202
amlodipine    0.184
insulin       0.160

Polynomial and interaction features

PolynomialFeatures builds interaction terms automatically.

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
poly.fit_transform(df[["age", "bmi", "glucose_mg_dl"]].head())
poly.get_feature_names_out()

Input features: 3, output features: 6
Names: ['age', 'bmi', 'glucose', 'age bmi', 'age glucose', 'bmi glucose']

interaction_only=True skips squared terms. Beware the explosion: 20 inputs at degree 2 give 210 features.

Guidelines

Technique	Best for	Watch out for
Domain features (BMI, ratios)	All models	Needs domain knowledge
Log transform	Skewed numerics	Helps tree models little
Binning	Known threshold effects	Loses information
Date extraction	Time-based data	Consider cyclical encoding
Keyword flags	Short text, known terms	Misses spelling variants
TF-IDF	Longer text	High dimensionality
Frequency encoding	High-cardinality categoricals	Compute inside CV to avoid leakage
Polynomial features	Linear models	Feature explosion

Next steps

With features engineered and selected, the next pages cover specific algorithms. Start with Regression, the fundamental models every data scientist should understand.