Feature Engineering

The principle that quietly runs modern ML: better features beat better models. A linear regression on the right columns will out-predict a deep network on the wrong ones. ML algorithms can only model what's in the columns you give them — they don't invent structure that isn't there.

Feature engineering is the craft of turning a cleaned DataFrame into the column shapes that make patterns visible to a learner. This lesson is the catalogue: numerical, categorical, date, text, interactions, aggregations, lags — plus the leakage trap that quietly invalidates more student projects than any other single bug.

1. Scaling Numerical Features

Most models — anything that uses distances or gradients (linear, SVM, kNN, neural nets) — care about scale. A feature in [0, 1] and a feature in [0, 1_000_000] are not on equal footing.

python

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Standardisation (z-score): (x - mean) / std → mean 0, std 1
StandardScaler().fit_transform(X)

# Min-max: (x - min) / (max - min) → [0, 1]
MinMaxScaler().fit_transform(X)

# Robust: (x - median) / IQR → outlier-resistant
RobustScaler().fit_transform(X)

+ setup added so this can run · defines X

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')

When to use which:

Scaler	Use when
Standardisation	Most models, roughly symmetric data, no extreme outliers. The default.
Min-max	You need a bounded range (e.g. neural net sigmoid output target, image pixel inputs).
Robust	Heavy-tailed data; a few outliers would dominate `mean`/`std`.

Tree-based models (random forest, gradient boosting) are scale-invariant — they only care about ordering. Don't bother scaling for XGBoost/LightGBM. Do bother for linear models, kNN, SVMs, neural nets.

2. Transforming Skewed Distributions

Right-skewed data (incomes, page views, count data) compresses 99% of observations into a tiny range with a long tail. A log transform spreads them out:

python

import numpy as np
df["log_spend"] = np.log1p(df["spend"])         # log(1 + x), safe for 0
df["sqrt_count"] = np.sqrt(df["count"])

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

log1p over log because real-world counts include zero, and log(0) = -inf. log1p(0) = 0. The Box-Cox / Yeo-Johnson transforms (sklearn.preprocessing.PowerTransformer) generalise this — they find the best exponent automatically.

A symptom that screams "log me": a describe() where max is 10000× the median. Histogram it, log it, histogram again — you'll usually see something that looks like a bell.

3. Binning — When Monotonicity Isn't Real

Some "numeric" features aren't really continuous in their effect:

python

df["age_bin"] = pd.cut(df["age"],
                       bins=[0, 12, 17, 24, 34, 54, 120],
                       labels=["child", "teen", "young_adult",
                               "adult", "middle_age", "senior"])

# Quantile binning — equal-sized buckets
df["spend_decile"] = pd.qcut(df["spend"], q=10, labels=False)

+ setup added so this can run · defines df, pd

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

Why bin? Because the relationship between age and clicked_ad isn't monotonic — a 16-year-old and a 60-year-old might click more than a 35-year-old. A linear model can't fit that with raw age but can with one-hot age bins.

Use pd.cut for fixed thresholds (with domain meaning), pd.qcut for equal-frequency bins (useful for "top 10% spenders").

4. Encoding Categorical Features

The four canonical strategies:

One-hot encoding

python

pd.get_dummies(df["plan"], prefix="plan", drop_first=True)
#    plan_pro  plan_enterprise
# 0       1                 0
# 1       0                 1

+ setup added so this can run · defines pd, df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

pd = _AutoMock('pd')
df = _AutoMock('df')

drop_first=True removes one column to avoid collinearity (useful for linear models). Each category becomes its own 0/1 column. Works well for low-cardinality (< ~50 unique values).

Label encoding

python

from sklearn.preprocessing import LabelEncoder
df["plan_id"] = LabelEncoder().fit_transform(df["plan"])
# Free→0, Pro→1, Enterprise→2

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Warning: this imposes an ordering (Free < Pro < Enterprise). Tree models tolerate it; linear models treat it as a numeric feature and learn nonsense. Only use label encoding when the categories are ordinal (Likert scales, education level).

Target encoding

Replace each category with the mean target value for that category:

python

mean_target = df.groupby("plan")["churned"].mean()
df["plan_te"] = df["plan"].map(mean_target)

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Effective for high-cardinality features (zip codes, product IDs) — but a leakage minefield. If you compute mean_target on the full dataset before splitting, you've baked the validation/test labels into the training feature. The fix: target-encode inside a cross-validation loop, or use sklearn.preprocessing.TargetEncoder (≥ 1.3) which handles this.

Hashing trick

python

from sklearn.feature_extraction import FeatureHasher
hasher = FeatureHasher(n_features=16, input_type="string")
hasher.transform([["zip_94107"], ["zip_10001"]])

For truly high-cardinality features (millions of zip codes, product SKUs) where one-hot would explode memory. Categories are hashed into a fixed-size feature space, with collisions accepted as the cost. Used in ad-tech and search ranking.

5. Date Features

A raw timestamp tells a model nothing. Extract the structure:

python

df["year"]       = df["joined"].dt.year
df["month"]      = df["joined"].dt.month
df["dow"]        = df["joined"].dt.dayofweek          # 0=Mon, 6=Sun
df["is_weekend"] = df["dow"].isin([5, 6]).astype(int)
df["is_eom"]     = df["joined"].dt.is_month_end.astype(int)
df["quarter"]    = df["joined"].dt.quarter

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Cyclical encoding

Day-of-week is circular — Sunday (6) is adjacent to Monday (0), but the raw integer encoding implies they're 6 apart. Sin/cos encoding fixes this:

python

df["dow_sin"] = np.sin(2 * np.pi * df["dow"] / 7)
df["dow_cos"] = np.cos(2 * np.pi * df["dow"] / 7)

# Similarly for hour-of-day (period 24), month-of-year (period 12)
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

+ setup added so this can run · defines df, np

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
np = _AutoMock('np')

The pair (sin, cos) together uniquely identifies a position on the circle and preserves distance — (hour=23, hour=0) are close in (sin, cos) space, as they should be. Linear models and neural nets benefit; tree models can usually figure out the cyclical structure from the raw integer.

See datetime for the timestamp arithmetic side of this.

6. Text Features — Brief Overview

Text is its own world; the one-page version:

python

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Bag-of-words: counts per word
CountVectorizer(max_features=1000).fit_transform(corpus)

# TF-IDF: counts down-weighted by document frequency
TfidfVectorizer(max_features=1000, ngram_range=(1, 2)).fit_transform(corpus)

+ setup added so this can run · defines corpus

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

corpus = _AutoMock('corpus')

TF-IDF (term-frequency × inverse-document-frequency) penalises words that appear everywhere ("the", "of") and amplifies words distinctive to specific documents. It's the workhorse for classical text classification — still a perfectly viable baseline in 2026 before reaching for embeddings.

For modern semantic embeddings (sentence-transformers, OpenAI text-embedding-3) and the rest of the NLP toolbox, see nlp.

7. Interaction Features

Sometimes the combination of two features is what matters:

python

df["price_per_sqft"]    = df["price"] / df["sqft"]              # ratio
df["spend_x_visits"]    = df["spend"] * df["visits"]            # product
df["recency_vs_tenure"] = df["recency_days"] - df["tenure_days"]  # difference

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

A linear model can't learn price / sqft from price and sqft alone — it only knows about additive combinations. Hand-crafting the ratio gives it the signal directly.

Tree models discover interactions through their splits, so they need this less; but a well-named interaction feature still helps interpretability and convergence speed.

8. Aggregations as Features

The most underrated trick in feature engineering: turn historical rows into features on the current row.

python

# Per-user average spend (cross-row aggregation)
df["user_avg_spend"] = df.groupby("user_id")["spend"].transform("mean")

# Rolling 30-day spend (per user, time-aware)
df = df.sort_values(["user_id", "date"])
df["spend_30d"] = (
    df.groupby("user_id")["spend"]
      .rolling(window="30D", on="date").sum()
      .reset_index(level=0, drop=True)
)

groupby().transform() is the magic — it broadcasts the group-level statistic back onto every row of that group, preserving the DataFrame shape. Useful for "this customer's average...", "this product's typical...", "this region's median...".

9. Lag Features (Time Series)

For sequential data, yesterday's value is often the best predictor of today's:

python

df = df.sort_values(["product_id", "date"])
df["sales_lag_1"] = df.groupby("product_id")["sales"].shift(1)
df["sales_lag_7"] = df.groupby("product_id")["sales"].shift(7)
df["sales_rolling_7"] = (
    df.groupby("product_id")["sales"]
      .shift(1)                          # ← critical: shift *before* rolling
      .rolling(7).mean()
      .reset_index(level=0, drop=True)
)

The shift-before-rolling pattern is critical. rolling(7).mean() on raw sales includes today, which is the target — that's leakage. Shift first, then roll, and the window contains only past data.

10. Feature Selection

Once you have 200 features, you usually don't need them all:

python

from sklearn.feature_selection import (
    VarianceThreshold, SelectKBest, mutual_info_classif, RFE
)

# 1. Variance threshold — drop near-constant columns
VarianceThreshold(threshold=0.01).fit_transform(X)

# 2. Correlation — drop one of any pair with |r| > 0.95
corr = X.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
to_drop = [c for c in upper.columns if any(upper[c] > 0.95)]

# 3. Mutual information — non-linear association with target
SelectKBest(mutual_info_classif, k=20).fit_transform(X, y)

# 4. Recursive feature elimination — wrap a model
from sklearn.linear_model import LogisticRegression
RFE(LogisticRegression(), n_features_to_select=10).fit_transform(X, y)

+ setup added so this can run · defines X, y, np

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')
np = _AutoMock('np')

Order of preference: drop the obviously dead (variance, near-perfect correlations) first, then use model-based methods on what remains. Throwing every feature at a single SelectKBest is wasteful when half of them have zero variance.

11. The Leakage Trap

This single section saves more failed projects than the previous ten combined. Leakage is when a feature contains information that wouldn't be available at prediction time — usually because it's derived from the target itself, or from future data.

Examples that have all happened:

"Next month's invoice ID" as a feature predicting "did they churn". The invoice ID exists only if they renewed. AUC: 0.99. Production: random guessing.
Target encoding before train/test split. The validation set's labels leak into the training set's features through the group mean.
Imputing missing values with the full-data mean before splitting. Test-set values influence the train-set fill.
Time-series features computed without .shift() — the rolling average includes today's target.
Standard-scaling on the full dataset before splitting. Test-set min/max leak into train normalisation.

The mental model: at prediction time, what would I actually know? If a feature requires data the production system wouldn't have yet, it's leakage.

The structural fix: fit transformers on TRAIN ONLY, then apply to test:

python

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)     # fit AND transform on train
X_test_scaled  = scaler.transform(X_test)          # ONLY transform on test

+ setup added so this can run · defines X, y, StandardScaler

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')
def StandardScaler(*_a, **_kw):
    print('-> StandardScaler() called')
    return _AutoMock('StandardScaler()')

fit_transform on train, transform on test. Mixing these up is the most common form of leakage in beginner code.

12. Pipeline Reproducibility

The structural answer to leakage and reproducibility: wrap everything in a Pipeline.

python

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

numeric_cols = ["age", "spend", "tenure_days"]
categorical_cols = ["plan", "country"]

preprocessor = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale",  StandardScaler()),
    ]), numeric_cols),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("ohe",    OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_cols),
])

pipeline = Pipeline([
    ("prep", preprocessor),
    ("clf",  RandomForestClassifier(n_estimators=200, random_state=0)),
])

pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)

+ setup added so this can run · defines X_train, y_train, X_test, y_test

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

What this buys you:

Leakage-proof by construction: every transformer is fit on X_train only when you call pipeline.fit.
One object to pickle, version, and load in production. No "step 7 of the notebook" to re-run.
Handles new categories at inference: OneHotEncoder(handle_unknown="ignore") returns all-zero rows for unseen categories instead of throwing.
Cross-validation works correctly: cross_val_score(pipeline, X, y, cv=5) re-fits the entire pipeline on each fold's training set. No leakage, no bookkeeping.

If you're writing manual fit_transform / transform calls across cells, you're going to leak. The pipeline pattern is the cure.

13. Common Mistakes

1. Scaling test data with test statistics
StandardScaler().fit_transform(X_test) re-fits on the test set. Always fit on train, then transform test.

2. Creating features after the split, on the full data
Computing df["user_avg_spend"] on the full df before splitting bakes test-row spend into train features. Compute per-user stats from X_train only.

3. Encoding "Unknown" as if it were a real category
A row with country = NaN shouldn't get its own one-hot column that the model learns to lean on. Impute first, or use handle_unknown="ignore".

4. Not handling new categories at inference time
Default OneHotEncoder raises on unseen categories. Pass handle_unknown="ignore". Test this in development — production will see categories you didn't train on.

5. Forgetting cyclical encoding for periodic features
Raw hour_of_day (0-23) puts 23 and 0 at opposite ends of a number line. They're adjacent in reality. Sin/cos or a model that handles it natively.

6. Target leakage through derived columns
Subtle ones: total_amount_invoiced predicting paid_in_full — the invoice is only generated for paying customers. Walk through every feature: could I compute this at prediction time?

🎯 Your Turn — Engineer User Features

You have a clean user table and need to build five features for a churn model.

Input columns: user_id, signup_date, last_login_date, total_spend, plan (Free/Pro/Enterprise).

Required features:

tenure_days — days from signup_date to a fixed reference_date.

recency_days — days from last_login_date to reference_date.

is_active — 1 if recency_days <= 30, else 0.

log_spend — log1p(total_spend).

plan_free, plan_pro, plan_enterprise — one-hot of plan (no drop_first, so the model sees them all).

Build make_features(df, reference_date) that returns a new DataFrame with the original columns dropped and only the engineered columns + user_id.

python

import pandas as pd

users = pd.DataFrame({
    "user_id":         [1, 2, 3, 4],
    "signup_date":     pd.to_datetime(
        ["2025-01-15", "2024-06-01", "2026-02-20", "2023-11-30"]
    ),
    "last_login_date": pd.to_datetime(
        ["2026-05-01", "2026-03-10", "2026-05-13", "2025-12-25"]
    ),
    "total_spend":     [120.0, 0.0, 450.0, 12.5],
    "plan":            ["Pro", "Free", "Enterprise", "Pro"],
})
# make_features(users, pd.Timestamp("2026-05-14")) → DataFrame with 7 columns

Skeleton:

python

import numpy as np
import pandas as pd

def make_features(df: pd.DataFrame, reference_date: pd.Timestamp) -> pd.DataFrame:
    df = df.copy()

    # TODO 1: tenure_days = (reference_date - signup_date).dt.days
    # TODO 2: recency_days = (reference_date - last_login_date).dt.days
    # TODO 3: is_active = (recency_days <= 30).astype(int)
    # TODO 4: log_spend = np.log1p(total_spend)
    # TODO 5: one-hot plan with prefix="plan"; concat onto df
    # TODO 6: return only user_id + engineered columns
    ...

Hint 1 — Timedelta subtraction

pd.Timestamp - pd.Series[datetime] returns a Series of Timedelta. Use .dt.days to pull the integer day count. Be careful with the subtraction direction — reference - signup gives positive days, not the other way round.

Hint 2 — One-hot with predictable columns

pd.get_dummies(df["plan"], prefix="plan") creates columns like plan_Free, plan_Pro, plan_Enterprise. To match the lowercase spec, lowercase the plan first: df["plan"].str.lower(). To avoid a missing column at inference (a category not seen in training), pass pd.Categorical(df["plan"], categories=["free", "pro", "enterprise"]) before get_dummies — that locks the column set.

Show full solution

python

import numpy as np
import pandas as pd

PLAN_CATEGORIES = ["free", "pro", "enterprise"]

def make_features(df: pd.DataFrame, reference_date: pd.Timestamp) -> pd.DataFrame:
    """Engineer 5 features for a churn model. Pure function — no side effects."""
    df = df.copy()

    # 1. Date-derived numeric features
    df["tenure_days"]  = (reference_date - df["signup_date"]).dt.days
    df["recency_days"] = (reference_date - df["last_login_date"]).dt.days

    # 2. Binary activity flag (engineered from recency)
    df["is_active"] = (df["recency_days"] <= 30).astype(int)

    # 3. Log transform for skewed monetary feature
    df["log_spend"] = np.log1p(df["total_spend"])

    # 4. One-hot the plan with a *locked* category set so unseen categories
    #    at inference time still produce the expected column shape.
    plan_cat = pd.Categorical(df["plan"].str.lower(), categories=PLAN_CATEGORIES)
    plan_oh = pd.get_dummies(plan_cat, prefix="plan").astype(int)
    df = pd.concat([df.reset_index(drop=True),
                    plan_oh.reset_index(drop=True)], axis=1)

    # 5. Project to feature columns only
    cols = ["user_id", "tenure_days", "recency_days", "is_active", "log_spend",
            "plan_free", "plan_pro", "plan_enterprise"]
    return df[cols]


users = pd.DataFrame({
    "user_id":         [1, 2, 3, 4],
    "signup_date":     pd.to_datetime(
        ["2025-01-15", "2024-06-01", "2026-02-20", "2023-11-30"]
    ),
    "last_login_date": pd.to_datetime(
        ["2026-05-01", "2026-03-10", "2026-05-13", "2025-12-25"]
    ),
    "total_spend":     [120.0, 0.0, 450.0, 12.5],
    "plan":            ["Pro", "Free", "Enterprise", "Pro"],
})

feats = make_features(users, pd.Timestamp("2026-05-14"))
print(feats)
#    user_id  tenure_days  recency_days  is_active  log_spend  plan_free  plan_pro  plan_enterprise
# 0        1          484            13          1   4.795791          0         1                0
# 1        2          712            65          0   0.000000          1         0                0
# 2        3           83             1          1   6.111467          0         0                1
# 3        4          896           141          0   2.602690          0         1                0

What the solution gets right:

Pure function — no mutation of the input. Re-runnable, testable.
Date arithmetic via Timedelta.dt.days — cleaner than manual subtraction.
is_active derived from recency_days — engineered feature on top of an engineered feature, in one pass. The model gets both, lets it decide which is more useful.
log1p over log — handles the zero-spend Free user without producing -inf.
PLAN_CATEGORIES locked at module level — means new plans at inference don't crash, and unused plans don't quietly drop columns. The same set of plan columns is produced every call.
reference_date as a parameter — not today(). Reproducible runs, time-travel testing, no hidden coupling to the clock.

In a real project, this make_features lives inside a sklearn.preprocessing.FunctionTransformer and slots into a Pipeline (Section 12). That gives you the same leakage protection automatically — features are computed from train data only when the pipeline is fit.

What You Learned

Scaling: standardisation default; min-max for bounded ranges; robust for outlier-heavy data. Trees don't need it.
Skew fixes: np.log1p, np.sqrt, PowerTransformer for right-skewed data.
Binning (pd.cut, pd.qcut) when the underlying relationship isn't monotonic.
Categorical encoding: one-hot for low cardinality, target encoding for high (with leakage care), hashing for huge.
Date features: extract year/month/dow/is_weekend; sin/cos for cyclical features.
Interactions (ratios, products, differences) hand-engineered for linear models; trees discover them.
Aggregations (groupby().transform()) and lag features (shift() before rolling) for relational/time-series structure.
Feature selection: variance threshold, correlation cutoff, mutual info, RFE — in that order.
The leakage trap — every feature must be computable at prediction time, with no peeking at test data or the target.
Pipeline + ColumnTransformer is the structural cure for leakage. One object, fit on train, applied to test, picklable for production.

Next: Statistics for Data Science — just enough stats to interpret your features, run an A/B test, and not embarrass yourself with a p-value.

1 · The lesson

1. Scaling Numerical Features

2. Transforming Skewed Distributions

3. Binning — When Monotonicity Isn't Real

4. Encoding Categorical Features

One-hot encoding

Label encoding

Target encoding

Hashing trick

5. Date Features

Cyclical encoding

6. Text Features — Brief Overview

7. Interaction Features

8. Aggregations as Features

9. Lag Features (Time Series)

10. Feature Selection

11. The Leakage Trap

12. Pipeline Reproducibility

13. Common Mistakes

🎯 Your Turn — Engineer User Features

What You Learned

Practice this