PythonMastery
intermediate 20 min read · lesson 5 of 9 in Data Science & ML

Feature Engineering

1 · The lesson

read

An important principle in ML: better features usually help more than a better model. A linear regression on the right columns can predict better than a deep network on the wrong ones. ML algorithms can only learn from what is in the columns you give them. They can't find patterns that the data doesn't contain.

Feature engineering is the skill of turning a cleaned DataFrame into columns that make patterns easy for a model to learn. This lesson covers the main kinds: numerical, categorical, date, text, interactions, aggregations and lags. It also covers leakage, a very common bug that makes results look good when they are not.

1. Scaling Numerical Features

Most models care about scale: anything that uses distances or gradients (linear models, SVM, kNN, neural nets). A feature in [0, 1] and a feature in [0, 1_000_000] are not treated equally.

python
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Standardisation (z-score): (x - mean) / std, giving mean 0 and std 1
StandardScaler().fit_transform(X)

# Min-max: (x - min) / (max - min), giving values in [0, 1]
MinMaxScaler().fit_transform(X)

# Robust: (x - median) / IQR, which outliers barely affect
RobustScaler().fit_transform(X)
+ setup added so this can run · defines X
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')

When to use which:

ScalerUse when
StandardisationMost models, roughly symmetric data, no extreme outliers. The default.
Min-maxYou need a bounded range (e.g. neural net sigmoid output target, image pixel inputs).
RobustHeavy-tailed data, where a few outliers would dominate mean/std.

Tree-based models (random forest, gradient boosting) are scale-invariant: they only use the order of the values. You don't need to scale for XGBoost/LightGBM. You do need to for linear models, kNN, SVMs and neural nets.

2. Transforming Skewed Distributions

Right-skewed data (incomes, page views, counts) has most observations in a small range and a long tail of large values. A log transform spreads them out:

python
import numpy as np
df["log_spend"] = np.log1p(df["spend"])         # log(1 + x), safe for 0
df["sqrt_count"] = np.sqrt(df["count"])
+ setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Use log1p instead of log, because real-world counts include zero, and log(0) = -inf. log1p(0) = 0. The Box-Cox / Yeo-Johnson transforms (sklearn.preprocessing.PowerTransformer) are a general version of this: they find the best exponent automatically.

A clear sign that a column needs a log transform: a describe() where max is 10000× the median. Plot a histogram, apply the log, and plot it again. You will usually see something close to a bell shape.

3. Binning: When the Effect Is Not Monotonic

Some "numeric" features aren't really continuous in their effect:

python
df["age_bin"] = pd.cut(df["age"],
                       bins=[0, 12, 17, 24, 34, 54, 120],
                       labels=["child", "teen", "young_adult",
                               "adult", "middle_age", "senior"])

# Quantile binning: each bucket gets about the same number of rows
df["spend_decile"] = pd.qcut(df["spend"], q=10, labels=False)
+ setup added so this can run · defines df, pd
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

Why bin? Because the relationship between age and clicked_ad isn't monotonic: a 16-year-old and a 60-year-old might both click more than a 35-year-old. A linear model can't fit that with raw age but can with one-hot age bins.

Use pd.cut for fixed thresholds (with domain meaning), pd.qcut for equal-frequency bins (useful for "top 10% spenders").

4. Encoding Categorical Features

The four standard strategies:

One-hot encoding

python
pd.get_dummies(df["plan"], prefix="plan", drop_first=True)
#    plan_free  plan_pro
# 0      False      True
# 1      False     False
+ setup added so this can run · defines pd, df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

pd = _AutoMock('pd')
df = _AutoMock('df')

drop_first=True removes one column to avoid collinearity (useful for linear models). Each category becomes its own 0/1 column. This works well for low-cardinality columns (fewer than about 50 unique values).

Label encoding

python
from sklearn.preprocessing import LabelEncoder
df["plan_id"] = LabelEncoder().fit_transform(df["plan"])
# enterprise=0, free=1, pro=2   (alphabetical, not your business order)
+ setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Warning: this adds an order, and it is alphabetical (enterprise < free < pro), not even your business order. Tree models can cope with it, but linear models treat it as a number and learn wrong patterns. Only use label encoding when the categories are ordinal (Likert scales, education level).

Target encoding

Replace each category with the mean target value for that category:

python
mean_target = df.groupby("plan")["churned"].mean()
df["plan_te"] = df["plan"].map(mean_target)
+ setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

This works well for high-cardinality features (zip codes, product IDs), but it is very easy to cause leakage. If you compute mean_target on the full dataset before splitting, the validation and test labels end up inside the training feature. The fix: target-encode inside a cross-validation loop, or use sklearn.preprocessing.TargetEncoder (≥ 1.3) which handles this for you.

Hashing trick

python
from sklearn.feature_extraction import FeatureHasher
hasher = FeatureHasher(n_features=16, input_type="string")
hasher.transform([["zip_94107"], ["zip_10001"]])

For features with a very large number of values (for example millions of product SKUs), where one-hot encoding would use far too much memory. Categories are hashed into a fixed number of columns. Sometimes two categories land in the same column (a collision), and you accept that as the cost. Used in ad-tech and search ranking.

5. Date Features

A raw timestamp tells a model very little. Extract its parts:

python
df["year"]       = df["joined"].dt.year
df["month"]      = df["joined"].dt.month
df["dow"]        = df["joined"].dt.dayofweek          # 0=Mon, 6=Sun
df["is_weekend"] = df["dow"].isin([5, 6]).astype(int)
df["is_eom"]     = df["joined"].dt.is_month_end.astype(int)
df["quarter"]    = df["joined"].dt.quarter
+ setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

Cyclical encoding

Day-of-week is circular: Sunday (6) comes right before Monday (0), but the raw integers suggest they are 6 apart. Sin/cos encoding fixes this:

python
df["dow_sin"] = np.sin(2 * np.pi * df["dow"] / 7)
df["dow_cos"] = np.cos(2 * np.pi * df["dow"] / 7)

# Do the same for hour of day (period 24) and month of year (period 12)
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)
+ setup added so this can run · defines df, np
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
np = _AutoMock('np')

The pair (sin, cos) together gives each position on the circle a unique value and keeps distances correct: (hour=23, hour=0) are close in (sin, cos) space, as they should be. Linear models and neural nets benefit from this. Tree models can usually learn the cycle from the raw integer.

See datetime for the timestamp arithmetic side of this.

6. Text Features: A Short Overview

Text is a large topic of its own. Here is a short version:

python
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Bag-of-words: counts per word
CountVectorizer(max_features=1000).fit_transform(corpus)

# TF-IDF: counts, made smaller for words found in many documents
TfidfVectorizer(max_features=1000, ngram_range=(1, 2)).fit_transform(corpus)
+ setup added so this can run · defines corpus
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

corpus = _AutoMock('corpus')

TF-IDF (term-frequency × inverse-document-frequency) gives low weight to words that appear everywhere ("the", "of") and high weight to words that are typical of specific documents. It is the standard method for classical text classification, and it is still a good baseline to try before you use embeddings.

For modern semantic embeddings (sentence-transformers, OpenAI text-embedding-3) and the rest of the NLP toolbox, see nlp.

7. Interaction Features

Sometimes the combination of two features is what matters:

python
df["price_per_sqft"]    = df["price"] / df["sqft"]              # ratio
df["spend_x_visits"]    = df["spend"] * df["visits"]            # product
df["recency_vs_tenure"] = df["recency_days"] - df["tenure_days"]  # difference
+ setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

A linear model can't learn price / sqft from price and sqft alone, because it can only add weighted features together. If you create the ratio yourself, the model gets that information directly.

Tree models find interactions through their splits, so they need this less. But a well-named interaction feature still makes the model easier to interpret and can help it train faster.

8. Aggregations as Features

A useful technique that people often forget: turn past rows into features on the current row.

python
# Per-user average spend (cross-row aggregation)
df["user_avg_spend"] = df.groupby("user_id")["spend"].transform("mean")

# Rolling 30-day spend (per user, time-aware)
df = df.sort_values(["user_id", "date"])
df["spend_30d"] = (
    df.groupby("user_id")
      .rolling("30D", on="date")["spend"].sum()
      .to_numpy()                        # same row order as the sorted df
)

groupby().transform() does the work here. It copies the group-level statistic back onto every row of that group, so the DataFrame keeps its shape. Useful for "this customer's average...", "this product's typical...", "this region's median...".

9. Lag Features (Time Series)

For sequential data, yesterday's value is often the best predictor of today's:

python
df = df.sort_values(["product_id", "date"])
df["sales_lag_1"] = df.groupby("product_id")["sales"].shift(1)
df["sales_lag_7"] = df.groupby("product_id")["sales"].shift(7)
df["sales_rolling_7"] = (
    df.groupby("product_id")["sales"]
      .transform(lambda s: s.shift(1).rolling(7).mean())   # important: shift *before* rolling, for each product
)

The shift-before-rolling pattern is essential. rolling(7).mean() on raw sales includes today, which is the target. That is leakage. Shift first, then apply the rolling window, and the window contains only past data.

10. Feature Selection

Once you have 200 features, you usually don't need them all:

python
from sklearn.feature_selection import (
    VarianceThreshold, SelectKBest, mutual_info_classif, RFE
)

# 1. Variance threshold: drop columns that are almost constant
VarianceThreshold(threshold=0.01).fit_transform(X)

# 2. Correlation: drop one column of any pair with |r| > 0.95
corr = X.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
to_drop = [c for c in upper.columns if any(upper[c] > 0.95)]

# 3. Mutual information: also finds non-linear links with the target
SelectKBest(mutual_info_classif, k=20).fit_transform(X, y)

# 4. Recursive feature elimination: wraps a model
from sklearn.linear_model import LogisticRegression
RFE(LogisticRegression(), n_features_to_select=10).fit_transform(X, y)
+ setup added so this can run · defines X, y, np
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')
np = _AutoMock('np')

Order of preference: first drop the features that are clearly useless (no variance, near-perfect correlation with another feature), then use model-based methods on what remains. Throwing every feature at a single SelectKBest is wasteful when half of them have zero variance.

11. The Leakage Trap

This section matters more than any other in the lesson. Leakage is when a feature contains information that wouldn't be available at prediction time, usually because it comes from the target itself, or from future data.

Common examples:

  • "Next month's invoice ID" as a feature predicting "did they churn". The invoice ID exists only if they renewed. The model scores an AUC of 0.99 in testing, then does no better than guessing in production.
  • Target encoding before train/test split. The validation set's labels leak into the training set's features through the group mean.
  • Imputing missing values with the full-data mean before splitting. Test-set values influence the train-set fill.
  • Time-series features computed without .shift(): the rolling average includes today's target.
  • Standard-scaling on the full dataset before splitting. The test set's mean and standard deviation leak into the training normalisation.

The question to ask: at prediction time, what would I really know? If a feature requires data the production system wouldn't have yet, it's leakage.

The fix: fit transformers on the training data only, then apply them to the test data:

python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)     # fit AND transform on train
X_test_scaled  = scaler.transform(X_test)          # ONLY transform on test
+ setup added so this can run · defines X, y, StandardScaler
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')
def StandardScaler(*_a, **_kw):
    print('-> StandardScaler() called')
    return _AutoMock('StandardScaler()')

fit_transform on train, transform on test. Mixing these up is a very common form of leakage in beginner code.

12. Pipeline Reproducibility

The best protection against leakage, and the easiest way to make results reproducible, is to put every step in a Pipeline.

python
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

numeric_cols = ["age", "spend", "tenure_days"]
categorical_cols = ["plan", "country"]

preprocessor = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale",  StandardScaler()),
    ]), numeric_cols),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("ohe",    OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_cols),
])

pipeline = Pipeline([
    ("prep", preprocessor),
    ("clf",  RandomForestClassifier(n_estimators=200, random_state=0)),
])

pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)
+ setup added so this can run · defines X_train, y_train, X_test, y_test
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

What this gives you:

  • No leakage by design: every transformer is fit on X_train only when you call pipeline.fit.
  • One object to pickle, version, and load in production. You don't need to remember to re-run "step 7 of the notebook".
  • Handles new categories at inference: OneHotEncoder(handle_unknown="ignore") returns all-zero rows for unseen categories instead of raising an error.
  • Cross-validation works correctly: cross_val_score(pipeline, X, y, cv=5) re-fits the entire pipeline on each fold's training set. No leakage, and nothing to track by hand.

If you write fit_transform / transform calls by hand across many cells, leakage will happen sooner or later. A pipeline prevents it.

13. Common Mistakes

1. Scaling test data with test statistics
StandardScaler().fit_transform(X_test) re-fits on the test set. Always fit on train, then transform test.

2. Creating features after the split, on the full data
Computing df["user_avg_spend"] on the full df before splitting puts the test rows' spend into the training features. Compute per-user stats from X_train only.

3. Encoding "Unknown" as if it were a real category
A row with country = NaN shouldn't get its own one-hot column that the model learns to depend on. Impute first (for example with SimpleImputer(strategy="most_frequent")). handle_unknown="ignore" doesn't help here: it is for categories that were not seen during training, and OneHotEncoder treats NaN as one more category.

4. Not handling new categories at inference time
Default OneHotEncoder raises on unseen categories. Pass handle_unknown="ignore". Test this in development, because production will see categories you didn't train on.

5. Forgetting cyclical encoding for periodic features
Raw hour_of_day (0-23) puts 23 and 0 at opposite ends of a number line. In reality, they are next to each other. Use sin/cos, or a model that handles this by itself.

6. Target leakage through derived columns
Some are hard to see: total_amount_invoiced predicting paid_in_full, when the invoice is only generated for paying customers. Walk through every feature: could I compute this at prediction time?

🎯 Your Turn: Engineer User Features

You have a clean user table and need to build five features for a churn model.

Input columns: user_id, signup_date, last_login_date, total_spend, plan (Free/Pro/Enterprise).

Required features:


  • tenure_days: days from signup_date to a fixed reference_date.

  • recency_days: days from last_login_date to reference_date.

  • is_active: 1 if recency_days <= 30, else 0.

  • log_spend: log1p(total_spend).

  • plan_free, plan_pro, plan_enterprise: one-hot of plan (no drop_first, so the model sees them all).

Build make_features(df, reference_date) that returns a new DataFrame with the original columns dropped and only the engineered columns + user_id.

python
import pandas as pd

users = pd.DataFrame({
    "user_id":         [1, 2, 3, 4],
    "signup_date":     pd.to_datetime(
        ["2025-01-15", "2024-06-01", "2026-02-20", "2023-11-30"]
    ),
    "last_login_date": pd.to_datetime(
        ["2026-05-01", "2026-03-10", "2026-05-13", "2025-12-25"]
    ),
    "total_spend":     [120.0, 0.0, 450.0, 12.5],
    "plan":            ["Pro", "Free", "Enterprise", "Pro"],
})
# make_features(users, pd.Timestamp("2026-05-14")) returns a DataFrame with 8 columns

Skeleton:

python
import numpy as np
import pandas as pd

def make_features(df: pd.DataFrame, reference_date: pd.Timestamp) -> pd.DataFrame:
    df = df.copy()

    # TODO 1: tenure_days = (reference_date - signup_date).dt.days
    # TODO 2: recency_days = (reference_date - last_login_date).dt.days
    # TODO 3: is_active = (recency_days <= 30).astype(int)
    # TODO 4: log_spend = np.log1p(total_spend)
    # TODO 5: one-hot plan with prefix="plan", then concat onto df
    # TODO 6: return only user_id and the new feature columns
    ...
Hint 1: Timedelta subtraction pd.Timestamp - pd.Series[datetime] returns a Series of Timedelta. Use .dt.days to pull the integer day count. Be careful with the order: reference - signup gives positive days, and the other way round gives negative days.
Hint 2: One-hot with predictable columns pd.get_dummies(df["plan"], prefix="plan") creates columns like plan_Free, plan_Pro, plan_Enterprise. To match the lowercase spec, lowercase the plan first: df["plan"].str.lower(). To avoid a missing column at inference (a category not seen in training), pass pd.Categorical(df["plan"], categories=["free", "pro", "enterprise"]) before get_dummies. That fixes the set of columns.
Show full solution
python
import numpy as np
import pandas as pd

PLAN_CATEGORIES = ["free", "pro", "enterprise"]

def make_features(df: pd.DataFrame, reference_date: pd.Timestamp) -> pd.DataFrame:
    """Build 5 features for a churn model. A pure function: it has no side effects."""
    df = df.copy()

    # 1. Date-derived numeric features
    df["tenure_days"]  = (reference_date - df["signup_date"]).dt.days
    df["recency_days"] = (reference_date - df["last_login_date"]).dt.days

    # 2. Binary activity flag (engineered from recency)
    df["is_active"] = (df["recency_days"] <= 30).astype(int)

    # 3. Log transform for skewed monetary feature
    df["log_spend"] = np.log1p(df["total_spend"])

    # 4. One-hot encode the plan with a *fixed* list of categories, so that new
    #    data with missing or unseen plans still gets the same columns.
    plan_cat = pd.Categorical(df["plan"].str.lower(), categories=PLAN_CATEGORIES)
    plan_oh = pd.get_dummies(plan_cat, prefix="plan").astype(int)
    df = pd.concat([df.reset_index(drop=True),
                    plan_oh.reset_index(drop=True)], axis=1)

    # 5. Keep only the feature columns
    cols = ["user_id", "tenure_days", "recency_days", "is_active", "log_spend",
            "plan_free", "plan_pro", "plan_enterprise"]
    return df[cols]


users = pd.DataFrame({
    "user_id":         [1, 2, 3, 4],
    "signup_date":     pd.to_datetime(
        ["2025-01-15", "2024-06-01", "2026-02-20", "2023-11-30"]
    ),
    "last_login_date": pd.to_datetime(
        ["2026-05-01", "2026-03-10", "2026-05-13", "2025-12-25"]
    ),
    "total_spend":     [120.0, 0.0, 450.0, 12.5],
    "plan":            ["Pro", "Free", "Enterprise", "Pro"],
})

feats = make_features(users, pd.Timestamp("2026-05-14"))
print(feats)
#    user_id  tenure_days  recency_days  is_active  log_spend  plan_free  plan_pro  plan_enterprise
# 0        1          484            13          1   4.795791          0         1                0
# 1        2          712            65          0   0.000000          1         0                0
# 2        3           83             1          1   6.111467          0         0                1
# 3        4          896           140          0   2.602690          0         1                0

What the solution gets right:

  • Pure function: it doesn't change the input, so you can re-run it and test it.
  • Date arithmetic with Timedelta.dt.days, which is cleaner than working out days by hand.
  • is_active derived from recency_days: one engineered feature built from another, in one pass. The model gets both and can decide which is more useful.
  • log1p instead of log: handles the zero-spend Free user without producing -inf.
  • PLAN_CATEGORIES fixed at module level: a new plan at inference doesn't crash the code, and a plan that doesn't appear in the data still gets its column. The same set of plan columns is produced every call.
  • reference_date as a parameter, not today(). Runs are reproducible, you can test with any date, and the result doesn't secretly depend on the clock.

In a real project, this make_features lives inside a sklearn.preprocessing.FunctionTransformer and goes into a Pipeline (Section 12). That gives you the same leakage protection automatically: features are computed from train data only when the pipeline is fit.

What You Learned

  • Scaling: standardisation by default, min-max for bounded ranges, robust for outlier-heavy data. Trees don't need it.
  • Skew fixes: np.log1p, np.sqrt, PowerTransformer for right-skewed data.
  • Binning (pd.cut, pd.qcut) when the underlying relationship isn't monotonic.
  • Categorical encoding: one-hot for low cardinality, target encoding for high (watch for leakage), hashing for very high.
  • Date features: extract year/month/dow/is_weekend, and use sin/cos for cyclical features.
  • Interactions (ratios, products, differences) built by hand for linear models. Trees find them by themselves.
  • Aggregations (groupby().transform()) and lag features (shift() before rolling) for grouped and time-series data.
  • Feature selection: variance threshold, correlation cutoff, mutual info, RFE, in that order.
  • The leakage trap: every feature must be computable at prediction time, without using test data or the target.
  • Pipeline + ColumnTransformer is the best protection against leakage. It is one object: fit on train, applied to test, and saved with pickle for production.

Next: Statistics for Data Science: enough statistics to interpret your features, run an A/B test and read a p-value correctly.

Practice this

on practicepython.in

Short exercises that run in your browser and tell you what your code did, as well as whether a test passed.

Found this useful? Share it with a friend who’s learning Python.