Decision Trees & Random Forests

For tabular data — the spreadsheets, CSV exports, and database rows that drive 90% of business ML — tree ensembles are almost always the right first attempt. They handle mixed numeric and categorical features without fuss, ignore feature scaling entirely, capture non-linear interactions out of the box, and routinely beat linear baselines and even deep nets on structured data.

This lesson walks the family: a single decision tree (the intuition), Random Forests (bagging), and gradient boosting (LightGBM in production). We close with permutation importance and SHAP — the modern way to explain what the model actually learned.

Runtime note: scikit-learn and LightGBM run right here — press Run on any block. shap does not have a WebAssembly build, so the two SHAP snippets need pip install shap on your own machine; their expected output is in comments.

1. The Intuition — A Cascade of If-Else Questions

A decision tree splits the data with a sequence of binary questions, each chosen to make the resulting groups purer in the target. For a churn model:

python

                Is monthly_spend < £25?
               /                       \
            Yes                          No
       Is tenure < 6m?              Is num_calls > 4?
        /         \                  /            \
   churn=80%   churn=20%        churn=60%      churn=5%

To classify a new customer, walk from the root to a leaf, answering each question, and read off the predicted probability (or majority class) at the leaf.

Splitting criterion: at each node the algorithm tries every feature × every threshold and picks the split that maximises information gain (for classification) or variance reduction (for regression). For classification, "purity" is measured by Gini impurity or entropy; both work, Gini is faster.

2. A Single Tree in sklearn

python

from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

print(tree.score(X_test, y_test))            # 1.0 — iris is trivially separable

plt.figure(figsize=(12, 6))
plot_tree(tree, feature_names=load_iris().feature_names,
          class_names=load_iris().target_names, filled=True)
plt.show()

plot_tree renders the full decision diagram — beginners often skip this and miss that they're looking at a 27-deep monster overfit to noise.

Big advantages of trees:

No scaling needed — splits are on raw thresholds.
Mixed data types handled natively (after sklearn-encoding categories or using pd.get_dummies).
Captures non-linear effects and feature interactions automatically.
Interpretable — every prediction has an exact decision path.

Big disadvantage: high variance. Resample the training data slightly and you can get a completely different tree. Which is why we ensemble.

3. Hyperparameters — The Overfitting Controls

By default DecisionTreeClassifier grows until every leaf is pure — almost always overfit. The knobs that matter:

Hyperparameter	Effect
`max_depth`	Hard cap on tree depth. Lower = simpler. Try 3–10
`min_samples_split`	Don't split a node with fewer than N samples. Try 10–50
`min_samples_leaf`	Every leaf must have ≥ N samples. Try 5–20
`max_features`	Subset of features considered at each split (for randomness)
`ccp_alpha`	Cost-complexity pruning — post-hoc tree shrinkage

python

tree = DecisionTreeClassifier(
    max_depth=5,
    min_samples_leaf=10,
    random_state=42,
)

+ setup added so this can run · defines DecisionTreeClassifier

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def DecisionTreeClassifier(*_a, **_kw):
    print('-> DecisionTreeClassifier() called')
    return _AutoMock('DecisionTreeClassifier()')

Leave max_depth=None (the default) and your tree will memorise the training set — train accuracy 1.0, test accuracy ugly.

4. Ensembles — Many Weak Models, One Strong One

A single tree is a high-variance estimator. The fix: train many trees on slightly different views of the data and average their predictions.

Method	Trees built	How they differ	Variance	Bias
Bagging / Random Forest	In parallel	Bootstrap samples + random feature subsets	↓↓	≈
Boosting	Sequentially	Each tree fixes the previous trees' mistakes	↓	↓↓

Both beat a single tree by a wide margin. RF is the "set it and forget it" default; boosting is the "I want maximum performance" default.

5. Random Forest

Each tree in a Random Forest is trained on:

1. A bootstrap sample — n rows drawn with replacement from the training set (~63% unique rows).
2. At each split, a random subset of features (max_features="sqrt" for classification, "1.0" aka all for regression).

Predictions are averaged (regression) or majority-voted (classification). The two randomness sources de-correlate the trees, so averaging cancels their individual errors.

python

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(
    n_estimators=300,         # number of trees — more is better up to diminishing returns
    max_depth=None,           # individual trees can be deep; ensemble averaging tames variance
    max_features="sqrt",      # √20 ≈ 4 features considered per split
    n_jobs=-1,                # use all CPU cores
    random_state=42,
)
rf.fit(X_train, y_train)
print(rf.score(X_test, y_test))               # ~0.93

Notable: in RF, individual trees can be left unpruned (max_depth=None) — the ensemble averaging handles variance. Compare to a single tree where unpruned = disaster.

Out-of-bag (OOB) evaluation: every tree saw only ~63% of rows; the other 37% can be used as a free validation set. RandomForestClassifier(oob_score=True) reports it as rf.oob_score_ — no separate CV split needed.

6. `feature_importances_` and Its Bias

python

import pandas as pd
importances = pd.Series(rf.feature_importances_, name="importance")
print(importances.sort_values(ascending=False).head())

+ setup added so this can run · defines rf

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')

This returns the mean decrease in impurity across all splits where each feature was used. It's free (already computed during training), but it has a well-documented bias: it inflates features with many unique values (high cardinality) and continuous features compared to low-cardinality categoricals, because they offer more split points.

Don't use feature_importances_ for high-stakes business decisions. Use permutation importance instead.

7. Permutation Importance — The Unbiased Alternative

For each feature, shuffle its column on the test set and measure the drop in model score. Features whose shuffling barely changes the score were not contributing; features whose shuffling tanks the score were doing real work.

python

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42, n_jobs=-1)

perm = pd.DataFrame({
    "feature": [f"f{i}" for i in range(X.shape[1])],
    "importance_mean": result.importances_mean,
    "importance_std":  result.importances_std,
}).sort_values("importance_mean", ascending=False)
print(perm.head())

+ setup added so this can run · defines rf, X_test, y_test, pd, X

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')
pd = _AutoMock('pd')
X = _AutoMock('X')

Properties:

Model-agnostic — works for any fitted estimator with score().
No cardinality bias — measures effect on predictions, not internal splits.
Computed on test data — reflects generalisation, not training memorisation.
Slower — fits scale with n_repeats × n_features. Tolerable on test sets up to ~100k rows.

Use this for talking to product managers, regulators, or anyone who'll act on "which features matter".

8. Gradient Boosting — Trees in Sequence

Instead of averaging independent trees, boosting trains trees one after another, with each tree learning to predict the residual errors of the previous ensemble. Mathematically:

python

F₀(x) = mean(y)
For m = 1..M:
    rᵢ = yᵢ - F_{m-1}(xᵢ)          # residuals
    fit tree h_m on (X, r)
    F_m(x) = F_{m-1}(x) + η · h_m(x)   # η is learning rate, small (~0.05)

This gives lower bias than RF (it actively reduces error) and usually wins kaggle-style tabular contests by a comfortable margin.

The major libraries:

Library	Strength	Notes
sklearn `GradientBoostingClassifier`	Built-in, easy	Slow on >100k rows; rarely used in production
XGBoost	Historical kaggle champion	Mature, well-documented, GPU support
LightGBM	Fastest, leaf-wise growth	Production default for most teams
CatBoost	Best on categorical-heavy data	Native categorical handling, no need to one-hot

You don't need to know all four — pick one. LightGBM is the most production-popular today.

9. LightGBM in Practice

python

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = lgb.LGBMClassifier(
    n_estimators=500,
    learning_rate=0.05,
    num_leaves=31,            # main complexity knob (LightGBM grows leaf-wise, not depth-wise)
    min_child_samples=20,
    subsample=0.8,            # row sampling per tree (extra regularisation)
    colsample_bytree=0.8,     # column sampling per tree
    random_state=42,
    n_jobs=-1,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    callbacks=[lgb.early_stopping(stopping_rounds=20)],
)

probs = model.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, probs))           # ~0.97

+ setup added so this can run · defines X, y

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')

The cheat sheet for LightGBM tuning:

Start with n_estimators=1000, learning_rate=0.05, num_leaves=31, early stopping on a validation set.
Overfit? Lower num_leaves or raise min_child_samples.
Underfit? Raise num_leaves, lower min_child_samples.
Slow? learning_rate=0.1 with fewer estimators.
Always use early stopping — it picks the right n_estimators for you.

10. Class Imbalance — Three Tools

Real-world classification is almost always imbalanced (fraud, churn, click-through, rare disease). Three knobs, in order of preference:

python

# 1. Class weighting — sklearn estimators
RandomForestClassifier(class_weight="balanced")
LogisticRegression(class_weight="balanced")

# 2. scale_pos_weight — boosting libraries (binary)
lgb.LGBMClassifier(scale_pos_weight=(neg_count / pos_count))

# 3. SMOTE — synthetic minority oversampling (last resort)
from imblearn.over_sampling import SMOTE
X_resampled, y_resampled = SMOTE(random_state=42).fit_resample(X_train, y_train)

+ setup added so this can run · defines RandomForestClassifier, LogisticRegression, X_train, y_train, lgb, neg_count, pos_count

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def RandomForestClassifier(*_a, **_kw):
    print('-> RandomForestClassifier() called')
    return _AutoMock('RandomForestClassifier()')
def LogisticRegression(*_a, **_kw):
    print('-> LogisticRegression() called')
    return _AutoMock('LogisticRegression()')
X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
lgb = _AutoMock('lgb')
neg_count = _AutoMock('neg_count')
pos_count = _AutoMock('pos_count')

Always evaluate with F1, AUC, or precision/recall at a chosen threshold — not accuracy. See the regression lesson Section 10.

11. SHAP — Modern Local Explanations

feature_importances_ says which features mattered on average; SHAP (SHapley Additive exPlanations) tells you which features pushed this specific prediction up or down, with mathematically sound additive contributions.

python

import shap

explainer = shap.TreeExplainer(model)         # works for any tree ensemble
shap_values = explainer.shap_values(X_test)

# Global summary — equivalent to a less biased feature importance
shap.summary_plot(shap_values, X_test)

# Single-prediction explanation
shap.force_plot(explainer.expected_value, shap_values[0], X_test[0])

+ setup added so this can run · defines model, X_test

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')
X_test = ["alpha", "beta", "gamma"]

SHAP has become the de-facto standard for explaining tree models in regulated industries (banking, healthcare, insurance) — it's the answer to "why did this customer get denied?". The maths is Shapley values from cooperative game theory; the practical effect is per-prediction, per-feature contributions that sum to the model output.

12. When Trees Win — and When They Don't

Use trees when:

Tabular data (rows × columns of numbers/categories).
Mixed feature types.
Non-linear effects and interactions expected.
Modest dataset (10² – 10⁷ rows).
You want interpretability via SHAP/permutation.

Don't use trees when:

Text/images/audio — use deep learning (transformers, CNNs).
Very tiny datasets (< ~100 rows) — a linear model with regularisation is more honest.
Need extrapolation outside training range — trees can't predict above max-seen y. Linear models can.
Strict linear-coefficient interpretation required (e.g. medical research papers) — use regularised linear models.

Common Mistakes

1. Leaving max_depth=None on a single tree. Default unpruned trees memorise training data — train acc 1.0, test acc 0.6. Either tune max_depth/min_samples_leaf or wrap the tree in an ensemble.

2. Trusting feature_importances_ for business decisions. It's biased toward high-cardinality and continuous features. Use permutation importance or SHAP for anything you'll act on.

3. Training on leaked features. A "future" feature accidentally in the training set (e.g. total_revenue when predicting churn — total_revenue is computed after churn happens) makes the model look brilliant in CV and fail in production. Audit your features for temporal leakage. If a feature wouldn't be available at prediction time, drop it.

4. Tuning without early stopping in boosting. Manually picking n_estimators is a coin flip. Always pass eval_set=[(X_val, y_val)] and early_stopping_rounds — the library finds the sweet spot for you.

5. One-hot encoding high-cardinality categoricals for tree models. A 10,000-level zip_code one-hot becomes 10,000 sparse columns and ruins both speed and tree quality. Use target encoding, frequency encoding, or libraries with native categorical support (LightGBM categorical_feature=, CatBoost out of the box).

6. Comparing trees and linear models on R² without checking calibration. A high-R² boosted model may produce overconfident probabilities; calibrate with CalibratedClassifierCV if you'll act on predict_proba outputs.

🎯 Your Turn — Random Forest + Permutation Importance

Train a RandomForestClassifier on the inline customer-churn data, then return the top 3 features by permutation importance with their mean scores.

Requirements:

1. Split with test_size=0.25, random_state=42, stratify=y.
2. Train a RandomForestClassifier(n_estimators=200, random_state=42).
3. Compute permutation importance on the test set with n_repeats=10.
4. Return a list of (feature_name, mean_importance) tuples sorted descending — the top 3.

Skeleton:

python

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

feature_names = ["tenure_months", "monthly_charges", "num_support_calls",
                 "has_contract", "data_usage_gb"]

X = np.array([
    [ 1, 75,  4, 0,  2.1], [24, 50, 0, 1, 12.0], [ 3, 90, 6, 0,  1.0],
    [36, 45, 1, 1, 15.5], [ 2, 80, 5, 0,  1.8], [48, 40, 0, 1, 20.0],
    [12, 60, 2, 1,  9.0], [ 4, 85, 7, 0,  0.5], [60, 35, 0, 1, 22.0],
    [ 1, 95, 8, 0,  0.3], [18, 55, 1, 1, 11.0], [ 6, 70, 3, 0,  4.0],
    [30, 48, 0, 1, 18.0], [ 2, 88, 6, 0,  1.5], [42, 42, 1, 1, 17.0],
    [ 5, 82, 4, 0,  2.0], [ 8, 65, 2, 1,  8.0], [54, 38, 0, 1, 21.0],
    [ 3, 92, 7, 0,  0.8], [15, 58, 1, 1, 10.0], [ 7, 78, 5, 0,  2.5],
    [40, 44, 0, 1, 19.0], [ 2, 89, 6, 0,  1.2], [20, 52, 1, 1, 13.0],
    [ 4, 84, 5, 0,  1.7], [33, 46, 0, 1, 16.5], [ 1, 91, 8, 0,  0.4],
    [25, 50, 1, 1, 12.5], [ 5, 83, 4, 0,  2.2], [50, 39, 0, 1, 20.5],
])
y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0,
              1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])

def top3_permutation_importance(X, y, feature_names):
    # TODO 1: train/test split (stratify=y)
    # TODO 2: fit RandomForestClassifier
    # TODO 3: permutation_importance on test set, n_repeats=10
    # TODO 4: sort by importances_mean descending, return top 3 (name, score) tuples
    ...

print(top3_permutation_importance(X, y, feature_names))

Hint 1 — Stratified split

Pass stratify=y to train_test_split so the test set preserves the class ratio. Without it, with only 30 rows you could get a test set that's 100% one class — and your test score becomes meaningless.

Hint 2 — Sorting and zipping

permutation_importance returns an object with .importances_mean (a NumPy array). Pair it with feature_names via zip, sort descending by the score (use key=lambda p: -p[1] or sorted(..., reverse=True) with key on index 1), and slice [:3].

Show full solution

python

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

feature_names = ["tenure_months", "monthly_charges", "num_support_calls",
                 "has_contract", "data_usage_gb"]

X = np.array([
    [ 1, 75,  4, 0,  2.1], [24, 50, 0, 1, 12.0], [ 3, 90, 6, 0,  1.0],
    [36, 45, 1, 1, 15.5], [ 2, 80, 5, 0,  1.8], [48, 40, 0, 1, 20.0],
    [12, 60, 2, 1,  9.0], [ 4, 85, 7, 0,  0.5], [60, 35, 0, 1, 22.0],
    [ 1, 95, 8, 0,  0.3], [18, 55, 1, 1, 11.0], [ 6, 70, 3, 0,  4.0],
    [30, 48, 0, 1, 18.0], [ 2, 88, 6, 0,  1.5], [42, 42, 1, 1, 17.0],
    [ 5, 82, 4, 0,  2.0], [ 8, 65, 2, 1,  8.0], [54, 38, 0, 1, 21.0],
    [ 3, 92, 7, 0,  0.8], [15, 58, 1, 1, 10.0], [ 7, 78, 5, 0,  2.5],
    [40, 44, 0, 1, 19.0], [ 2, 89, 6, 0,  1.2], [20, 52, 1, 1, 13.0],
    [ 4, 84, 5, 0,  1.7], [33, 46, 0, 1, 16.5], [ 1, 91, 8, 0,  0.4],
    [25, 50, 1, 1, 12.5], [ 5, 83, 4, 0,  2.2], [50, 39, 0, 1, 20.5],
])
y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0,
              1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])

def top3_permutation_importance(X, y, feature_names):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=42, stratify=y,
    )

    rf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
    rf.fit(X_train, y_train)

    result = permutation_importance(
        rf, X_test, y_test,
        n_repeats=10, random_state=42, n_jobs=-1,
    )

    paired = list(zip(feature_names, result.importances_mean))
    paired.sort(key=lambda p: p[1], reverse=True)
    return paired[:3]


print(top3_permutation_importance(X, y, feature_names))
# Expected (approximate — small dataset, scores vary):
# [('has_contract', 0.31), ('tenure_months', 0.22), ('monthly_charges', 0.18)]

Why this is the right answer rather than rf.feature_importances_:

Permutation importance is evaluated on test data — it reflects what the model generalises on, not what it memorised. feature_importances_ is computed from training-time splits.
Cardinality unbiased — monthly_charges (many distinct values) would be inflated by impurity-based importance; permutation gives all features a fair shake.
Model-agnostic — swap RandomForestClassifier for LGBMClassifier and the same code works unchanged.

For the regulated-industries answer ("explain this customer's denial"), you'd add SHAP on top:

python

import shap
explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

+ setup added so this can run · defines rf, X_test, feature_names

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')
X_test = _AutoMock('X_test')
feature_names = _AutoMock('feature_names')

What You Learned

A decision tree splits data with a cascade of if-else questions, maximising information gain (classification) or variance reduction (regression).
Trees handle mixed types, don't need scaling, and capture non-linear effects — but a single tree is high-variance.
Random Forest averages many trees grown on bootstrap samples with random feature subsets; the canonical "strong baseline" for tabular data.
Gradient boosting (XGBoost, LightGBM, CatBoost) builds trees sequentially to reduce residual error — usually the best tabular performer. LightGBM is the production default.
feature_importances_ is fast but biased; use permutation importance for honest rankings and SHAP for per-prediction explanations.
Class imbalance: class_weight="balanced", scale_pos_weight, or SMOTE — pick one and evaluate with F1/AUC, not accuracy.
Always use early stopping in boosting; always tune max_depth/min_samples_leaf for single trees.
Trees aren't for text/images (use deep learning) or tiny datasets (use linear).

Next: Clustering — Unsupervised Pattern Discovery — finding structure when you have no labels.

1. The Intuition — A Cascade of If-Else Questions

2. A Single Tree in sklearn

3. Hyperparameters — The Overfitting Controls

4. Ensembles — Many Weak Models, One Strong One

5. Random Forest

6. feature_importances_ and Its Bias

7. Permutation Importance — The Unbiased Alternative

8. Gradient Boosting — Trees in Sequence

9. LightGBM in Practice

10. Class Imbalance — Three Tools

11. SHAP — Modern Local Explanations

12. When Trees Win — and When They Don't

Common Mistakes

🎯 Your Turn — Random Forest + Permutation Importance

What You Learned

6. `feature_importances_` and Its Bias