Decision Trees & Random Forests

For tabular data (spreadsheets, CSV exports and database rows, which most business ML is built on), tree ensembles are almost always the right first model to try. They handle a mix of numeric and categorical features easily, don't need feature scaling, capture non-linear interactions automatically, and often beat linear baselines, and even deep nets, on structured data.

This lesson covers the whole family: a single decision tree (to build the intuition), Random Forests (bagging), and gradient boosting (LightGBM, as used in production). It ends with permutation importance and SHAP, the current ways to explain what a model has learned.

Runtime note: scikit-learn and LightGBM run right here: press Run on any block. shap does not have a WebAssembly build, so the two SHAP snippets need pip install shap on your own machine. Their expected output is in comments.

1. The Intuition: A Chain of If-Else Questions

A decision tree splits the data with a sequence of binary questions. Each question is chosen so that the groups it creates are purer, meaning each group contains mostly one target value. For a churn model:

python

                Is monthly_spend < £25?
               /                       \
            Yes                          No
       Is tenure < 6m?              Is num_calls > 4?
        /         \                  /            \
   churn=80%   churn=20%        churn=60%      churn=5%

To classify a new customer, walk from the root to a leaf, answering each question, and read off the predicted probability (or majority class) at the leaf.

Splitting criterion: at each node the algorithm tries every feature × every threshold and picks the split that maximises information gain (for classification) or variance reduction (for regression). For classification, "purity" is measured by Gini impurity or entropy. Both work, and Gini is faster to compute.

2. A Single Tree in sklearn

python

from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

print(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate

plt.figure(figsize=(12, 6))
plot_tree(tree, feature_names=load_iris().feature_names,
          class_names=load_iris().target_names, filled=True)
plt.show()

plot_tree draws the full decision diagram. Beginners often skip this step, and so they don't notice that their tree is 27 levels deep and has learned the noise in the data.

Big advantages of trees:

No scaling needed, because splits use thresholds on the raw values.
Numeric and categorical features can be mixed (in sklearn, after you encode the categories, for example with pd.get_dummies).
Captures non-linear effects and feature interactions automatically.
Interpretable: every prediction has an exact decision path.

Big disadvantage: high variance. Change the training data slightly, and you can get a completely different tree. This is why we use ensembles.

3. Hyperparameters: Controlling Overfitting

By default, DecisionTreeClassifier grows until every leaf is pure, which almost always overfits. The settings that matter:

Hyperparameter	Effect
`max_depth`	Hard cap on tree depth. Lower = simpler. Try 3–10
`min_samples_split`	Don't split a node with fewer than N samples. Try 10–50
`min_samples_leaf`	Every leaf must have ≥ N samples. Try 5–20
`max_features`	Subset of features considered at each split (for randomness)
`ccp_alpha`	Cost-complexity pruning: cuts back the tree after it is grown

python

tree = DecisionTreeClassifier(
    max_depth=5,
    min_samples_leaf=10,
    random_state=42,
)

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def DecisionTreeClassifier(*_a, **_kw):
    print('-> DecisionTreeClassifier() called')
    return _AutoMock('DecisionTreeClassifier()')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split
    import matplotlib.pyplot as plt
except Exception:
    pass
_pm_earlier = [
    'from sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42)\ntree.fit(X_train, y_train)\n\nprint(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=load_iris().feature_names,\n          class_names=load_iris().target_names, filled=True)\nplt.show()',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

If you leave max_depth=None (the default), your tree will memorise the training set: train accuracy 1.0, and much lower test accuracy.

4. Ensembles: Many Weak Models, One Strong One

A single tree has high variance. The fix is to train many trees on slightly different versions of the data and average their predictions.

Method	Trees built	How they differ	Variance	Bias
Bagging / Random Forest	In parallel	Bootstrap samples + random feature subsets	↓↓	≈
Boosting	Sequentially	Each tree fixes the previous trees' mistakes	↓	↓↓

Both beat a single tree by a wide margin. RF is the default when you want good results with little tuning. Boosting is the default when you want the best possible performance.

5. Random Forest

Each tree in a Random Forest is trained on:

1. A bootstrap sample: n rows drawn with replacement from the training set (~63% unique rows).
2. At each split, a random subset of features (max_features="sqrt" for classification, 1.0 (all features) for regression).

Predictions are averaged (regression) or majority-voted (classification). The two sources of randomness make the trees less similar to each other, so averaging cancels out much of their individual errors.

python

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(
    n_estimators=300,         # number of trees: more helps, but each extra tree helps less
    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance
    max_features="sqrt",      # √20 ≈ 4 features considered per split
    n_jobs=-1,                # use all CPU cores
    random_state=42,
)
rf.fit(X_train, y_train)
print(rf.score(X_test, y_test))               # ~0.93

Note that in RF, individual trees can be left unpruned (max_depth=None), because averaging reduces the variance. A single unpruned tree, by contrast, overfits badly.

Out-of-bag (OOB) evaluation: every tree saw only ~63% of rows, so the other 37% can be used as a validation set at no extra cost. RandomForestClassifier(oob_score=True) reports it as rf.oob_score_, so you don't need a separate CV split.

6. `feature_importances_` and Its Bias

python

import pandas as pd
importances = pd.Series(rf.feature_importances_, name="importance")
print(importances.sort_values(ascending=False).head())

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.datasets import make_classification
    from sklearn.model_selection import train_test_split
except Exception:
    pass
_pm_earlier = [
    'from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nrf = RandomForestClassifier(\n    n_estimators=300,         # number of trees: more helps, but each extra tree helps less\n    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance\n    max_features="sqrt",      # √20 ≈ 4 features considered per split\n    n_jobs=-1,                # use all CPU cores\n    random_state=42,\n)\nrf.fit(X_train, y_train)\nprint(rf.score(X_test, y_test))               # ~0.93',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

This returns the mean decrease in impurity across all splits where each feature was used. It costs nothing extra (it is computed during training), but it has a well-known bias: it inflates features with many unique values (high cardinality) and continuous features compared to low-cardinality categoricals, because they offer more split points.

Don't use feature_importances_ for high-stakes business decisions. Use permutation importance instead.

7. Permutation Importance: The Unbiased Alternative

For each feature, shuffle its column on the test set and measure the drop in model score. Features whose shuffling barely changes the score were not contributing. Features whose shuffling makes the score drop sharply were important.

python

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42, n_jobs=-1)

perm = pd.DataFrame({
    "feature": [f"f{i}" for i in range(X.shape[1])],
    "importance_mean": result.importances_mean,
    "importance_std":  result.importances_std,
}).sort_values("importance_mean", ascending=False)
print(perm.head())

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')
pd = _AutoMock('pd')
X = _AutoMock('X')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split
    import matplotlib.pyplot as plt
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.datasets import make_classification
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'from sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42)\ntree.fit(X_train, y_train)\n\nprint(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=load_iris().feature_names,\n          class_names=load_iris().target_names, filled=True)\nplt.show()',
    'from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nrf = RandomForestClassifier(\n    n_estimators=300,         # number of trees: more helps, but each extra tree helps less\n    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance\n    max_features="sqrt",      # √20 ≈ 4 features considered per split\n    n_jobs=-1,                # use all CPU cores\n    random_state=42,\n)\nrf.fit(X_train, y_train)\nprint(rf.score(X_test, y_test))               # ~0.93',
    'import pandas as pd\nimportances = pd.Series(rf.feature_importances_, name="importance")\nprint(importances.sort_values(ascending=False).head())',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Properties:

Model-agnostic: works for any fitted estimator with score().
No cardinality bias: measures the effect on predictions, not internal splits.
Computed on test data, so it shows what the model learned that works on new data, not what it memorised.
Slower: the model is scored n_repeats × n_features times. That is acceptable on test sets up to about 100k rows.

Use this for talking to product managers, regulators, or anyone who'll act on "which features matter".

8. Gradient Boosting: Trees in Sequence

Instead of averaging independent trees, boosting trains trees one after another, with each tree learning to predict the residual errors of the previous ensemble. Mathematically:

python

F₀(x) = mean(y)
For m = 1..M:
    rᵢ = yᵢ - F_{m-1}(xᵢ)          # residuals
    fit tree h_m on (X, r)
    F_m(x) = F_{m-1}(x) + η · h_m(x)   # η is the learning rate, a small number (about 0.05)

This gives lower bias than RF (each tree reduces the remaining error), and it often wins Kaggle-style competitions on tabular data.

The major libraries:

Library	Strength	Notes
sklearn `GradientBoostingClassifier`	Built-in, easy	Slow on >100k rows, rarely used in production
XGBoost	Historical kaggle champion	Mature, well-documented, GPU support
LightGBM	Fastest, leaf-wise growth	Widely used in production
CatBoost	Best on categorical-heavy data	Native categorical handling, no need to one-hot

You don't need to know all four. Pick one. LightGBM is widely used in production.

9. LightGBM in Practice

python

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = lgb.LGBMClassifier(
    n_estimators=500,
    learning_rate=0.05,
    num_leaves=31,            # main complexity setting (LightGBM grows leaf by leaf, not level by level)
    min_child_samples=20,
    subsample=0.8,            # row sampling per tree (extra regularisation)
    colsample_bytree=0.8,     # column sampling per tree
    random_state=42,
    n_jobs=-1,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    callbacks=[lgb.early_stopping(stopping_rounds=20)],
)

probs = model.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, probs))           # ~0.97

+ setup added so this can run · re-runs 2 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split
    import matplotlib.pyplot as plt
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.datasets import make_classification
except Exception:
    pass
_pm_earlier = [
    'from sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42)\ntree.fit(X_train, y_train)\n\nprint(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=load_iris().feature_names,\n          class_names=load_iris().target_names, filled=True)\nplt.show()',
    'from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nrf = RandomForestClassifier(\n    n_estimators=300,         # number of trees: more helps, but each extra tree helps less\n    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance\n    max_features="sqrt",      # √20 ≈ 4 features considered per split\n    n_jobs=-1,                # use all CPU cores\n    random_state=42,\n)\nrf.fit(X_train, y_train)\nprint(rf.score(X_test, y_test))               # ~0.93',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

A quick guide to tuning LightGBM:

Start with n_estimators=1000, learning_rate=0.05, num_leaves=31, early stopping on a validation set.
Overfit? Lower num_leaves or raise min_child_samples.
Underfit? Raise num_leaves, lower min_child_samples.
Slow? learning_rate=0.1 with fewer estimators.
Always use early stopping. It picks the right n_estimators for you.

10. Class Imbalance: Three Tools

Real-world classification is almost always imbalanced (fraud, churn, click-through, rare disease). Three options, in order of preference:

python

# 1. Class weighting: for sklearn estimators
RandomForestClassifier(class_weight="balanced")
LogisticRegression(class_weight="balanced")

# 2. scale_pos_weight: for boosting libraries (binary problems)
lgb.LGBMClassifier(scale_pos_weight=(neg_count / pos_count))

# 3. SMOTE: creates synthetic samples of the minority class (last resort)
from imblearn.over_sampling import SMOTE
X_resampled, y_resampled = SMOTE(random_state=42).fit_resample(X_train, y_train)

+ setup added so this can run · defines RandomForestClassifier, LogisticRegression, X_train, y_train, lgb, neg_count, pos_count

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def RandomForestClassifier(*_a, **_kw):
    print('-> RandomForestClassifier() called')
    return _AutoMock('RandomForestClassifier()')
def LogisticRegression(*_a, **_kw):
    print('-> LogisticRegression() called')
    return _AutoMock('LogisticRegression()')
X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
lgb = _AutoMock('lgb')
neg_count = _AutoMock('neg_count')
pos_count = _AutoMock('pos_count')

Always evaluate with F1, AUC, or precision/recall at a chosen threshold, not accuracy. See the regression lesson Section 10.

11. SHAP: Explaining Single Predictions

feature_importances_ says which features mattered on average. SHAP (SHapley Additive exPlanations) tells you which features pushed this specific prediction up or down, and the contributions add up exactly, based on sound mathematics.

python

import shap

explainer = shap.TreeExplainer(model)         # works for any tree ensemble
shap_values = explainer.shap_values(X_test)

# Global summary: like feature importance, but less biased
shap.summary_plot(shap_values, X_test)

# Single-prediction explanation
shap.force_plot(explainer.expected_value, shap_values[0], X_test[0])

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')
X_test = ["alpha", "beta", "gamma"]
X = _AutoMock('X')
y = _AutoMock('y')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split
    import matplotlib.pyplot as plt
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.datasets import make_classification
    import lightgbm as lgb
    from sklearn.metrics import roc_auc_score
except Exception:
    pass
_pm_earlier = [
    'from sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42)\ntree.fit(X_train, y_train)\n\nprint(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=load_iris().feature_names,\n          class_names=load_iris().target_names, filled=True)\nplt.show()',
    'from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nrf = RandomForestClassifier(\n    n_estimators=300,         # number of trees: more helps, but each extra tree helps less\n    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance\n    max_features="sqrt",      # √20 ≈ 4 features considered per split\n    n_jobs=-1,                # use all CPU cores\n    random_state=42,\n)\nrf.fit(X_train, y_train)\nprint(rf.score(X_test, y_test))               # ~0.93',
    'import lightgbm as lgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nmodel = lgb.LGBMClassifier(\n    n_estimators=500,\n    learning_rate=0.05,\n    num_leaves=31,            # main complexity setting (LightGBM grows leaf by leaf, not level by level)\n    min_child_samples=20,\n    subsample=0.8,            # row sampling per tree (extra regularisation)\n    colsample_bytree=0.8,     # column sampling per tree\n    random_state=42,\n    n_jobs=-1,\n)\n\nmodel.fit(\n    X_train, y_train,\n    eval_set=[(X_test, y_test)],\n    callbacks=[lgb.early_stopping(stopping_rounds=20)],\n)\n\nprobs = model.predict_proba(X_test)[:, 1]\nprint(roc_auc_score(y_test, probs))           # ~0.97',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

SHAP has become the usual way to explain tree models in regulated industries (banking, healthcare, insurance). It answers questions like "why was this customer denied?". The maths comes from Shapley values in cooperative game theory. In practice, you get one contribution per feature for each prediction, and the contributions add up to the model output.

12. When Trees Win, and When They Don't

Use trees when:

Tabular data (rows × columns of numbers/categories).
Mixed feature types.
Non-linear effects and interactions expected.
Modest dataset (10² – 10⁷ rows).
You want interpretability via SHAP/permutation.

Don't use trees when:

Text/images/audio: use deep learning (transformers, CNNs).
Very small datasets (fewer than about 100 rows): a linear model with regularisation is more reliable.
You need to predict outside the training range: trees can't predict above max-seen y. Linear models can.
Strict linear-coefficient interpretation required (e.g. medical research papers): use regularised linear models.

Common Mistakes

1. Leaving max_depth=None on a single tree. Unpruned trees (the default) memorise the training data: train accuracy is often 1.0, while test accuracy is much lower. Either tune max_depth/min_samples_leaf or wrap the tree in an ensemble.

2. Trusting feature_importances_ for business decisions. It's biased toward high-cardinality and continuous features. Use permutation importance or SHAP for anything you'll act on.

3. Training on leaked features. A "future" feature included by mistake (e.g. total_revenue when predicting churn, when total revenue is only known after churn happens) makes the model look excellent in CV and then fail in production. Audit your features for temporal leakage. If a feature wouldn't be available at prediction time, drop it.

4. Tuning without early stopping in boosting. If you choose n_estimators by hand, you are mostly guessing. Always pass eval_set=[(X_val, y_val)] and turn on early stopping, and the library finds the best value for you. In LightGBM 4, early stopping is a callback: callbacks=[lgb.early_stopping(stopping_rounds=20)], as in section 9. fit() no longer accepts early_stopping_rounds.

5. One-hot encoding high-cardinality categoricals for tree models. A 10,000-level zip_code one-hot becomes 10,000 sparse columns and makes training slower and the trees worse. Use target encoding, frequency encoding, or libraries with native categorical support (LightGBM categorical_feature=, CatBoost by default).

6. Comparing classifiers on AUC alone, without checking calibration. A boosted model with a high AUC may produce overconfident probabilities. Calibrate with CalibratedClassifierCV if you'll act on predict_proba outputs.

🎯 Your Turn: Random Forest + Permutation Importance

Train a RandomForestClassifier on the inline customer-churn data, then return the top 3 features by permutation importance with their mean scores.

Requirements:

1. Split with test_size=0.25, random_state=42, stratify=y.
2. Train a RandomForestClassifier(n_estimators=200, random_state=42).
3. Compute permutation importance on the test set with n_repeats=10.
4. Return a list of (feature_name, mean_importance) tuples sorted in descending order: the top 3.

Skeleton:

python

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

feature_names = ["tenure_months", "monthly_charges", "num_support_calls",
                 "has_contract", "data_usage_gb"]

X = np.array([
    [ 1, 75,  4, 0,  2.1], [24, 50, 0, 1, 12.0], [ 3, 90, 6, 0,  1.0],
    [36, 45, 1, 1, 15.5], [ 2, 80, 5, 0,  1.8], [48, 40, 0, 1, 20.0],
    [12, 60, 2, 1,  9.0], [ 4, 85, 7, 0,  0.5], [60, 35, 0, 1, 22.0],
    [ 1, 95, 8, 0,  0.3], [18, 55, 1, 1, 11.0], [ 6, 70, 3, 0,  4.0],
    [30, 48, 0, 1, 18.0], [ 2, 88, 6, 0,  1.5], [42, 42, 1, 1, 17.0],
    [ 5, 82, 4, 0,  2.0], [ 8, 65, 2, 1,  8.0], [54, 38, 0, 1, 21.0],
    [ 3, 92, 7, 0,  0.8], [15, 58, 1, 1, 10.0], [ 7, 78, 5, 0,  2.5],
    [40, 44, 0, 1, 19.0], [ 2, 89, 6, 0,  1.2], [20, 52, 1, 1, 13.0],
    [ 4, 84, 5, 0,  1.7], [33, 46, 0, 1, 16.5], [ 1, 91, 8, 0,  0.4],
    [25, 50, 1, 1, 12.5], [ 5, 83, 4, 0,  2.2], [50, 39, 0, 1, 20.5],
])
y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0,
              1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])

def top3_permutation_importance(X, y, feature_names):
    # TODO 1: train/test split (stratify=y)
    # TODO 2: fit RandomForestClassifier
    # TODO 3: permutation_importance on test set, n_repeats=10
    # TODO 4: sort by importances_mean descending, return top 3 (name, score) tuples
    ...

print(top3_permutation_importance(X, y, feature_names))

Hint 1: Stratified split

Pass stratify=y to train_test_split so the test set preserves the class ratio. Without it, with only 30 rows you could get a test set that is 100% one class, and then your test score means nothing.

Hint 2: Sorting and zipping

permutation_importance returns an object with .importances_mean (a NumPy array). Pair it with feature_names via zip, sort descending by the score (use key=lambda p: -p[1] or sorted(..., reverse=True) with key on index 1), and slice [:3].

Show full solution

python

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

feature_names = ["tenure_months", "monthly_charges", "num_support_calls",
                 "has_contract", "data_usage_gb"]

X = np.array([
    [ 1, 75,  4, 0,  2.1], [24, 50, 0, 1, 12.0], [ 3, 90, 6, 0,  1.0],
    [36, 45, 1, 1, 15.5], [ 2, 80, 5, 0,  1.8], [48, 40, 0, 1, 20.0],
    [12, 60, 2, 1,  9.0], [ 4, 85, 7, 0,  0.5], [60, 35, 0, 1, 22.0],
    [ 1, 95, 8, 0,  0.3], [18, 55, 1, 1, 11.0], [ 6, 70, 3, 0,  4.0],
    [30, 48, 0, 1, 18.0], [ 2, 88, 6, 0,  1.5], [42, 42, 1, 1, 17.0],
    [ 5, 82, 4, 0,  2.0], [ 8, 65, 2, 1,  8.0], [54, 38, 0, 1, 21.0],
    [ 3, 92, 7, 0,  0.8], [15, 58, 1, 1, 10.0], [ 7, 78, 5, 0,  2.5],
    [40, 44, 0, 1, 19.0], [ 2, 89, 6, 0,  1.2], [20, 52, 1, 1, 13.0],
    [ 4, 84, 5, 0,  1.7], [33, 46, 0, 1, 16.5], [ 1, 91, 8, 0,  0.4],
    [25, 50, 1, 1, 12.5], [ 5, 83, 4, 0,  2.2], [50, 39, 0, 1, 20.5],
])
y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0,
              1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])

def top3_permutation_importance(X, y, feature_names):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=42, stratify=y,
    )

    rf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
    rf.fit(X_train, y_train)

    result = permutation_importance(
        rf, X_test, y_test,
        n_repeats=10, random_state=42, n_jobs=-1,
    )

    paired = [(name, round(float(score), 3)) for name, score in zip(feature_names, result.importances_mean)]
    paired.sort(key=lambda p: p[1], reverse=True)
    return paired[:3]


print(top3_permutation_importance(X, y, feature_names))
# [('tenure_months', 0.0), ('monthly_charges', 0.0), ('num_support_calls', 0.0)]

Every score is 0.0, and that is the point of the exercise. In this toy data each churn signal (short tenure, high charges, many support calls, no contract) separates the two classes on its own, so shuffling any one column never costs the forest a single prediction. Permutation importance measures what the model loses without a feature, and correlated features cover for each other. On real data, permute correlated features as a group.

Why this is the right answer rather than rf.feature_importances_:

Permutation importance is evaluated on test data, so it reflects what works on new data, not what the model memorised. feature_importances_ is computed from training-time splits.
No cardinality bias: impurity-based importance would overrate monthly_charges (many distinct values), but permutation treats all features fairly.
Model-agnostic: swap RandomForestClassifier for LGBMClassifier and the same code works unchanged.

For the question regulated industries ask ("explain this customer's denial"), you would add SHAP:

python

import shap
explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

rf = _AutoMock('rf')
X_test = _AutoMock('X_test')
feature_names = _AutoMock('feature_names')
X = _AutoMock('X')
y = _AutoMock('y')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split
    import matplotlib.pyplot as plt
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.datasets import make_classification
    import lightgbm as lgb
    from sklearn.metrics import roc_auc_score
    import numpy as np
    from sklearn.inspection import permutation_importance
except Exception:
    pass
_pm_earlier = [
    'from sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42)\ntree.fit(X_train, y_train)\n\nprint(tree.score(X_test, y_test))            # 1.0  because the iris classes are easy to separate\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=load_iris().feature_names,\n          class_names=load_iris().target_names, filled=True)\nplt.show()',
    'from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=2000, n_features=20, n_informative=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nrf = RandomForestClassifier(\n    n_estimators=300,         # number of trees: more helps, but each extra tree helps less\n    max_depth=None,           # each tree can grow deep, and averaging many trees reduces variance\n    max_features="sqrt",      # √20 ≈ 4 features considered per split\n    n_jobs=-1,                # use all CPU cores\n    random_state=42,\n)\nrf.fit(X_train, y_train)\nprint(rf.score(X_test, y_test))               # ~0.93',
    'import lightgbm as lgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nmodel = lgb.LGBMClassifier(\n    n_estimators=500,\n    learning_rate=0.05,\n    num_leaves=31,            # main complexity setting (LightGBM grows leaf by leaf, not level by level)\n    min_child_samples=20,\n    subsample=0.8,            # row sampling per tree (extra regularisation)\n    colsample_bytree=0.8,     # column sampling per tree\n    random_state=42,\n    n_jobs=-1,\n)\n\nmodel.fit(\n    X_train, y_train,\n    eval_set=[(X_test, y_test)],\n    callbacks=[lgb.early_stopping(stopping_rounds=20)],\n)\n\nprobs = model.predict_proba(X_test)[:, 1]\nprint(roc_auc_score(y_test, probs))           # ~0.97',
    'import numpy as np\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.inspection import permutation_importance\n\nfeature_names = ["tenure_months", "monthly_charges", "num_support_calls",\n                 "has_contract", "data_usage_gb"]\n\nX = np.array([\n    [ 1, 75,  4, 0,  2.1], [24, 50, 0, 1, 12.0], [ 3, 90, 6, 0,  1.0],\n    [36, 45, 1, 1, 15.5], [ 2, 80, 5, 0,  1.8], [48, 40, 0, 1, 20.0],\n    [12, 60, 2, 1,  9.0], [ 4, 85, 7, 0,  0.5], [60, 35, 0, 1, 22.0],\n    [ 1, 95, 8, 0,  0.3], [18, 55, 1, 1, 11.0], [ 6, 70, 3, 0,  4.0],\n    [30, 48, 0, 1, 18.0], [ 2, 88, 6, 0,  1.5], [42, 42, 1, 1, 17.0],\n    [ 5, 82, 4, 0,  2.0], [ 8, 65, 2, 1,  8.0], [54, 38, 0, 1, 21.0],\n    [ 3, 92, 7, 0,  0.8], [15, 58, 1, 1, 10.0], [ 7, 78, 5, 0,  2.5],\n    [40, 44, 0, 1, 19.0], [ 2, 89, 6, 0,  1.2], [20, 52, 1, 1, 13.0],\n    [ 4, 84, 5, 0,  1.7], [33, 46, 0, 1, 16.5], [ 1, 91, 8, 0,  0.4],\n    [25, 50, 1, 1, 12.5], [ 5, 83, 4, 0,  2.2], [50, 39, 0, 1, 20.5],\n])\ny = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0,\n              1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])\n\ndef top3_permutation_importance(X, y, feature_names):\n    X_train, X_test, y_train, y_test = train_test_split(\n        X, y, test_size=0.25, random_state=42, stratify=y,\n    )\n\n    rf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)\n    rf.fit(X_train, y_train)\n\n    result = permutation_importance(\n        rf, X_test, y_test,\n        n_repeats=10, random_state=42, n_jobs=-1,\n    )\n\n    paired = [(name, round(float(score), 3)) for name, score in zip(feature_names, result.importances_mean)]\n    paired.sort(key=lambda p: p[1], reverse=True)\n    return paired[:3]\n\n\nprint(top3_permutation_importance(X, y, feature_names))\n# [(\'tenure_months\', 0.0), (\'monthly_charges\', 0.0), (\'num_support_calls\', 0.0)]',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

What You Learned

A decision tree splits data with a chain of if-else questions, maximising information gain (classification) or variance reduction (regression).
Trees handle mixed types, don't need scaling, and capture non-linear effects, but a single tree has high variance.
Random Forest averages many trees grown on bootstrap samples with random feature subsets. It is the standard strong baseline for tabular data.
Gradient boosting (XGBoost, LightGBM, CatBoost) builds trees one after another to reduce the remaining error. It usually performs best on tabular data, and LightGBM is a common choice in production.
feature_importances_ is fast but biased. Use permutation importance for reliable rankings and SHAP for per-prediction explanations.
Class imbalance: class_weight="balanced", scale_pos_weight, or SMOTE. Pick one and evaluate with F1/AUC, not accuracy.
Always use early stopping in boosting, and always tune max_depth/min_samples_leaf for single trees.
Trees aren't for text/images (use deep learning) or tiny datasets (use linear).

Next: Clustering: Unsupervised Pattern Discovery: finding structure when you have no labels.

1. The Intuition: A Chain of If-Else Questions

2. A Single Tree in sklearn

3. Hyperparameters: Controlling Overfitting

4. Ensembles: Many Weak Models, One Strong One

5. Random Forest

6. feature_importances_ and Its Bias

7. Permutation Importance: The Unbiased Alternative

8. Gradient Boosting: Trees in Sequence

9. LightGBM in Practice

10. Class Imbalance: Three Tools

11. SHAP: Explaining Single Predictions

12. When Trees Win, and When They Don't

Common Mistakes

🎯 Your Turn: Random Forest + Permutation Importance

What You Learned

6. `feature_importances_` and Its Bias