Decision Trees: Learning If-Else Rules from Data — Machine Learning Fundamentals

A decision tree is what a careful junior developer would write if you asked them to predict something with if-else. Look at one feature and branch. Look at the next feature and branch again. When you reach a leaf, return the answer.

The structure is simple. The clever part is that the tree learns the questions and their order from the data. You feed it rows, it works out which feature to split on first, what value to split at, and how deep to go.

For tabular data with mixed column types and non-linear relationships, trees and the ensembles built from them (random forests, gradient boosting) are among the tools ML engineers use most.

Run these right here: scikit-learn, pandas and matplotlib all work in the browser. The first scikit-learn import takes a few seconds while it downloads. After that it's instant. Expected output is also shown in comments below each block.

1. The Idea: a Worked Example

Predict whether a customer will buy based on three features: age, income, is_existing_customer.

A trained 2-level tree might look like:

python

                  income > 50k ?
                 ╱              ╲
              yes                no
              ╱                    ╲
   is_existing_customer?         age > 35 ?
       ╱        ╲                ╱        ╲
     yes        no             yes        no
      ▼          ▼              ▼          ▼
    BUY      NO BUY           BUY      NO BUY

For a new customer (age 40, income 60k, existing) you walk the tree:
1. income > 50k? → yes, go left.
2. is_existing_customer? → yes, go left.
3. Leaf says BUY.

That's the whole prediction algorithm. There are no matrix multiplications and no gradients: you just walk down a tree of questions. You can read the tree out loud to a non-technical colleague. A neural network can't be explained like that, and even a linear model with many features is harder to explain.

2. How Does It Pick the Questions?

At each node the tree asks: "of all possible questions I could ask (every feature, every split value), which one separates the classes best?"

"Best" is measured by impurity: how mixed the classes are after the split. The most common measures:

Gini impurity (default for classification): the probability of labelling a random sample from the node wrongly, if you chose its label at random using the class proportions in that node.
Entropy: a measure of mixing from information theory.
Variance reduction: used for regression trees.

You don't need to compute these by hand. scikit-learn picks the best split at each node, one node at a time (a "greedy" method). The main point: the tree finds out by itself which features matter and in what order.

3. Code: A 3-Level Tree

python

import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)

print("Test accuracy:", tree.score(X_test, y_test))
# → Test accuracy: 1.0

plt.figure(figsize=(12, 6))
plot_tree(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"],
          class_names=["setosa", "versicolor", "virginica"], filled=True)
plt.show()

You'd see a small tree with three levels of splits, each node labelled with the question (e.g. petal_len <= 2.45), the Gini score, and the class distribution. The leaves are pure (a single class) or almost pure.

The same code works for regression with DecisionTreeRegressor.

4. The Settings: Hyperparameters That Matter

Trees overfit easily. If you don't stop them, they keep growing until every leaf is pure, often with a single training point per leaf. That fits the training data perfectly but is useless on test data. You control growth with three main settings.

Hyperparameter	What it does	Sensible starting value
`max_depth`	Hard cap on tree depth.	3 – 10
`min_samples_split`	Min rows in a node before it's allowed to split.	10 – 50
`min_samples_leaf`	Min rows that must remain in each leaf.	5 – 20
`criterion`	`"gini"` (default) or `"entropy"`. Rarely matters.	leave default

The default max_depth=None puts no limit on the depth. Always set it. A max_depth=3 model is often a better baseline than a max_depth=None model, because the depth limit forces it to generalise.

5. Why Trees Beat Linear Models (Sometimes)

Trees win when:

The relationship is non-linear: sharp thresholds, interactions, sudden changes. Trees handle these without any extra work.
Features are of mixed types: numeric, categorical, boolean. No scaling, no one-hot encoding required by the tree itself (though scikit-learn's implementation still expects numeric input, so encode categorical columns before training).
You want interpretability: plot_tree draws the exact logic the model uses.
Some features have very different scales: trees are not affected by scale. A column in millions and a column in [0,1] work fine together. You don't need StandardScaler.

Trees lose when:

The data is very small (under ~50 rows). A linear baseline usually does better here.
The data is text or images. Trees can't see structure across pixels or sequences of words. Deep learning does much better.
You need smooth predictions for something that changes smoothly. A regression tree's predictions form steps, not curves.

6. The High-Variance Problem and Its Fix

Single decision trees have high variance: a small change to the training data can produce a completely different tree, with a completely different decision boundary. They overfit, and they are unstable.

One method reduces both problems: fit many trees and combine their predictions.

Random Forest: fit many independent trees

python

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=100, max_depth=10,
                                random_state=42).fit(X_train, y_train)
print("Random forest:", forest.score(X_test, y_test))
# → Random forest: 1.0

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import matplotlib.pyplot as plt
    from sklearn.datasets import load_iris
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.model_selection import train_test_split
except Exception:
    pass
_pm_earlier = [
    'import matplotlib.pyplot as plt\nfrom sklearn.datasets import load_iris\nfrom sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)\n\nprint("Test accuracy:", tree.score(X_test, y_test))\n# → Test accuracy: 1.0\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"],\n          class_names=["setosa", "versicolor", "virginica"], filled=True)\nplt.show()',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Each tree sees a random sample of the rows, drawn with replacement (bootstrapping), and a random subset of the features at each split. The forest combines the predictions of all the trees. The random differences between trees cancel out, the boundary becomes smoother, and accuracy usually improves.

n_estimators=100 is a good default. More trees rarely make results worse. They just take longer to train.

Gradient Boosting: fit trees one after another, each correcting the previous ones

python

from sklearn.ensemble import GradientBoostingClassifier

gbm = GradientBoostingClassifier(n_estimators=100, max_depth=3,
                                 learning_rate=0.1,
                                 random_state=42).fit(X_train, y_train)
print("Gradient boosting:", gbm.score(X_test, y_test))
# → Gradient boosting: 1.0

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import matplotlib.pyplot as plt
    from sklearn.datasets import load_iris
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.model_selection import train_test_split
except Exception:
    pass
_pm_earlier = [
    'import matplotlib.pyplot as plt\nfrom sklearn.datasets import load_iris\nfrom sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)\n\nprint("Test accuracy:", tree.score(X_test, y_test))\n# → Test accuracy: 1.0\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"],\n          class_names=["setosa", "versicolor", "virginica"], filled=True)\nplt.show()',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Boosting trains tree #2 to fix tree #1's mistakes, tree #3 to fix the residual after #1+#2, and so on. The trees are kept shallow (depth 3 is typical). Each tree on its own is weak, but together they are strong.

In Kaggle and in industry, gradient boosting (often via XGBoost, LightGBM, or CatBoost) is very often the best approach on tabular data. Random forest is safer and easier to tune.

Forward link: deeper coverage in trees and ensembles.

7. Feature Importance

Trees give you feature_importances_: a score for each feature that shows how much it contributed to the splits in the tree.

python

import pandas as pd

importances = pd.Series(forest.feature_importances_,
                        index=["sepal_len", "sepal_wid", "petal_len", "petal_wid"])
print(importances.sort_values(ascending=False))
# → petal_len    0.439994
# → petal_wid    0.421522
# → sepal_len    0.108098
# → sepal_wid    0.030387
# → dtype: float64

+ setup added so this can run · re-runs 2 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

forest = _AutoMock('forest')
X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import matplotlib.pyplot as plt
    from sklearn.datasets import load_iris
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    from sklearn.model_selection import train_test_split
    from sklearn.ensemble import RandomForestClassifier
except Exception:
    pass
_pm_earlier = [
    'import matplotlib.pyplot as plt\nfrom sklearn.datasets import load_iris\nfrom sklearn.tree import DecisionTreeClassifier, plot_tree\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\n\ntree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)\n\nprint("Test accuracy:", tree.score(X_test, y_test))\n# → Test accuracy: 1.0\n\nplt.figure(figsize=(12, 6))\nplot_tree(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"],\n          class_names=["setosa", "versicolor", "virginica"], filled=True)\nplt.show()',
    'from sklearn.ensemble import RandomForestClassifier\n\nforest = RandomForestClassifier(n_estimators=100, max_depth=10,\n                                random_state=42).fit(X_train, y_train)\nprint("Random forest:", forest.score(X_test, y_test))\n# → Random forest: 1.0',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Two warnings before you show a "feature importance" chart to your manager:

1. Biased toward high-cardinality features. A column with 1000 unique values has more possible split points than a binary column, so it can get a higher score only because it has more values.
2. It shows correlation, not causation. A feature could be important because it's a proxy for the real cause.

For a more reliable measure, use permutation importance (sklearn.inspection.permutation_importance): shuffle one feature and measure how much accuracy drops.

Common Mistakes

Leaving max_depth=None. Trees grown with the default settings memorise their training set. Always cap the depth.
Trusting a single tree. Run the same code with a different random_state and the tree may look entirely different. For decisions that matter, use a forest.
One-hot encoding everything. Trees handle ordinal categoricals fine if encoded as integers. One-hot encoding greatly increases the number of features and slows training. (In scikit-learn, DecisionTreeClassifier and RandomForestClassifier still need numeric input. HistGradientBoostingClassifier can handle categorical columns natively with its categorical_features parameter.)
Reading too much into the exact tree structure. "Look, the model says income > £49,237.50!" That exact number comes from this particular training sample. The feature is meaningful, but the exact threshold can change a lot with different data.
Forgetting to set n_jobs=-1 on random forests. The trees in a forest can be trained in parallel, and that one setting uses all your CPU cores.

🎯 Your Turn: Tree on the Wine Dataset

The built-in load_wine dataset has 178 wines with 13 chemical features and 3 cultivar classes.

Your task:

1. Load the dataset.
2. Split 80/20 with random_state=42.
3. Train a DecisionTreeClassifier(max_depth=3, random_state=42).
4. Return the test accuracy rounded to 3 decimals and the top feature by importance, as a tuple.

python

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

def train_wine_tree():
    data = load_wine()
    X, y = data.data, data.target
    feature_names = data.feature_names

    # TODO 1: split 80/20, random_state=42
    # TODO 2: fit DecisionTreeClassifier(max_depth=3, random_state=42)
    # TODO 3: compute test_acc = round(tree.score(X_test, y_test), 3)
    # TODO 4: find the feature name with the highest .feature_importances_
    # TODO 5: return (test_acc, top_feature_name)

    pass

print(train_wine_tree())
# → expected something like (0.944, 'flavanoids')

Hint 1: feature importances are an array

tree.feature_importances_ is a NumPy array, one entry per feature in the same order as feature_names. To find the top one, use feature_names[tree.feature_importances_.argmax()].

Hint 2: the answer should be a tuple of (float, str)

Don't forget to round the accuracy. Final line: return (test_acc, top_feature_name).

Show full solution

python

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

def train_wine_tree():
    data = load_wine()
    X, y = data.data, data.target
    feature_names = data.feature_names

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )

    tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)

    test_acc = round(tree.score(X_test, y_test), 3)
    top_feature_name = feature_names[tree.feature_importances_.argmax()]

    return (test_acc, top_feature_name)

print(train_wine_tree())
# → (0.944, 'flavanoids')

A depth-3 tree on the raw wine measurements reaches 94% test accuracy. Try max_depth=10: the training accuracy will probably reach 1.0, but the test accuracy won't improve much. That is overfitting. Then try RandomForestClassifier and compare the test accuracy.

What You Learned

A decision tree is a learned hierarchy of if-else questions. Each node splits on the feature that best separates the classes.
The three knobs that control overfitting: max_depth, min_samples_split, min_samples_leaf. Don't leave max_depth=None.
Trees handle non-linear relationships and different scales without extra work. No scaling is needed.
Single trees are high-variance. Fix with ensembles: random forest (parallel, averaged) or gradient boosting (sequential, error-correcting).
feature_importances_ is useful but biased. Prefer permutation importance for careful analysis.
Gradient boosting (XGBoost / LightGBM) often gives the best results on tabular data, and random forest is the easier baseline.

Next: Your First ML Pipeline: combining preprocessing and a model into one object that is hard to misuse.