Linear & Logistic Regression

Most supervised learning in production is one of two tasks: predict a number (regression) or predict a category (classification). Linear regression and logistic regression are the standard models for these two tasks. They are still the right place to start for nearly every tabular ML problem. They don't always win, but they give you a baseline score that any more complex model must beat.

This lesson goes through the full pipeline (split, fit, evaluate, regularise, tune) for both regression and classification, with the metrics that matter and the traps that catch beginners. By the end, you'll know why accuracy can mislead you, why Ridge always needs scaled inputs, and where data leakage hides.

Runtime note: scikit-learn, pandas and NumPy run right here: press Run on any block. The first scikit-learn import takes a few seconds while the package downloads. After that it's instant. Expected output is also shown in comments.

1. The Two Standard Tasks

Task	Output	Example	Loss
Regression	Real number	House price, daily revenue, temperature	MSE / MAE
Classification	Discrete label	Spam vs ham, churn vs stay, 10-class digit	Log-loss / cross-entropy

Both models compute the same thing, a weighted sum of the inputs. They differ only in the last step. Regression returns the sum directly. Classification passes the sum through a non-linear function to get a probability, then compares it with a threshold.

2. Linear Regression: Fit the Best Line

Given points (x, y), find the line y = w·x + b that minimises the sum of squared errors between predicted and actual y. With multiple inputs the line becomes a hyperplane:

python

y = w₁·x₁ + w₂·x₂ + ... + wₙ·xₙ + b

The weights wᵢ say how much y moves when xᵢ moves by one unit, all else held constant. The intercept b is the prediction when every xᵢ = 0.

python

# Local: pip install scikit-learn numpy
import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3], [4], [5]])      # one feature, five samples
y = np.array([2.1, 4.0, 6.1, 7.9, 10.2])      # roughly y = 2x

model = LinearRegression()
model.fit(X, y)

print(model.coef_)                            # [2.01]   slope is about 2
print(model.intercept_)                       # 0.03      intercept is about 0
print(model.predict([[6]]))                   # [12.09]   a prediction outside the training range
print(model.score(X, y))                      # 0.9987    the R² score

fit(X, y) solves the least-squares problem directly, with no iteration. predict applies the learned weights. score returns R² (coefficient of determination) by default.

3. Multiple Features and Coefficient Interpretation

python

import numpy as np
from sklearn.linear_model import LinearRegression

# rows are samples, columns are [rooms, age_years, sqft]
X = np.array([[3, 10, 1500],
              [4, 25, 2200],
              [2, 40,  900],
              [5,  5, 2800],
              [3, 15, 1700]])
y = np.array([420_000, 530_000, 220_000, 740_000, 460_000])

model = LinearRegression().fit(X, y)
print(dict(zip(["rooms", "age", "sqft"], model.coef_)))
# {'rooms': np.float64(-3671.87...), 'age': np.float64(-2693.36...), 'sqft': np.float64(228.61...)}

Read coefficients carefully: "holding age and sqft fixed, one extra room is worth about −£3.7k". A negative price per room looks wrong, and it is a typical small-data problem. With 5 houses, rooms and sqft rise together, so the model can't separate their effects and divides the effect between them in an arbitrary way. Two more pitfalls:

Scale dependence: sqft has a tiny coefficient but a huge range. rooms has a big coefficient but a small range. You can't rank features by the size of their raw coefficients unless you have standardised the inputs first (mean 0, std 1). See feature.
Causation: a coefficient measures an association, not a cause. If sunrooms and higher prices only appear together in your data, adding a sunroom won't necessarily cause the price to rise.

4. Train/Test Split: Always Required

Evaluating a model on the data it learned from is like testing a student with questions they have already seen the answers to. You will always overestimate its performance.

The 5 houses from section 3 are too few to split: 20% of 5 is one house, and R² needs at least two houses to compare. So this example makes 50 houses from a known price rule plus some random noise.

python

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
rooms = rng.integers(1, 6, size=50)
age   = rng.integers(0, 50, size=50)
sqft  = rng.integers(600, 3000, size=50)
X = np.column_stack([rooms, age, sqft])
y = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)                                             # 40 houses to train on, 10 to test on

model = LinearRegression().fit(X_train, y_train)
print(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)

test_size=0.2: keep 20% aside for evaluation.
random_state=42: makes the split reproducible. Pick any integer, but use the same one across the team.
For time series, never split randomly. Use TimeSeriesSplit, so the test set always comes after the training data in time.

5. Regression Metrics: When to Use Which

python

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_pred = model.predict(X_test)

mae  = mean_absolute_error(y_test, y_pred)
mse  = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2   = r2_score(y_test, y_pred)

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')
model = _AutoMock('model')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
except Exception:
    pass
_pm_earlier = [
    '# Local: pip install scikit-learn numpy\nimport numpy as np\nfrom sklearn.linear_model import LinearRegression\n\nX = np.array([[1], [2], [3], [4], [5]])      # one feature, five samples\ny = np.array([2.1, 4.0, 6.1, 7.9, 10.2])      # roughly y = 2x\n\nmodel = LinearRegression()\nmodel.fit(X, y)\n\nprint(model.coef_)                            # [2.01]   slope is about 2\nprint(model.intercept_)                       # 0.03      intercept is about 0\nprint(model.predict([[6]]))                   # [12.09]   a prediction outside the training range\nprint(model.score(X, y))                      # 0.9987    the R² score',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\n\n# rows are samples, columns are [rooms, age_years, sqft]\nX = np.array([[3, 10, 1500],\n              [4, 25, 2200],\n              [2, 40,  900],\n              [5,  5, 2800],\n              [3, 15, 1700]])\ny = np.array([420_000, 530_000, 220_000, 740_000, 460_000])\n\nmodel = LinearRegression().fit(X, y)\nprint(dict(zip(["rooms", "age", "sqft"], model.coef_)))\n# {\'rooms\': np.float64(-3671.87...), \'age\': np.float64(-2693.36...), \'sqft\': np.float64(228.61...)}',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Metric	What it measures	Use when
MAE	Average absolute error, same units as `y`	You want a robust, easy-to-read "off by £X on average"
MSE	Average squared error	Penalises big misses more. The value the model minimises
RMSE	√MSE, back in original units	Reporting. The most common metric in industry
R²	Fraction of variance explained, in [-∞, 1]	Comparing to a "predict the mean" baseline

R² > 0 means you beat the mean baseline, and R² = 1 is perfect. R² can go negative if your model is worse than predicting the mean. RMSE is always at least as large as MAE. If RMSE is much larger, some of your errors are very large.

6. The Textbook Assumptions (and Reality)

Statistics textbooks say that linear regression assumes:

1. Linearity: the true relationship is linear in the inputs.
2. Independence: observations don't influence each other.
3. Homoscedasticity: error variance is constant across x.
4. Normality of residuals: errors are Gaussian.
5. No multicollinearity: features aren't redundant.

In practice, few production teams check all five. We check residual plots for obvious structure (curvature → add polynomial features, funnel shape → log-transform y), check collinearity roughly with correlation matrices or VIF, and rely on cross-validated test-set RMSE to tell us whether the model is useful. The assumptions matter when you need inference (p-values, confidence intervals). For pure prediction, the error on new data is what matters most.

7. Regularisation: Ridge, Lasso, ElasticNet

When you have many features (or few samples), plain linear regression overfits: it learns the noise in the training data. Regularisation adds a penalty on coefficient size, shrinking weights toward zero.

python

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Regularised models NEED scaled features, because the penalty depends on coefficient size
ridge = Pipeline([
    ("scale", StandardScaler()),
    ("model", Ridge(alpha=1.0)),              # L2: shrinks all coefficients smoothly
])

lasso = Pipeline([
    ("scale", StandardScaler()),
    ("model", Lasso(alpha=0.1)),              # L1: pushes some coefficients to ZERO (feature selection)
])

enet = Pipeline([
    ("scale", StandardScaler()),
    ("model", ElasticNet(alpha=0.1, l1_ratio=0.5)),   # mix of L1 and L2
])

Model	Penalty	Effect	When
Ridge	`α · Σ wᵢ²`	Shrinks coefs smoothly, keeps all features	Many correlated features, and you want stability
Lasso	`α · Σ \\|wᵢ\\|`	Drives weak coefs to exactly 0	You want automatic feature selection
ElasticNet	mix of both	Some of each effect	Correlated features + selection wanted

alpha is the regularisation strength: a bigger value means more shrinkage. Tune it via cross-validation (Section 13).

Why scaling is required: the penalty Σ wᵢ² punishes large weights. A feature on a scale of 0–1,000,000 needs only a tiny weight to have an effect, so it is hardly penalised. A feature on a scale of 0–1 needs a huge weight, so it is penalised heavily. Without scaling, regularisation is biased toward whichever features happen to have small numeric ranges.

8. Logistic Regression: Same Idea, Different Output

For binary classification (e.g. spam vs ham), compute the same linear combination and pass it through the sigmoid function:

python

z = w₁·x₁ + ... + wₙ·xₙ + b
p = 1 / (1 + e⁻ᶻ)              # between 0 and 1, read as P(class = 1)

Predict class 1 if p ≥ 0.5, else class 0. The 0.5 boundary is only a convention (more on threshold tuning below).

python

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)

print(clf.predict(X_test[:5]))               # [0 1 0 1 0]
print(clf.predict_proba(X_test[:5]))         # probabilities per class
print(clf.score(X_test, y_test))             # accuracy

Coefficients of logistic regression are log-odds: a one-unit increase in xᵢ multiplies the odds of class 1 by exp(wᵢ). This is useful for inference, but people rarely look at it in pure prediction work.

9. Threshold Tuning

The default 0.5 threshold is fine when classes are balanced and false positives and false negatives cost the same. In real problems, they almost never cost the same.

python

import numpy as np
probs = clf.predict_proba(X_test)[:, 1]      # P(class = 1)

# Aggressive: flag anything above 30% probability (high recall, more false positives)
y_pred_aggressive = (probs > 0.30).astype(int)

# Conservative: flag only above 70% probability (high precision, more false negatives)
y_pred_conservative = (probs > 0.70).astype(int)

+ setup added so this can run · re-runs 2 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_test = _AutoMock('X_test')
clf = _AutoMock('clf')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
except Exception:
    pass
_pm_earlier = [
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Cancer screening tolerates false positives (re-test), but fraud blocking tolerates false negatives (manual review queue). Pick the threshold by cost, not by habit.

10. Classification Metrics: Why Accuracy Misleads

python

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, confusion_matrix, classification_report
)

y_pred = clf.predict(X_test)
print(confusion_matrix(y_test, y_pred))
# [[TN FP]
#  [FN TP]]

print(classification_report(y_test, y_pred))
#               precision    recall  f1-score   support
#            0       0.79      0.84      0.82        89
#            1       0.87      0.82      0.84       111
#     accuracy                           0.83       200

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_test = _AutoMock('X_test')
clf = _AutoMock('clf')
y_test = _AutoMock('y_test')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
except Exception:
    pass
_pm_earlier = [
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
    'import numpy as np\nprobs = clf.predict_proba(X_test)[:, 1]      # P(class = 1)\n\n# Aggressive: flag anything above 30% probability (high recall, more false positives)\ny_pred_aggressive = (probs > 0.30).astype(int)\n\n# Conservative: flag only above 70% probability (high precision, more false negatives)\ny_pred_conservative = (probs > 0.70).astype(int)',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Metric	Formula	Use when
Accuracy	`(TP+TN) / total`	Balanced classes, equal costs
Precision	`TP / (TP+FP)`	False positives are expensive (spam filter: don't flag real mail)
Recall	`TP / (TP+FN)`	False negatives are expensive (cancer: don't miss a case)
F1	harmonic mean of P and R	Balance both. Default for imbalanced
ROC-AUC	area under ROC curve	Ranking quality across all thresholds

Accuracy misleads on imbalanced data. If 99% of transactions are legitimate, a model that predicts "legitimate" for everything scores 99% accuracy and catches zero fraud. Always check the confusion matrix. For imbalanced problems, optimise F1 or AUC (or precision/recall at a chosen threshold tied to business cost).

11. ROC-AUC in One Snippet

python

from sklearn.metrics import roc_auc_score, roc_curve

probs = clf.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, probs)
print(auc)                                    # 0.91  closer to 1 is better, and 0.5 is no better than random
fpr, tpr, thresholds = roc_curve(y_test, probs)

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

y_test = _AutoMock('y_test')
X_test = _AutoMock('X_test')
clf = _AutoMock('clf')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
    from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, confusion_matrix, classification_report
except Exception:
    pass
_pm_earlier = [
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
    'import numpy as np\nprobs = clf.predict_proba(X_test)[:, 1]      # P(class = 1)\n\n# Aggressive: flag anything above 30% probability (high recall, more false positives)\ny_pred_aggressive = (probs > 0.30).astype(int)\n\n# Conservative: flag only above 70% probability (high precision, more false negatives)\ny_pred_conservative = (probs > 0.70).astype(int)',
    'from sklearn.metrics import (\n    accuracy_score, precision_score, recall_score, f1_score,\n    roc_auc_score, confusion_matrix, classification_report\n)\n\ny_pred = clf.predict(X_test)\nprint(confusion_matrix(y_test, y_pred))\n# [[TN FP]\n#  [FN TP]]\n\nprint(classification_report(y_test, y_pred))\n#               precision    recall  f1-score   support\n#            0       0.79      0.84      0.82        89\n#            1       0.87      0.82      0.84       111\n#     accuracy                           0.83       200',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

AUC = 0.5 is no better than a coin flip, and 1.0 is perfect. It measures the probability that a random positive is ranked higher than a random negative. It doesn't depend on a threshold, so it is the right summary when you haven't picked a threshold yet.

12. Cross-Validation: One Split Isn't Enough

A single train/test split is one sample of model performance. The results can vary a lot between splits: your "85% accuracy" might be anything from 78% to 92%, depending on the seed. k-fold cross-validation runs k train/test splits across the dataset and averages the scores.

python

from sklearn.model_selection import cross_val_score

scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5, scoring="f1")
print(scores)                                 # [0.87 0.85 0.85 0.83 0.88]
print(scores.mean(), scores.std())            # 0.855 ± 0.016

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')
def LogisticRegression(*_a, **_kw):
    print('-> LogisticRegression() called')
    return _AutoMock('LogisticRegression()')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
except Exception:
    pass
_pm_earlier = [
    '# Local: pip install scikit-learn numpy\nimport numpy as np\nfrom sklearn.linear_model import LinearRegression\n\nX = np.array([[1], [2], [3], [4], [5]])      # one feature, five samples\ny = np.array([2.1, 4.0, 6.1, 7.9, 10.2])      # roughly y = 2x\n\nmodel = LinearRegression()\nmodel.fit(X, y)\n\nprint(model.coef_)                            # [2.01]   slope is about 2\nprint(model.intercept_)                       # 0.03      intercept is about 0\nprint(model.predict([[6]]))                   # [12.09]   a prediction outside the training range\nprint(model.score(X, y))                      # 0.9987    the R² score',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\n\n# rows are samples, columns are [rooms, age_years, sqft]\nX = np.array([[3, 10, 1500],\n              [4, 25, 2200],\n              [2, 40,  900],\n              [5,  5, 2800],\n              [3, 15, 1700]])\ny = np.array([420_000, 530_000, 220_000, 740_000, 460_000])\n\nmodel = LinearRegression().fit(X, y)\nprint(dict(zip(["rooms", "age", "sqft"], model.coef_)))\n# {\'rooms\': np.float64(-3671.87...), \'age\': np.float64(-2693.36...), \'sqft\': np.float64(228.61...)}',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

cv=5 is the default and a good balance. Use StratifiedKFold (sklearn's default for classifiers) so each fold preserves the class ratio. For time-series, TimeSeriesSplit. Report mean ± std, because a single number hides the variation.

13. Hyperparameter Tuning with `GridSearchCV`

python

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipe = Pipeline([("scale", StandardScaler()), ("model", Ridge())])

param_grid = {"model__alpha": [0.001, 0.01, 0.1, 1, 10, 100]}

search = GridSearchCV(pipe, param_grid, cv=5, scoring="neg_root_mean_squared_error")
search.fit(X_train, y_train)

print(search.best_params_)                    # {'model__alpha': 10}
print(-search.best_score_)                    # 0.346 (the RMSE, where the target is the 0/1 label from section 9)
print(search.score(X_test, y_test))           # held-out test score

+ setup added so this can run · re-runs 2 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
except Exception:
    pass
_pm_earlier = [
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Key points:

model__alpha: the double underscore selects the alpha parameter on the step named model inside the pipeline.
Scoring is negated (neg_root_mean_squared_error) because GridSearchCV always maximises the score. For losses that you want to minimise, it flips the sign.
Always fit the pipeline inside the CV loop. Scaling before the split leaks test statistics into training.

For large grids, RandomizedSearchCV samples instead of exhaustively trying every combination.

14. Multi-Class Logistic

Three or more classes? Two strategies:

python

from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier

LogisticRegression()                       # multinomial (softmax): one model, probabilities over all K classes
OneVsRestClassifier(LogisticRegression())  # one-vs-rest: K separate binary classifiers

Multinomial (softmax) is usually more accurate and gives better-calibrated probabilities, but one-vs-rest is faster to train and easier to explain. A plain LogisticRegression() already uses multinomial for three or more classes. The old multi_class= argument was deprecated in scikit-learn 1.5 and removed in 1.8, so use OneVsRestClassifier when you want one-vs-rest.

Common Mistakes

1. Not scaling features for regularised regression. Ridge/Lasso/ElasticNet penalties depend on coefficient magnitude. Unscaled features give a biased penalty, and so wrong coefficients. Always put StandardScaler before the model in a pipeline. Plain LinearRegression doesn't care.

2. Accuracy on imbalanced data.

python

y_test = np.array([0] * 99 + [1])            # 99% class 0, 1% class 1
y_pred = np.zeros_like(y_test)                # predict 0 for everything
print(accuracy_score(y_test, y_pred))         # 0.99  looks great, but means nothing
print(recall_score(y_test, y_pred))           # 0.0   it caught none of the positives

+ setup added so this can run · re-runs 7 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

np = _AutoMock('np')
def accuracy_score(*_a, **_kw):
    print('-> accuracy_score() called')
    return _AutoMock('accuracy_score()')
def recall_score(*_a, **_kw):
    print('-> recall_score() called')
    return _AutoMock('recall_score()')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')
model = _AutoMock('model')
clf = _AutoMock('clf')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
    from sklearn.linear_model import LogisticRegression
    from sklearn.datasets import make_classification
    from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, confusion_matrix, classification_report
except Exception:
    pass
_pm_earlier = [
    '# Local: pip install scikit-learn numpy\nimport numpy as np\nfrom sklearn.linear_model import LinearRegression\n\nX = np.array([[1], [2], [3], [4], [5]])      # one feature, five samples\ny = np.array([2.1, 4.0, 6.1, 7.9, 10.2])      # roughly y = 2x\n\nmodel = LinearRegression()\nmodel.fit(X, y)\n\nprint(model.coef_)                            # [2.01]   slope is about 2\nprint(model.intercept_)                       # 0.03      intercept is about 0\nprint(model.predict([[6]]))                   # [12.09]   a prediction outside the training range\nprint(model.score(X, y))                      # 0.9987    the R² score',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\n\n# rows are samples, columns are [rooms, age_years, sqft]\nX = np.array([[3, 10, 1500],\n              [4, 25, 2200],\n              [2, 40,  900],\n              [5,  5, 2800],\n              [3, 15, 1700]])\ny = np.array([420_000, 530_000, 220_000, 740_000, 460_000])\n\nmodel = LinearRegression().fit(X, y)\nprint(dict(zip(["rooms", "age", "sqft"], model.coef_)))\n# {\'rooms\': np.float64(-3671.87...), \'age\': np.float64(-2693.36...), \'sqft\': np.float64(228.61...)}',
    'import numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nrng = np.random.default_rng(0)\nrooms = rng.integers(1, 6, size=50)\nage   = rng.integers(0, 50, size=50)\nsqft  = rng.integers(600, 3000, size=50)\nX = np.column_stack([rooms, age, sqft])\ny = 20_000 * rooms - 2_000 * age + 200 * sqft + rng.normal(0, 20_000, size=50)\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)                                             # 40 houses to train on, 10 to test on\n\nmodel = LinearRegression().fit(X_train, y_train)\nprint(model.score(X_test, y_test))            # 0.978 (R² on 10 houses it never saw)',
    'from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\nimport numpy as np\n\ny_pred = model.predict(X_test)\n\nmae  = mean_absolute_error(y_test, y_pred)\nmse  = mean_squared_error(y_test, y_pred)\nrmse = np.sqrt(mse)\nr2   = r2_score(y_test, y_pred)',
    'from sklearn.linear_model import LogisticRegression\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(n_samples=1000, n_features=10, random_state=42)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nclf = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n\nprint(clf.predict(X_test[:5]))               # [0 1 0 1 0]\nprint(clf.predict_proba(X_test[:5]))         # probabilities per class\nprint(clf.score(X_test, y_test))             # accuracy',
    'import numpy as np\nprobs = clf.predict_proba(X_test)[:, 1]      # P(class = 1)\n\n# Aggressive: flag anything above 30% probability (high recall, more false positives)\ny_pred_aggressive = (probs > 0.30).astype(int)\n\n# Conservative: flag only above 70% probability (high precision, more false negatives)\ny_pred_conservative = (probs > 0.70).astype(int)',
    'from sklearn.metrics import (\n    accuracy_score, precision_score, recall_score, f1_score,\n    roc_auc_score, confusion_matrix, classification_report\n)\n\ny_pred = clf.predict(X_test)\nprint(confusion_matrix(y_test, y_pred))\n# [[TN FP]\n#  [FN TP]]\n\nprint(classification_report(y_test, y_pred))\n#               precision    recall  f1-score   support\n#            0       0.79      0.84      0.82        89\n#            1       0.87      0.82      0.84       111\n#     accuracy                           0.83       200',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Use F1, AUC, or recall@precision-floor instead. Set class_weight="balanced" to give the minority class more weight in the loss.

3. Predicting on un-transformed data. If you scaled X_train with StandardScaler, you must apply the same fitted scaler to X_test (and any new prediction input). The easiest fix is to wrap everything in a Pipeline. Then fit-transform on train and transform-only on test happen automatically.

4. Data leakage via pre-split transformation.

python · run on your machine

# WRONG: the scaler sees the test data's statistics
scaler = StandardScaler().fit(X)              # fit on EVERYTHING
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, ...)

The scaler now includes the test set's mean and std, so your reported test score is too optimistic. Always split first, then fit transformers on X_train only. Pipelines with cross-validation do this correctly by default.

5. Over-reading coefficients. Reading raw model.coef_ without (a) standardising features or (b) statistical significance tests is a beginner mistake. To rank feature importance reliably, use permutation importance (covered in trees).

6. max_iter warnings. LogisticRegression defaults to 100 iterations. Large or unscaled problems reach that limit before converging, and sklearn shows a warning. Either scale the data or raise it to max_iter=1000.

🎯 Your Turn: A Full Ridge Pipeline

Build a complete regression workflow on the inline boston_lite data: split, scale, fit Ridge with α tuned via 5-fold CV, predict on the held-out set, report RMSE and R².

Requirements:

1. Use train_test_split with test_size=0.2, random_state=42.
2. Wrap StandardScaler and Ridge in a Pipeline.
3. Use GridSearchCV to tune alpha over [0.01, 0.1, 1, 10, 100] with 5-fold CV.
4. Report best_params_, the test-set RMSE, and the test-set R².

Skeleton:

python

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import mean_squared_error, r2_score

# Small dataset: features [rooms, age, lstat], target = median house value ($1000s)
X = np.array([
    [6.575, 65.2, 4.98], [6.421, 78.9, 9.14], [7.185, 61.1, 4.03],
    [6.998, 45.8, 2.94], [7.147, 54.2, 5.33], [6.430, 58.7, 5.21],
    [6.012, 66.6, 12.43], [6.172, 96.1, 19.15], [5.631, 100.0, 29.93],
    [6.004, 85.9, 17.10], [6.377, 70.3, 20.45], [6.009, 94.3, 13.27],
    [5.889, 39.0, 15.71], [7.249, 21.9, 4.81], [6.625, 17.5, 3.81],
    [5.927, 42.6, 18.66], [6.137, 32.1, 13.44], [5.951, 38.5, 12.13],
    [6.456, 71.9, 11.10], [6.815, 40.5, 5.25],
])
y = np.array([24.0, 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 17.5, 16.5, 18.9,
              15.0, 18.9, 21.7, 35.4, 28.7, 19.6, 23.1, 21.7, 22.2, 30.8])

# TODO 1: split into train/test (test_size=0.2, random_state=42)
# TODO 2: build Pipeline([("scale", StandardScaler()), ("model", Ridge())])
# TODO 3: GridSearchCV over model__alpha with cv=5, scoring="neg_root_mean_squared_error"
# TODO 4: fit on training data, predict on test
# TODO 5: print best_params_, RMSE, R²

Hint 1: Pipeline parameter names

Inside GridSearchCV, parameters belonging to a pipeline step are named stepname__paramname, with two underscores. So Ridge's alpha inside a step called "model" becomes "model__alpha". If you get this wrong, you'll see a confusing ValueError: Invalid parameter.

Hint 2: RMSE from MSE

mean_squared_error returns MSE by default. Either use root_mean_squared_error (sklearn ≥ 1.4, and the old squared=False was removed in 1.6), or take np.sqrt(mean_squared_error(...)). Also remember GridSearchCV's best_score_ is negated, so flip its sign before reporting it.

Show full solution

python

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import mean_squared_error, r2_score

X = np.array([
    [6.575, 65.2, 4.98], [6.421, 78.9, 9.14], [7.185, 61.1, 4.03],
    [6.998, 45.8, 2.94], [7.147, 54.2, 5.33], [6.430, 58.7, 5.21],
    [6.012, 66.6, 12.43], [6.172, 96.1, 19.15], [5.631, 100.0, 29.93],
    [6.004, 85.9, 17.10], [6.377, 70.3, 20.45], [6.009, 94.3, 13.27],
    [5.889, 39.0, 15.71], [7.249, 21.9, 4.81], [6.625, 17.5, 3.81],
    [5.927, 42.6, 18.66], [6.137, 32.1, 13.44], [5.951, 38.5, 12.13],
    [6.456, 71.9, 11.10], [6.815, 40.5, 5.25],
])
y = np.array([24.0, 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 17.5, 16.5, 18.9,
              15.0, 18.9, 21.7, 35.4, 28.7, 19.6, 23.1, 21.7, 22.2, 30.8])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", Ridge()),
])

param_grid = {"model__alpha": [0.01, 0.1, 1.0, 10.0, 100.0]}

search = GridSearchCV(
    pipe, param_grid, cv=5,
    scoring="neg_root_mean_squared_error",
)
search.fit(X_train, y_train)

y_pred = search.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2   = r2_score(y_test, y_pred)

print(f"Best alpha: {search.best_params_['model__alpha']}")
print(f"CV RMSE:    {-search.best_score_:.3f}")
print(f"Test RMSE:  {rmse:.3f}")
print(f"Test R²:    {r2:.3f}")

# Output:
# Best alpha: 1.0
# CV RMSE:    3.121
# Test RMSE:  2.893
# Test R²:    -2.447

That negative R² is a real result, and it is worth understanding: the test set is only 4 houses, priced close together, so "always predict their average" is a very strong baseline and R² punishes even small misses. RMSE (under $3k) is the more useful number here. On tiny test sets, trust RMSE over R².

Why this solution is correct:

Split first: the scaler never sees X_test, so there is no leakage.
Pipeline: the same scaler that fit on X_train is automatically applied at predict time on X_test. No risk of forgetting to transform.
Cross-validated tuning: alpha=1.0 is picked because it gave the best mean RMSE across the 5 folds of the training set, not because of one lucky split.
Final evaluation on the held-out test set: the only trustworthy performance number is the one on data that touched neither the scaler's .fit() nor the alpha search.

Drop in Lasso instead of Ridge and you get automatic feature selection: the coefficients of weak features become exactly zero. It is the same pipeline with a one-word change.

What You Learned

Regression predicts a number, and classification predicts a category. Linear and logistic regression are the standard starting models for each.
Linear regression fits y = w·x + b by minimising squared errors. Coefficients are interpretable only after standardising features.
train_test_split with a fixed random_state gives you a fair, reproducible evaluation on held-out data.
Regression metrics: RMSE for industry reporting, MAE for robustness to outliers, R² to compare against the mean baseline.
Regularisation: Ridge (L2) shrinks smoothly, Lasso (L1) zeroes out weak features, and ElasticNet mixes both. All three require scaled inputs.
Logistic regression = linear combination → sigmoid → probability → threshold. The default of 0.5 is only a convention.
Classification metrics: accuracy misleads on imbalanced data, so use F1, AUC, or precision/recall tied to business cost.
k-fold cross-validation beats a single split for estimating performance and tuning hyperparameters.
GridSearchCV + Pipeline is the safe, leakage-free pattern for hyperparameter tuning.
Multi-class logistic uses one-vs-rest or softmax (multinomial), and sklearn's default is a sensible choice.

Next: Decision Trees & Random Forests: non-linear models, ensembles, and tree models that often beat linear baselines on tabular data.

1. The Two Standard Tasks

2. Linear Regression: Fit the Best Line

3. Multiple Features and Coefficient Interpretation

4. Train/Test Split: Always Required

5. Regression Metrics: When to Use Which

6. The Textbook Assumptions (and Reality)

7. Regularisation: Ridge, Lasso, ElasticNet

8. Logistic Regression: Same Idea, Different Output

9. Threshold Tuning

10. Classification Metrics: Why Accuracy Misleads

11. ROC-AUC in One Snippet

12. Cross-Validation: One Split Isn't Enough

13. Hyperparameter Tuning with GridSearchCV

14. Multi-Class Logistic

Common Mistakes

🎯 Your Turn: A Full Ridge Pipeline

What You Learned

13. Hyperparameter Tuning with `GridSearchCV`