Classification: Predicting Categories
1 · The lesson
readYou're predicting a category. Spam or not-spam. Fraud or legit. Cat, dog, or bird. The target is discrete — a label, not a number.
This is the most common kind of ML problem you'll meet in industry. Almost every "is this X?" question is classification. This lesson covers the workhorse classifier (logistic regression), what a decision boundary actually is, and — most importantly — the four metrics that beat accuracy on imbalanced data.
Run these right here — scikit-learn and NumPy both work in the browser. The first scikit-learn import takes a few seconds while it downloads; after that it's instant. Expected output is also shown in comments below each block.
1. Binary vs Multi-class
Binary — exactly two classes. Spam / not-spam. Click / no-click. Healthy / diseased. By convention, label them 0 and 1, with 1 usually being the "positive" or "interesting" class.
Multi-class — three or more classes. Cat / dog / bird. Iris-setosa / versicolor / virginica. Customer-tier bronze / silver / gold.
scikit-learn handles both with the same API. Most binary classifiers extend to multi-class automatically (one-vs-rest under the hood). Your job is to know which problem you're solving so you can pick the right metric.
2. The Decision Boundary
A classifier learns a decision boundary — a line, curve, or surface that separates one class from the other. Anything on this side is predicted class 0, anything on that side is class 1.
x₂ ▲ ● ● class 0 │ ● ● ● ╲ ← decision boundary │ ● ● ╲ │ ╲ ▲ ▲ │ ╲▲ ▲ ▲ class 1 │ ╲ ▲ ▲ └────────────────► x₁
Different model families draw boundaries of different shapes:
- Logistic regression → a straight line (or flat hyperplane).
- Decision trees → axis-aligned rectangles.
- KNN → wiggly Voronoi cells.
- SVM with RBF kernel → smooth curves.
That's basically the only difference between classifier families: what kind of boundary they can draw.
3. Logistic Regression — the Default First Move
Despite the name, logistic regression is a classifier, not a regressor. (The name is historical and unhelpful.)
It's linear regression's classifier sibling: same idea of summing weighted features, but then squashed through a function called the sigmoid that maps any real number into the range [0, 1]. That output is interpreted as a probability:
linear part squash ↓ ↓ z = w·x + b → P(y=1) = 1 / (1 + e^(-z))
If P(y=1) > 0.5, predict class 1; otherwise class 0. That's it. Fast to train, interpretable coefficients, sensible probability output. Always your first classifier on a new problem.
from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split X, y = make_classification(n_samples=500, n_features=5, n_informative=3, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) print("Predictions: ", model.predict(X_test[:5])) print("Probabilities: ", model.predict_proba(X_test[:5]).round(2)) print("Accuracy: ", model.score(X_test, y_test)) # → Predictions: [1 1 0 0 1] # → Probabilities: [[0.12 0.88] [0.21 0.79] [0.91 0.09] [0.83 0.17] [0.05 0.95]] # → Accuracy: 0.87
predict_proba returns one column per class — [P(class=0), P(class=1)]. The columns sum to 1. Use these when you need calibrated risk scores, not just a verdict.
4. Threshold Tuning — 0.5 Is Just a Default
predict() uses a 0.5 cutoff. Probability > 0.5 → predict 1, else 0. You can change this.
When does it matter?
- Fraud detection. A false negative (missed fraud) is expensive. You'd rather flag suspicious transactions at probability > 0.3 even if half are wrong.
- Spam. A false positive (real email in spam folder) is very expensive. Maybe only flag at > 0.8.
import numpy as np probs = model.predict_proba(X_test)[:, 1] # P(class=1) for every row y_pred_strict = (probs > 0.8).astype(int) y_pred_lenient = (probs > 0.3).astype(int)
setup added so this can run · defines X_test, model
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) X_test = _AutoMock('X_test') model = _AutoMock('model')
You're trading precision against recall — the next section.
5. The Confusion Matrix
Four possible outcomes for any binary prediction:
| Predicted 0 | Predicted 1 | |
|---|---|---|
| Actually 0 | True Negative (TN) | False Positive (FP) |
| Actually 1 | False Negative (FN) | True Positive (TP) |
Memorise the corner names. Every binary metric is built from these four numbers.
from sklearn.metrics import confusion_matrix y_pred = model.predict(X_test) print(confusion_matrix(y_test, y_pred)) # → [[42 6] # → [ 7 45]] # # Reading: 42 TN, 6 FP (top row); 7 FN, 45 TP (bottom row).
setup added so this can run · defines X_test, model, y_test
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) X_test = _AutoMock('X_test') model = _AutoMock('model') y_test = _AutoMock('y_test')
6. The Four Metrics That Beat Accuracy
Accuracy — the basic one
accuracy = (TP + TN) / total
setup added so this can run · defines total, TP, TN
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) total = _AutoMock('total') TP = _AutoMock('TP') TN = _AutoMock('TN')
"What fraction did we get right?" Reasonable when classes are balanced. Useless when they aren't (see Section 7).
Precision — how clean are my positive predictions?
precision = TP / (TP + FP)
setup added so this can run · defines TP, FP
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) TP = _AutoMock('TP') FP = _AutoMock('FP')
"Of everything I flagged as positive, how many really were?" Optimise for precision when false positives are costly — spam filtering, sending alerts.
Recall — how complete is my positive coverage?
recall = TP / (TP + FN)
setup added so this can run · defines TP, FN
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) TP = _AutoMock('TP') FN = _AutoMock('FN')
"Of all the actual positives, how many did I catch?" Optimise for recall when false negatives are costly — fraud detection, cancer screening, security.
F1 — the harmonic balance
F1 = 2 · (precision · recall) / (precision + recall)A single number that punishes you for being good at one and bad at the other. Default metric for imbalanced classification.
ROC-AUC — threshold-independent
How well does the model rank positives above negatives, regardless of the 0.5 threshold? AUC of 1.0 = perfect ranker, 0.5 = random. Useful when you'll be tuning the threshold later anyway.
from sklearn.metrics import classification_report, roc_auc_score print(classification_report(y_test, y_pred)) print("ROC-AUC:", roc_auc_score(y_test, probs)) # → precision recall f1-score support # → 0 0.86 0.88 0.87 48 # → 1 0.88 0.87 0.88 52 # → accuracy 0.87 100 # → ROC-AUC: 0.93
setup added so this can run · defines y_test, y_pred, probs
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) y_test = _AutoMock('y_test') y_pred = _AutoMock('y_pred') probs = _AutoMock('probs')
classification_report is your one-shot summary. Print it every time.
7. The Imbalance Trap
99% of credit-card transactions are legit. A model that predicts "legit" for every single transaction gets 99% accuracy. It also catches zero fraud and is therefore worthless.
import numpy as np y_true = np.array([0] * 990 + [1] * 10) # 99-1 imbalance y_pred = np.zeros(1000, dtype=int) # always predict 0 from sklearn.metrics import accuracy_score, f1_score, recall_score print("Accuracy:", accuracy_score(y_true, y_pred)) # 0.99 — looks great print("F1: ", f1_score(y_true, y_pred)) # 0.00 — disaster print("Recall: ", recall_score(y_true, y_pred)) # 0.00 — caught nothing # → Accuracy: 0.99 # → F1: 0.0 # → Recall: 0.0
Rule: on imbalanced data, accuracy lies. Report F1, recall on the minority class, and ROC-AUC. Never accuracy alone.
Common imbalanced problems: fraud, churn, rare disease, conversion, defect detection. Basically anything interesting.
8. Other Classifiers — Brief Tour
You'll meet logistic regression first, but here's the rest of the toolkit and when each shines.
| Model | Boundary shape | Strength | Weakness |
|---|---|---|---|
| Logistic Regression | Straight line | Fast, interpretable, calibrated probabilities | Can't bend |
| SVM (RBF kernel) | Smooth curves | Great on small/medium data with clean boundaries | Slow on big data, needs scaling |
| KNN | Voronoi cells | Zero training, intuitive | Slow at prediction time, scales badly |
| Naive Bayes | Probabilistic | Lightning fast, great baseline for text | Strong independence assumption |
| Decision Tree | Axis-aligned rectangles | Interpretable, handles mixed types | High variance — see trees |
| Random Forest | Many trees averaged | Strong default for tabular data | Less interpretable, slower |
| Gradient Boosting | Many trees, sequential | Often best on tabular data | More hyperparameters to tune |
Default progression for a new problem: logistic regression → random forest → gradient boosting. Stop as soon as one is good enough.
Common Mistakes
- Trusting accuracy on imbalanced data. Repeat after me: 99% accuracy on a 99-1 split is the do-nothing baseline. Use F1 or AUC.
- Forgetting
predict_probawhen you need calibrated probabilities. Random forests and SVMs return probabilities, but they're often miscalibrated — a "0.9" doesn't mean 90% likely. UseCalibratedClassifierCVif probabilities need to mean something. - Inconsistent
random_state. Your colleague gets a different score. Pin the seed ontrain_test_split, the model, and any shuffle step.random_state=42everywhere is the lazy convention. - Comparing models on different train/test splits. Use the same split (or, better, the same
cross_val_score) for every model you benchmark. - Training on a feature that's a leak from the future. If "customer_signed_up_yesterday" is a feature for predicting "customer_will_sign_up", you're cheating. Audit features for leakage before reporting.
🎯 Your Turn — Fit and Report
Inline data:
from sklearn.datasets import make_classification X, y = make_classification(n_samples=400, n_features=8, n_informative=5, weights=[0.7, 0.3], random_state=42)
The dataset is mildly imbalanced (70-30). Your task:
1. Split 80/20, random_state=42.
2. Fit a LogisticRegression.
3. Print classification_report(y_test, y_pred) and confusion_matrix(y_test, y_pred).
4. Return the test set's macro-averaged F1, rounded to 3 decimals.
from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import classification_report, confusion_matrix, f1_score def fit_and_report(): X, y = make_classification(n_samples=400, n_features=8, n_informative=5, weights=[0.7, 0.3], random_state=42) # TODO 1: split 80/20 with random_state=42 # TODO 2: fit LogisticRegression(max_iter=1000) # TODO 3: predict on test set # TODO 4: print classification_report and confusion_matrix # TODO 5: return round(f1_score(y_test, y_pred, average="macro"), 3) pass print(fit_and_report())
Hint 1 — macro vs binary F1
f1_score(y_test, y_pred, average="macro") averages the F1 across both classes equally, which is what you want when classes are imbalanced and both matter. average="binary" only looks at class 1.
Hint 2 — get y_pred once, use it twice
Cally_pred = model.predict(X_test) once, then pass y_pred to both classification_report and confusion_matrix. Cleaner than calling predict multiple times.
Show full solution
from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import classification_report, confusion_matrix, f1_score def fit_and_report(): X, y = make_classification(n_samples=400, n_features=8, n_informative=5, weights=[0.7, 0.3], random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) y_pred = model.predict(X_test) print(classification_report(y_test, y_pred)) print(confusion_matrix(y_test, y_pred)) return round(f1_score(y_test, y_pred, average="macro"), 3) print(fit_and_report()) # → precision recall f1-score support # → 0 0.89 0.91 0.90 56 # → 1 0.78 0.75 0.77 24 # → accuracy 0.86 80 # → ... # → 0.832
Notice the report shows the minority class (1) with lower F1 than the majority. That's normal — and a starting point for tuning. Try LogisticRegression(class_weight="balanced") and watch the minority recall jump.
What You Learned
- Classification predicts a category; regression predicts a number.
- Logistic regression: a linear model squashed into a probability. Fast, interpretable, default first move.
- The 0.5 prediction threshold is a default — change it to trade precision against recall.
- The confusion matrix is the source of every binary metric: TP, FP, TN, FN.
- Accuracy lies on imbalanced data. Use precision, recall, F1, ROC-AUC instead.
classification_reportgives you all of them. - Logistic regression → random forest → gradient boosting is the default escalation ladder.
Next: Decision Trees — when relationships aren't linear, and you want to see exactly how the model decides.
Practice this
on practicepython.inShort exercises that run in your browser and tell you what your code actually did, not just whether a test passed.