Your Data, Your Model — Machine Learning Fundamentals

Every supervised ML problem comes down to one rule: given X, predict y.

X is what the model sees. y is what you want it to predict. If you get this shape right, most of scikit-learn works without problems. If you get it wrong, you can spend an afternoon trying to understand a "shape mismatch" error.

This lesson takes you from "I have a CSV" to "I have a model that predicts something". It includes the smallest complete ML script.

Run these right here: scikit-learn, pandas and matplotlib all work in the browser. The first scikit-learn import takes a few seconds while it downloads. After that it's instant. Expected output is also shown in comments below each block.

1. Features and Target

The convention that almost every ML library follows:

python

X = features  (the inputs, plural)
y = target    (the thing you predict, singular)

Imagine predicting house prices:

sqft	bedrooms	age	price
1200	2	12	320,000
1800	3	5	480,000
950	1	30	210,000

X is the first three columns: sqft, bedrooms, age. Shape (3, 3): three rows, three features.
y is the last column: price. Shape (3,): one value per row.

In NumPy / pandas terms:

python

import pandas as pd

df = pd.DataFrame({
    "sqft":     [1200, 1800, 950],
    "bedrooms": [2,    3,    1],
    "age":      [12,   5,    30],
    "price":    [320_000, 480_000, 210_000],
})

X = df[["sqft", "bedrooms", "age"]]
y = df["price"]

print(X.shape, y.shape)
# → (3, 3) (3,)

The capital X and lowercase y come from maths notation, where X is a matrix and y is a vector. Almost every example you read uses this convention. Use it too.

2. The Train/Test Split

A model that only "predicts" what it has already seen is useless: that's just a lookup table. The goal is generalisation: doing well on data it has never seen.

So before training, you set aside part of the data and hide it. You train on the rest, and score the model on the hidden part. That score is a fair estimate of how the model will perform in real use.

python

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X = _AutoMock('X')
y = _AutoMock('y')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "sqft":     [1200, 1800, 950],\n    "bedrooms": [2,    3,    1],\n    "age":      [12,   5,    30],\n    "price":    [320_000, 480_000, 210_000],\n})\n\nX = df[["sqft", "bedrooms", "age"]]\ny = df["price"]\n\nprint(X.shape, y.shape)\n# → (3, 3) (3,)',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

test_size=0.2: keep 20% for testing.
random_state=42: fix the shuffle so you get the same split every time. Use any integer. Pick one and stick with it across the project.

After this line you have four arrays:

python

X_train, y_train  →  model sees these during .fit()
X_test,  y_test   →  model never sees these until evaluation

If you train on test data, your score is meaningless. This is the most serious mistake in ML. It is like marking your own exam when you have already seen the answers.

3. The sklearn API in Three Methods

Every scikit-learn model that makes predictions (and there are hundreds) has the same three methods:

python

model.fit(X_train, y_train)      # learn from data
model.predict(X_test)            # make predictions on new data
model.score(X_test, y_test)      # a single number that measures performance

+ setup added so this can run · defines X_train, y_train, X_test, y_test, model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')
model = _AutoMock('model')

That's the whole API. Once you've written this code for one model, you can use the same code for the others. Swap LinearRegression() for RandomForestRegressor() and the surrounding code is identical.

.score() returns a sensible default metric:

For classifiers, it's accuracy.

For regressors, it's R² (the share of the variance explained: 1.0 is perfect, 0.0 means "no better than always predicting the mean").

You'll often want better metrics than .score() (lesson 4 covers them), but it is fine for a first check.

4. The Smallest Complete ML Script

It takes about ten lines. Learn this structure well.

python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
print(f"Test accuracy: {model.score(X_test, y_test):.2f}")
# → Test accuracy: 1.00

Look at the structure:

1. Load the data.
2. Split it.
3. Instantiate the model.
4. Fit on the training data.
5. Score on the test data.

This structure works for most classical ML problems. You'll change the model, the data and the metric, but the structure stays the same.

5. Underfit, Overfit, and Just Right

When a model performs badly, it's failing in one of two ways.

Underfitting: too simple

The model is too simple to fit even the training data. Training error is high, and test error is high. Imagine drawing a straight line through a clearly curved cloud of points.

Overfitting: too complex

The model memorises the training data, including the noise. Training error is tiny, but test error is large. Imagine drawing a wavy line that passes through every single training point. It fits the past data perfectly, but it is useless for new data.

A classic example: fitting a noisy sine wave with polynomials of different degrees.

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
X = np.linspace(0, 2 * np.pi, 30).reshape(-1, 1)
y = np.sin(X).ravel() + rng.normal(0, 0.2, size=30)

for degree, label in [(1, "underfit"), (4, "just right"), (15, "overfit")]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    plt.scatter(X, y, alpha=0.5)
    grid = np.linspace(0, 2 * np.pi, 200).reshape(-1, 1)
    plt.plot(grid, model.predict(grid), label=f"degree {degree}: {label}")

plt.legend(); plt.title("Underfit / Just right / Overfit"); plt.show()

You'd see:

Degree 1: a straight line through a curve. Underfit.

Degree 4: follows the sine smoothly. Just right.

Degree 15: bends up and down to pass close to every point. Overfit.

The right complexity is the one that gives the lowest error on data the model hasn't seen. Section 6 shows how to measure this correctly, with a validation set, so that the test set stays untouched.

6. Validation Set vs Test Set

For real projects, two splits aren't enough. You want three roles:

Train: fit the model.
Validation: tune hyperparameters (the settings you choose for the model).
Test: the final score, on data you haven't touched. Only look at this once.

If you tune settings using the test set, the test set becomes part of your training process, and its score is no longer fair. This is called leakage.

The simple way is to call train_test_split twice:

python

X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42)
# Now: 60% train, 20% validation, 20% test

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def train_test_split(*_a, **_kw):
    print('-> train_test_split() called')
    return _AutoMock('train_test_split()')
X = _AutoMock('X')
y = _AutoMock('y')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    from sklearn.model_selection import train_test_split
    from sklearn.datasets import load_iris
    from sklearn.linear_model import LogisticRegression
    import numpy as np
    import matplotlib.pyplot as plt
    from sklearn.linear_model import LinearRegression
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.pipeline import make_pipeline
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "sqft":     [1200, 1800, 950],\n    "bedrooms": [2,    3,    1],\n    "age":      [12,   5,    30],\n    "price":    [320_000, 480_000, 210_000],\n})\n\nX = df[["sqft", "bedrooms", "age"]]\ny = df["price"]\n\nprint(X.shape, y.shape)\n# → (3, 3) (3,)',
    'from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)',
    'from sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\n\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\n\nmodel = LogisticRegression(max_iter=1000)\nmodel.fit(X_train, y_train)\nprint(f"Test accuracy: {model.score(X_test, y_test):.2f}")\n# → Test accuracy: 1.00',
    'import numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.pipeline import make_pipeline\n\nrng = np.random.default_rng(42)\nX = np.linspace(0, 2 * np.pi, 30).reshape(-1, 1)\ny = np.sin(X).ravel() + rng.normal(0, 0.2, size=30)\n\nfor degree, label in [(1, "underfit"), (4, "just right"), (15, "overfit")]:\n    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())\n    model.fit(X, y)\n    plt.scatter(X, y, alpha=0.5)\n    grid = np.linspace(0, 2 * np.pi, 200).reshape(-1, 1)\n    plt.plot(grid, model.predict(grid), label=f"degree {degree}: {label}")\n\nplt.legend(); plt.title("Underfit / Just right / Overfit"); plt.show()',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Cross-validation: when you have little data

If you only have 200 rows, setting aside 40% for validation and test leaves little for training. K-fold cross-validation lets every row take a turn in the validation set.

python

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split

# back to the iris model from section 4 (section 5 reused X, y and model for the sine curves)
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LogisticRegression(max_iter=1000)

scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"CV mean: {scores.mean():.3f}, std: {scores.std():.3f}")
# → CV mean: 0.967, std: 0.049

cv=5: split the training data into 5 parts (folds).
Train on 4, score on 1. Repeat 5 times so every fold gets a turn.
Return 5 scores. The mean is your estimate of performance, and the standard deviation (std) tells you how much it varies.

Cross-validation is the usual choice for small and medium-sized datasets. We use it heavily in pipelines.

Common Mistakes

Training on the test set. Sometimes accidental: you call train_test_split after fitting a scaler on all the data. The test set was already "seen" through the scaler's statistics. Lesson 6 fixes this with Pipeline.
Predicting on un-preprocessed data. Whatever you did to X_train (scaling, encoding, imputing) must happen to X_test with the train-fitted transformer. Forgetting this is a very common bug in production ML systems.
Reporting the training score. "My model is 99% accurate!" On which data? If it's training accuracy, it tells us nothing about generalisation.
Not setting random_state. Your colleague runs your notebook again, gets a different split and a different number, and reports a bug. Fix the seed.

🎯 Your Turn: Train Your First Classifier

Fill in the gaps to:

1. Generate a synthetic 2-class dataset.
2. Split it 80/20.
3. Fit any sklearn classifier of your choice.
4. Return (train_accuracy, test_accuracy) as a tuple, both rounded to 3 decimals.

python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression  # or any other

def train_and_score():
    X, y = make_classification(n_samples=500, n_features=10,
                               n_informative=5, random_state=42)

    # TODO 1: split X, y into train/test (80/20), random_state=42
    # TODO 2: instantiate a classifier
    # TODO 3: fit on training data
    # TODO 4: compute train_acc and test_acc using model.score(...)
    # TODO 5: return (round(train_acc, 3), round(test_acc, 3))

    pass

print(train_and_score())
# → expected something like (0.853, 0.82)

Hint 1: the split

train_test_split(X, y, test_size=0.2, random_state=42) returns four arrays in the order X_train, X_test, y_train, y_test.

Hint 2: two .score() calls

model.score(X_train, y_train) gives training accuracy. model.score(X_test, y_test) gives test accuracy. Both return a float between 0 and 1.

Show full solution

python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

def train_and_score():
    X, y = make_classification(n_samples=500, n_features=10,
                               n_informative=5, random_state=42)

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )

    model = LogisticRegression(max_iter=1000)
    model.fit(X_train, y_train)

    train_acc = model.score(X_train, y_train)
    test_acc  = model.score(X_test,  y_test)

    return (round(train_acc, 3), round(test_acc, 3))

print(train_and_score())
# → (0.853, 0.82)

It is normal for training accuracy to be slightly higher than test accuracy. If they were identical, your model might be too simple. If training was 0.99 and test was 0.65, the model would be overfitting. A small gap, like the one here, is what you want.

What You Learned

The rule: X (features) and y (target). Capital X, lowercase y.
train_test_split sets aside a part of the data that you never use during training.
Every sklearn model that predicts has .fit(), .predict(), .score(). The code stays the same when you change the model.
The smallest complete ML script: load → split → create the model → fit → score. Five steps.
Underfit = too simple. Overfit = memorised noise. Just right = best score on unseen data.
Use a validation set (or cross-validation) for tuning, keep the test set untouched until the end.

Next: Linear Regression: predicting numbers with the simplest model there is.