Training & Improving Your Network — Deep Learning Fundamentals

The first time you train a neural network it almost certainly doesn't work. The loss is either too high, the validation accuracy diverges from training accuracy, or both numbers sit stubbornly flat. This isn't failure — it's the normal starting point. The skill that separates someone who's "done a deep learning tutorial" from someone who actually ships models is diagnosing the training curve and knowing which lever to pull.

Run code in a Colab notebook or locally with pip install tensorflow. Expected outputs shown inline.

1. The One Plot That Tells You Everything

python

import matplotlib.pyplot as plt

history = model.fit(X_train, y_train,
                    epochs=50,
                    validation_split=0.2,
                    verbose=0)

plt.plot(history.history['loss'],     label='train loss')
plt.plot(history.history['val_loss'], label='val loss')
plt.xlabel('epoch'); plt.ylabel('loss'); plt.legend(); plt.show()

+ setup added so this can run · defines X_train, y_train, model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')

That plot is your single most important diagnostic. Before you tune anything, look at the two curves.

2. The Four Curves and What They Mean

Pattern	Diagnosis	First thing to try
Both losses high, flat	Underfit — model too weak	Bigger network, more epochs, lower learning rate
Both drop, settle near each other	Good fit — ship it	Tune for performance, not error
Train drops, val rises	Overfit — memorising training data	Dropout, L2, early stopping, more data
Both noisy, no trend	Just noise — broken setup	Check learning rate, data normalisation, label correctness

A picture for each:

python

underfit            good fit            overfit             noise
loss                loss                loss                loss
 |\                  |\                  |\__              |\/\/\
 | \________train    | \                 |   \___train     |\/\/\
 |  \________val     |  \_____train      |       \__       |\/\/\
 |                   |   \____val        |       ___val    |
 +------ epoch       +------ epoch       |      /          +------
                                         |    _/
                                         +------ epoch

The trick is to identify which of these you have, then pull the matching lever.

3. Fighting Underfit

If train and val loss are both stuck high:

Make the network bigger — more neurons per layer or more layers. An underfit model literally cannot represent the function you're asking it to learn.
Train longer — increase epochs. Sometimes 50 isn't enough.
Lower the learning rate — a too-high rate keeps bouncing past the minimum.
Better features — neural networks aren't magic. Garbage features → garbage learning. See feature engineering.

4. Fighting Overfit — Four Tools

Overfit means the model has memorised the training set instead of learning the underlying pattern. Train accuracy is great, val accuracy is poor. Four tools, roughly in the order you should reach for them.

Tool 1: Dropout

Randomly zero out a fraction of neurons during training. The network can't rely on any single neuron, so it has to spread the learned representation across many — which generalises better.

python

from tensorflow.keras.layers import Dense, Dropout

model = Sequential([
    Dense(128, activation='relu', input_shape=(20,)),
    Dropout(0.3),                       # drop 30% of activations each step
    Dense(64,  activation='relu'),
    Dropout(0.3),
    Dense(1,   activation='sigmoid'),
])

+ setup added so this can run · defines Sequential

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')

0.2 – 0.5 is the typical range. Higher rates regularise harder but slow learning. Dropout is on during training, off during evaluation — Keras handles that automatically.

Tool 2: L2 Weight Regularisation

Penalise large weights by adding a term to the loss proportional to the sum of squared weights. Forces the model to use small weights, which produces smoother decision boundaries.

python

from tensorflow.keras.regularizers import l2

Dense(64, activation='relu', kernel_regularizer=l2(0.01))

+ setup added so this can run · defines Dense

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Dense(*_a, **_kw):
    print('-> Dense() called')
    return _AutoMock('Dense()')

0.001 – 0.01 is the usual range. Bigger numbers regularise harder.

Tool 3: Early Stopping

Stop training when validation loss stops improving — before overfit really sets in. This is the single highest-leverage callback in Keras.

python

from tensorflow.keras.callbacks import EarlyStopping

es = EarlyStopping(monitor='val_loss',
                   patience=5,
                   restore_best_weights=True)

model.fit(X_train, y_train,
          epochs=200,                  # set high — early stopping decides when to quit
          validation_split=0.2,
          callbacks=[es])
# → Epoch 27: val_loss did not improve from 0.184 — stopping. Restoring weights from epoch 22.

+ setup added so this can run · defines X_train, y_train, model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')

patience=5 means "give it 5 more epochs to recover before quitting". restore_best_weights=True rolls the model back to its best validation epoch — without this you keep the final (possibly worse) weights.

Tool 4: Batch Normalisation

Normalise activations between layers. Acts as a gentle regulariser and lets you use higher learning rates.

python

from tensorflow.keras.layers import BatchNormalization

model = Sequential([
    Dense(128, input_shape=(20,)),
    BatchNormalization(),
    keras.layers.Activation('relu'),
    Dense(64),
    BatchNormalization(),
    keras.layers.Activation('relu'),
    Dense(1, activation='sigmoid'),
])

+ setup added so this can run · defines Sequential, Dense, keras

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
def Dense(*_a, **_kw):
    print('-> Dense() called')
    return _AutoMock('Dense()')
keras = _AutoMock('keras')

Convention places BN between the Dense and its activation. In practice you can also place it after — both work.

5. The Other Lever — More Data

No amount of regularisation beats more data. If your model overfits and you can collect/generate more training samples, that's almost always the best move. For images, data augmentation (rotations, flips, crops) effectively multiplies your dataset for free — covered in CNNs.

6. Optimisers — Pick Adam and Move On

Optimiser	When
Adam	Default. Use this unless you have a reason not to.
SGD with momentum	Image classification at very large scale (some research papers)
AdamW	Modern variant of Adam with decoupled weight decay — common in transformer training
RMSprop	Older default for RNNs

You'll spend 95% of your career on Adam. The Keras default learning rate is 0.001, which works for most problems.

python

from tensorflow.keras.optimizers import Adam, AdamW, SGD

model.compile(optimizer=Adam(learning_rate=1e-3), loss='binary_crossentropy')

+ setup added so this can run · defines model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')

7. Learning Rate — The Single Most Important Knob

If you only tune one hyperparameter, tune learning rate.

Too high — loss diverges, goes to NaN, bounces wildly. You overshoot the minimum each step.
Too low — loss drops glacially, validation never reaches its potential.
Just right — smooth, steady decrease, eventual plateau.

Typical values: 1e-4 to 1e-2. Default 1e-3 is a good first try.

ReduceLROnPlateau — Adaptive Rate

Halve the learning rate whenever validation loss stops improving. Lets you start aggressive and refine later.

python

from tensorflow.keras.callbacks import ReduceLROnPlateau

rlrop = ReduceLROnPlateau(monitor='val_loss',
                          factor=0.5,
                          patience=3,
                          min_lr=1e-6)

model.fit(X_train, y_train,
          epochs=100,
          callbacks=[es, rlrop])
# → Epoch 15: val_loss plateau — reducing learning rate to 0.0005.

+ setup added so this can run · defines X_train, y_train, model, es

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')
es = _AutoMock('es')

Pair it with early stopping. Free performance.

8. Batch Size — The Other Knob

Batch size	Effect
Small (8–32)	Noisier gradients, often better generalisation, slower wall-clock
Medium (64–128)	The usual sweet spot
Large (256+)	Smoother gradients, fastest training on GPU, can hurt generalisation

If you're starting fresh, use 32. Increase if training is too slow and the GPU has spare memory. Decrease if you're seeing erratic loss curves and have a small dataset.

9. Data Augmentation — A Preview

For images, you can manufacture more data by applying random transformations: flip, rotate, crop, zoom, colour-jitter. Each epoch sees slightly different inputs, which forces the model to learn invariances rather than memorise.

python

from tensorflow.keras import layers

augment = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),
    layers.RandomZoom(0.1),
])

+ setup added so this can run · defines keras

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

keras = _AutoMock('keras')

Full treatment in CNNs.

10. Experiment Tracking

Once you're running more than a few models, you need to track them. Two options:

TensorBoard — built into TensorFlow. Add keras.callbacks.TensorBoard(log_dir='logs/') to your callbacks and run tensorboard --logdir logs/. Free, local, plots loss curves and weight histograms.
Weights & Biases (wandb) — hosted, gorgeous UI, free for individuals. One-line setup, tracks code/hyperparams/metrics across hundreds of runs.

For your first ten models, plotting history.history is fine. After that, install one of the above. You will not remember what hyperparameters gave you that 92% number last Tuesday.

Common Mistakes

Train accuracy 99%, val accuracy 60%. Classic overfit. Add Dropout(0.3), add EarlyStopping, possibly shrink the network. Don't keep training.
Both stuck at chance. Either underfit (network too small or learning rate too low) or broken setup. Check that your labels actually pair with the right inputs, that you normalised inputs to a sensible range, and that the loss matches the task.
Loss goes to NaN. Learning rate too high. Drop it 10x. Also check for log(0) issues if you wrote a custom loss.
Not setting a seed. Different runs give different results. Acceptable in production, infuriating during debugging. tf.keras.utils.set_random_seed(42) covers Python, NumPy, and TF in one call.
Tweaking on the test set. If you're picking hyperparameters by looking at test-set numbers, your test set has become a validation set and you have no real holdout left. Keep test untouched until the end.

🎯 Your Turn — Tame an Overfitting Model

You're given this overfit-prone model trained on a small tabular dataset. Train accuracy hits 99%, validation accuracy stalls at 75%. Add Dropout(0.3) after each hidden layer and add an EarlyStopping callback with patience=5 and restore_best_weights=True. Retrain.

Skeleton:

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential([
    Dense(256, activation='relu', input_shape=(20,)),
    # TODO 1: dropout
    Dense(128, activation='relu'),
    # TODO 2: dropout
    Dense(64,  activation='relu'),
    # TODO 3: dropout
    Dense(1,   activation='sigmoid'),
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# TODO 4: early stopping callback

history = model.fit(X_train, y_train,
                    epochs=100,
                    validation_split=0.2,
                    callbacks=[ ... ])     # TODO 5

+ setup added so this can run · defines X_train, y_train

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')

Hint 1 — Where Dropout goes

Dropout goes after the layer whose activations you want to drop. Place one after each hidden Dense — not after the output layer (you never dropout the output).

Hint 2 — EarlyStopping wants val_loss

EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True). Without restore_best_weights, you keep the final weights, which may be slightly worse than the best epoch.

Show full solution

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential([
    Dense(256, activation='relu', input_shape=(20,)),
    Dropout(0.3),
    Dense(128, activation='relu'),
    Dropout(0.3),
    Dense(64,  activation='relu'),
    Dropout(0.3),
    Dense(1,   activation='sigmoid'),
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss',
                   patience=5,
                   restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=100,
                    validation_split=0.2,
                    callbacks=[es],
                    verbose=2)
# → Epoch 23: val_loss did not improve from 0.378 — stopping.
#   Restoring weights from epoch 18.
#   Final val_accuracy: 0.84  (up from 0.75 without regularisation)

+ setup added so this can run · defines X_train, y_train

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')

The dropout shrinks the training/validation gap; early stopping makes sure you cash in the best model rather than the latest one.

What You Learned

The train-vs-val loss curve is your first diagnostic — under, good, over, or noise.
Underfit → bigger model, more epochs, better features. Overfit → Dropout, L2, early stopping, more data.
Adam is the default optimiser; learning rate is the most important knob.
EarlyStopping(patience=5, restore_best_weights=True) and ReduceLROnPlateau(factor=0.5, patience=3) belong in nearly every callbacks list.
Set a seed, track experiments, and never tune on the test set.

Next: Convolutional Neural Networks — applying everything above to images.