Training & Improving Your Network — Deep Learning Fundamentals

The first time you train a neural network it almost certainly doesn't work. The loss stays too high, or the validation accuracy moves away from the training accuracy, or both numbers stay flat. This is not a failure. It is the normal starting point. The skill that people who build real models have, and tutorial users often don't, is reading the training curve and knowing which setting to change.

Run code in a Colab notebook or locally with pip install tensorflow. Expected outputs are shown inline.

1. The Most Useful Plot

python

import matplotlib.pyplot as plt

history = model.fit(X_train, y_train,
                    epochs=50,
                    validation_split=0.2,
                    verbose=0)

plt.plot(history.history['loss'],     label='train loss')
plt.plot(history.history['val_loss'], label='val loss')
plt.xlabel('epoch'); plt.ylabel('loss'); plt.legend(); plt.show()

+ setup added so this can run · defines X_train, y_train, model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')

This plot is your most important check. Before you tune anything, look at the two curves.

2. The Four Curves and What They Mean

Pattern	Diagnosis	First thing to try
Both losses high, flat	Underfit: model too weak	Bigger network, more epochs, lower learning rate
Both drop, settle near each other	Good fit: ready to use	Tune for performance, not error
Train drops, val rises	Overfit: memorising training data	Dropout, L2, early stopping, more data
Both noisy, no trend	Just noise: broken setup	Check learning rate, data normalisation, label correctness

A picture for each:

python

underfit            good fit            overfit             noise
loss                loss                loss                loss
 |\                  |\                  |\__              |\/\/\
 | \________train    | \                 |   \___train     |\/\/\
 |  \________val     |  \_____train      |       \__       |\/\/\
 |                   |   \____val        |       ___val    |
 +------ epoch       +------ epoch       |      /          +------
                                         |    _/
                                         +------ epoch

First find out which of these you have, then apply the matching fix.

3. Fighting Underfit

If train and val loss are both stuck high:

Make the network bigger: more neurons per layer or more layers. An underfit model may be too small to represent the function you're asking it to learn.
Train longer: increase epochs. Sometimes 50 isn't enough.
Lower the learning rate: if the rate is too high, each step jumps past the minimum.
Better features: a neural network can't learn a pattern that is not in its input features. Poor features give poor results. See feature engineering.

4. Fighting Overfit: Four Tools

Overfit means the model has memorised the training set instead of learning the underlying pattern. Train accuracy is great, val accuracy is poor. Here are four tools, roughly in the order you should try them.

Tool 1: Dropout

Randomly zero out a fraction of neurons during training. The network can't rely on any single neuron, so it has to spread what it learns across many neurons. This generalises better.

python

from tensorflow.keras.layers import Dense, Dropout

model = Sequential([
    Dense(128, activation='relu', input_shape=(20,)),
    Dropout(0.3),                       # drop 30% of activations each step
    Dense(64,  activation='relu'),
    Dropout(0.3),
    Dense(1,   activation='sigmoid'),
])

+ setup added so this can run · defines Sequential

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')

0.2 – 0.5 is the typical range. Higher rates give stronger regularisation but slow down learning. Dropout is on during training, off during evaluation. Keras handles that automatically.

Tool 2: L2 Weight Regularisation

Penalise large weights by adding a term to the loss proportional to the sum of squared weights. This forces the model to use small weights, which gives smoother decision boundaries.

python

from tensorflow.keras.regularizers import l2

Dense(64, activation='relu', kernel_regularizer=l2(0.01))

+ setup added so this can run · defines Dense

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Dense(*_a, **_kw):
    print('-> Dense() called')
    return _AutoMock('Dense()')

0.001 – 0.01 is the usual range. Bigger numbers give stronger regularisation.

Tool 3: Early Stopping

Stop training when the validation loss stops improving, before the model starts to overfit badly. For the effort it takes, this is the most useful callback in Keras.

python

from tensorflow.keras.callbacks import EarlyStopping

es = EarlyStopping(monitor='val_loss',
                   patience=5,
                   restore_best_weights=True)

model.fit(X_train, y_train,
          epochs=200,                  # set high, because early stopping decides when to stop
          validation_split=0.2,
          callbacks=[es])
# → Epoch 27: val_loss did not improve from 0.184. Stops and restores epoch 22 weights.

+ setup added so this can run · defines X_train, y_train, model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')

patience=5 means "wait 5 more epochs for the validation loss to improve before stopping". restore_best_weights=True puts back the weights from the epoch with the best validation loss. Without it, you keep the final (possibly worse) weights.

Tool 4: Batch Normalisation

Normalise activations between layers. It also has a small regularising effect, and it lets you use higher learning rates.

python

from tensorflow.keras.layers import BatchNormalization

model = Sequential([
    Dense(128, input_shape=(20,)),
    BatchNormalization(),
    keras.layers.Activation('relu'),
    Dense(64),
    BatchNormalization(),
    keras.layers.Activation('relu'),
    Dense(1, activation='sigmoid'),
])

+ setup added so this can run · defines Sequential, Dense, keras

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
def Dense(*_a, **_kw):
    print('-> Dense() called')
    return _AutoMock('Dense()')
keras = _AutoMock('keras')

By convention, BN goes between the Dense layer and its activation. In practice you can also place it after the activation. Both work.

5. The Other Fix: More Data

More data usually helps more than any regularisation. If your model overfits and you can collect or generate more training samples, that is almost always the best thing to do. For images, data augmentation (rotations, flips, crops) gives you many more training examples without collecting new images. It is covered in CNNs.

6. Optimisers: Start with Adam

Optimiser	When
Adam	Default. Use this unless you have a reason not to.
SGD with momentum	Image classification at very large scale (some research papers)
AdamW	Modern variant of Adam with decoupled weight decay, common in transformer training
RMSprop	Older default for RNNs

Most of the time, you will use Adam. The Keras default learning rate is 0.001, which works for most problems.

python

from tensorflow.keras.optimizers import Adam, AdamW, SGD

model.compile(optimizer=Adam(learning_rate=1e-3), loss='binary_crossentropy')

+ setup added so this can run · defines model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')

7. Learning Rate: The Most Important Setting

If you only tune one hyperparameter, tune learning rate.

Too high: the loss diverges, goes to NaN, or jumps up and down. You overshoot the minimum each step.
Too low: the loss drops very slowly, and validation results stay worse than they could be.
Just right: a smooth, steady decrease, then a plateau.

Typical values: 1e-4 to 1e-2. Default 1e-3 is a good first try.

ReduceLROnPlateau: Adaptive Rate

Halve the learning rate whenever validation loss stops improving. This lets you start with a high rate and make smaller steps later.

python

from tensorflow.keras.callbacks import ReduceLROnPlateau

rlrop = ReduceLROnPlateau(monitor='val_loss',
                          factor=0.5,
                          patience=3,
                          min_lr=1e-6)

model.fit(X_train, y_train,
          epochs=100,
          validation_split=0.2,              # the callbacks watch val_loss, so it has to exist
          callbacks=[es, rlrop])
# → Epoch 15: ReduceLROnPlateau reducing learning rate to 0.0005.

+ setup added so this can run · defines X_train, y_train, model, es

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
model = _AutoMock('model')
es = _AutoMock('es')

Use it together with early stopping. It often improves results with no extra work.

8. Batch Size: The Other Setting

Batch size	Effect
Small (8–32)	Noisier gradients, often better generalisation, slower wall-clock
Medium (64–128)	Usually the best balance
Large (256+)	Smoother gradients, fastest training on GPU, can hurt generalisation

If you're starting fresh, use 32. Increase if training is too slow and the GPU has spare memory. Decrease if you're seeing erratic loss curves and have a small dataset.

9. Data Augmentation: A Preview

For images, you can create more data by applying random transformations: flip, rotate, crop, zoom, colour-jitter. Each epoch sees slightly different inputs, which forces the model to learn features that don't change under these transformations, instead of memorising.

python

from tensorflow.keras import layers

augment = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),
    layers.RandomZoom(0.1),
])

+ setup added so this can run · defines keras

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

keras = _AutoMock('keras')

This is covered fully in CNNs.

10. Experiment Tracking

Once you're running more than a few models, you need to track them. Two options:

TensorBoard: built into TensorFlow. Add keras.callbacks.TensorBoard(log_dir='logs/') to your callbacks and run tensorboard --logdir logs/. Free, local, plots loss curves and weight histograms.
Weights & Biases (wandb): a hosted service with a polished interface, free for individuals. One-line setup, tracks code/hyperparams/metrics across hundreds of runs.

For your first ten models, plotting history.history is fine. After that, install one of the above. Without a record, you will forget which hyperparameters gave you that 92% result last week.

Common Mistakes

Train accuracy 99%, val accuracy 60%. This is classic overfitting. Add Dropout(0.3), add EarlyStopping, possibly shrink the network. Don't keep training.
Both stuck at chance. Either underfit (network too small or learning rate too low) or broken setup. Check that each label belongs to the right input, that you normalised inputs to a sensible range, and that the loss matches the task.
Loss goes to NaN. The learning rate is too high. Divide it by 10. Also check for log(0) issues if you wrote a custom loss.
Not setting a seed. Different runs give different results. This is acceptable in production, but very frustrating when you are debugging. tf.keras.utils.set_random_seed(42) covers Python, NumPy, and TF in one call.
Tweaking on the test set. If you're picking hyperparameters by looking at test-set numbers, your test set has become a validation set and you have no real holdout left. Keep test untouched until the end.

🎯 Your Turn: Fix an Overfitting Model

You're given this overfit-prone model trained on a small tabular dataset. Train accuracy hits 99%, validation accuracy stalls at 75%. Add Dropout(0.3) after each hidden layer and add an EarlyStopping callback with patience=5 and restore_best_weights=True. Retrain.

Skeleton:

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential([
    Dense(256, activation='relu', input_shape=(20,)),
    # TODO 1: dropout
    Dense(128, activation='relu'),
    # TODO 2: dropout
    Dense(64,  activation='relu'),
    # TODO 3: dropout
    Dense(1,   activation='sigmoid'),
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# TODO 4: early stopping callback

history = model.fit(X_train, y_train,
                    epochs=100,
                    validation_split=0.2,
                    callbacks=[ ... ])     # TODO 5

+ setup added so this can run · defines X_train, y_train

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')

Hint 1: Where Dropout goes

Dropout goes after the layer whose activations you want to drop. Place one after each hidden Dense, but not after the output layer (you never apply dropout to the output).

Hint 2: EarlyStopping monitors val_loss

EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True). Without restore_best_weights, you keep the final weights, which may be slightly worse than the best epoch.

Show full solution

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential([
    Dense(256, activation='relu', input_shape=(20,)),
    Dropout(0.3),
    Dense(128, activation='relu'),
    Dropout(0.3),
    Dense(64,  activation='relu'),
    Dropout(0.3),
    Dense(1,   activation='sigmoid'),
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss',
                   patience=5,
                   restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=100,
                    validation_split=0.2,
                    callbacks=[es],
                    verbose=2)
# → Epoch 23: val_loss did not improve from 0.378. Training stops.
#   Restoring weights from epoch 18.
#   Final val_accuracy: 0.84  (up from 0.75 without regularisation)

+ setup added so this can run · defines X_train, y_train

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')

Dropout makes the gap between training and validation smaller. Early stopping makes sure you keep the best model, not the last one.

What You Learned

The train-vs-val loss curve is your first check: underfit, good fit, overfit or noise.
Underfit → bigger model, more epochs, better features. Overfit → Dropout, L2, early stopping, more data.
Adam is the default optimiser, and learning rate is the most important setting.
EarlyStopping(patience=5, restore_best_weights=True) and ReduceLROnPlateau(factor=0.5, patience=3) belong in nearly every callbacks list.
Set a seed, track experiments, and never tune on the test set.

Next: Convolutional Neural Networks, where you apply all of this to images.