Convolutional Neural Networks — Deep Learning Fundamentals

A 224×224 RGB image is 150,528 numbers. A fully-connected (Dense) network with a single 1,000-neuron hidden layer would need 150 million weights just for that first layer, and many of them would learn the same edge detector at slightly different pixel positions. That is wasteful and slow, and it makes overfitting very likely.

Convolutional Neural Networks fix this with two ideas: local receptive fields (each filter looks at a small patch) and weight sharing (the same filter slides across the whole image). The result is a much smaller model. Because the same filter is used everywhere, it detects a pattern wherever the pattern appears in the image (this is called translation equivariance: if the pattern moves, the filter's output moves with it). CNNs work very well for vision tasks.

Run in Colab (the free GPU is strongly recommended) or pip install tensorflow. Outputs are shown inline.

1. The Convolution: A Sliding Filter

Imagine a small 3×3 grid of weights, called a filter or kernel. Slide it across the image one pixel at a time. At each position, compute a weighted sum of the 9 pixels under the filter. The output is a new grid, called a feature map, slightly smaller than the input.

python

Image (5×5)        Filter (3×3)        Output (3×3)
[1 1 1 0 0]        [1 0 -1]            [.  .  .]
[0 1 1 1 0]        [1 0 -1]   = slide  [.  .  .]
[0 0 1 1 1]        [1 0 -1]            [.  .  .]
[0 0 1 1 0]
[0 1 1 0 0]

The same 9 weights are used at every position. This is the weight sharing that makes CNNs efficient. On a grayscale image, a first layer with 10 filters of size 3×3 has only 9·10 = 90 weights (plus 10 biases), whatever the size of the image.

A filter learns one pattern: maybe a vertical edge, a 45° edge, a red blob. A convolutional layer is a stack of filters, each learning a different pattern.

2. The Hierarchy Visualised

What CNNs learn, layer by layer:

Layer 1: edges at different angles, and colour blobs. These are similar to the features found in the early stages of the human visual system.
Layer 2: textures, corners, simple curves.
Layer 3: object parts, such as wheels, eyes and fur patches.
Deep layers: detectors for whole objects, such as "car", "cat", "stop sign".

Nobody programs this hierarchy. It comes from training on labelled images, because useful features at one layer make more useful features possible at the next. The first layers of most image CNNs end up looking like a similar set of edge detectors (similar to Gabor filters). This is one of the most interesting results in machine learning.

3. The Canonical Pattern

python

Conv → Pool → Conv → Pool → ... → Flatten → Dense → output

Conv2D: extract features (replaces Dense).
Pooling: shrink the feature map, which also makes the output less sensitive to small shifts.
Repeat: deeper layers see larger areas of the original image.
Flatten: turn the 3D feature map into a 1D vector.
Dense: a classifier on top of the learned features.

Almost every image CNN you'll see follows this template. Modern architectures (ResNet, EfficientNet) mainly add skip connections or change how the layers connect to each other.

4. A Minimal Keras CNN

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(10, activation='softmax'),
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.summary()
# Total params: ~34,800 (far fewer than a network of only Dense layers would need)

That's a complete MNIST classifier. Read it as: extract 32 features with 3×3 filters, halve the spatial dimensions, extract 64 features at the next scale, halve again, flatten, classify into 10 digits.

Train it on MNIST:

python

from tensorflow.keras.datasets import mnist
import numpy as np

(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalise to [0, 1] and add channel dimension
X_train = X_train.astype('float32') / 255.0
X_test  = X_test.astype('float32')  / 255.0
X_train = X_train[..., np.newaxis]      # shape (60000, 28, 28, 1)
X_test  = X_test[...,  np.newaxis]

model.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.1)
# → Epoch 5/5  loss: 0.020 - accuracy: 0.993 - val_loss: 0.035 - val_accuracy: 0.990

+ setup added so this can run · defines model

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')

Five epochs give 99% on MNIST, the standard first dataset in computer vision. This is the advantage of CNNs.

5. Conv2D Parameters

python

Conv2D(filters, kernel_size, strides=(1, 1), padding='valid', activation=None)

+ setup added so this can run · defines Conv2D, filters, kernel_size

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Conv2D(*_a, **_kw):
    print('-> Conv2D() called')
    return _AutoMock('Conv2D()')
filters = _AutoMock('filters')
kernel_size = _AutoMock('kernel_size')

filters: how many independent filters this layer learns. 32, 64, 128 are typical.
kernel_size: the spatial size of each filter. (3, 3) is the modern default. Older architectures sometimes used larger filters (5×5, 7×7) in early layers.
strides: how many pixels the filter moves at each step. (1, 1) keeps full resolution, and (2, 2) halves it (an alternative to pooling).
padding: what to do at the edges:

- 'valid': no padding. Output shrinks by kernel_size − 1 per dimension. - 'same': pad with zeros so output keeps the input's spatial size.

activation: almost always 'relu'.

A 3×3 conv with padding='same' is used very often: the output has the same shape as the input, and the layer learns local features.

6. Pooling

A pooling layer aggregates a small region of the feature map into one value. There are two common types:

MaxPooling2D((2,2)): keep the strongest signal in each 2×2 region. Default choice.
AveragePooling2D((2,2)): keep the average. Sometimes preferred in modern architectures.

Pooling does two useful things at once:

1. Shrinks the feature map (less compute downstream).
2. Makes the output less sensitive to small shifts: if the input moves by a pixel or two, the pooled output barely changes.

Most architectures pool with stride 2, halving each spatial dimension.

A modern alternative is a Conv2D with strides=(2, 2) and no pooling layer. It shrinks the feature map and extracts features in one step. Either pattern is fine.

7. Data Shape: Get This Right First

Keras CNNs expect inputs of shape:

python

(batch_size, height, width, channels)

+ setup added so this can run · defines batch_size, height, width, channels

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

batch_size = _AutoMock('batch_size')
height = _AutoMock('height')
width = _AutoMock('width')
channels = _AutoMock('channels')

Grayscale (MNIST): channels = 1.
RGB (CIFAR, ImageNet): channels = 3.
RGBA (with alpha): channels = 4.

If you have (N, H, W) for grayscale, add the channel axis: X = X[..., np.newaxis].

Pixel values must be normalised: divide by 255 to put them in [0, 1], or subtract a mean to get them roughly centred. Raw 0–255 pixels make training unstable.

8. Data Augmentation: More Data Without Collecting It

For small image datasets, this is the most useful technique. Apply random transformations during training so the model sees a slightly different version of each image each epoch.

The modern Keras way uses preprocessing layers directly in the model:

python

from tensorflow.keras import layers

data_augmentation = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),         # up to ±10% of a full turn (±36 degrees)
    layers.RandomZoom(0.1),
    layers.RandomContrast(0.1),
])

model = Sequential([
    data_augmentation,                  # active only during training
    Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
    MaxPooling2D((2, 2)),
    # ... rest of the network
])

+ setup added so this can run · defines Sequential, keras, Conv2D, MaxPooling2D

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
keras = _AutoMock('keras')
def Conv2D(*_a, **_kw):
    print('-> Conv2D() called')
    return _AutoMock('Conv2D()')
def MaxPooling2D(*_a, **_kw):
    print('-> MaxPooling2D() called')
    return _AutoMock('MaxPooling2D()')

These layers do nothing at inference time, so you don't need to "turn them off" yourself.

Common augmentations and what each one helps with:

Augmentation	Builds invariance to
RandomFlip(horizontal)	left/right pose (NOT for text, NOT for asymmetric objects)
RandomRotation	small rotations
RandomZoom	scale changes
RandomCrop	framing differences
RandomBrightness/Contrast	lighting conditions

Pick the ones that match real variation in your data. Don't flip handwritten digits. After a horizontal flip, a 6 is no longer a real digit.

9. Transfer Learning: Start from a Trained Network

In industry, you almost never train a CNN from scratch. You take a network that was already trained on millions of ImageNet images (ResNet, EfficientNet, MobileNet), remove its final classifier, and add your own small classifier (the "head"):

python

from tensorflow.keras.applications import EfficientNetB0

base = EfficientNetB0(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
base.trainable = False                  # freeze the pretrained weights

model = Sequential([
    base,
    layers.GlobalAveragePooling2D(),
    layers.Dense(128, activation='relu'),
    layers.Dense(num_classes, activation='softmax'),
])

+ setup added so this can run · defines Sequential, num_classes, layers

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
num_classes = _AutoMock('num_classes')
layers = _AutoMock('layers')

A model like this can often reach good accuracy on a custom dataset of only about 1,000 images. Training from scratch usually needs far more data. This is covered fully in transfer.

10. The CNN Mental Checklist

Before you model.fit() an image model, verify:

Input shape is (H, W, channels) and the first layer's input_shape matches.
Pixels are normalised to [0, 1] or [-1, 1].
Labels are the right format for the loss (integer for sparse_categorical_crossentropy, one-hot for categorical_crossentropy).
Final layer has correct activation: softmax(N) for N-class, sigmoid(1) for binary.
For small datasets, you use augmentation, a pretrained backbone, or both.

Common Mistakes

Wrong input shape. Grayscale data with shape (N, 28, 28) raises an error when the first Conv2D expects (N, 28, 28, 1). Add np.newaxis.
Not normalising pixels. 0–255 inputs make the first layer's gradients huge. Loss often goes to NaN within a few batches. Divide by 255.
Tiny dataset + huge CNN from scratch. It will overfit very quickly. Use a pretrained backbone and freeze most of it.
Augmenting the test set. Augmentation belongs in training only. The Keras layers.RandomXxx preprocessing layers handle this automatically, but older ImageDataGenerator requires a separate non-augmented generator for validation.
Forgetting Flatten before Dense. Conv2D outputs 3D feature maps, but Dense expects 1D. Without Flatten() (or a GlobalAveragePooling2D()) the shapes won't connect.

🎯 Your Turn: MNIST CNN

Build a small CNN for MNIST. Input shape (28, 28, 1). Two Conv→Pool blocks (32 then 64 filters, 3×3 kernels), then Flatten, then Dense(10, softmax). Compile and train for 5 epochs.

Assume X_train, y_train, X_test, y_test are loaded and normalised (see Section 4).

Skeleton:

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential([
    # TODO 1: Conv2D with 32 filters, 3x3, ReLU, input_shape for MNIST
    # TODO 2: MaxPooling2D 2x2
    # TODO 3: Conv2D with 64 filters, 3x3, ReLU
    # TODO 4: MaxPooling2D 2x2
    # TODO 5: Flatten
    # TODO 6: Dense 10 softmax
])

# TODO 7: compile with adam, sparse_categorical_crossentropy, accuracy
# TODO 8: fit for 5 epochs with batch_size=64 and validation_split=0.1

Hint 1: MNIST shape

input_shape=(28, 28, 1) on the first layer only. Grayscale, so channels=1.

Hint 2: Which loss

Labels are integers 0–9, not one-hot. Use sparse_categorical_crossentropy. If you'd one-hot encoded them, you'd use plain categorical_crossentropy.

Show full solution

python

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(10, activation='softmax'),
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(X_train, y_train,
                    epochs=5,
                    batch_size=64,
                    validation_split=0.1,
                    verbose=2)
# → Epoch 5/5  loss: 0.021 - accuracy: 0.993 - val_loss: 0.037 - val_accuracy: 0.989

test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"test accuracy: {test_acc:.3f}")
# → test accuracy: 0.990

+ setup added so this can run · defines X_train, y_train, X_test, y_test

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train = _AutoMock('X_train')
y_train = _AutoMock('y_train')
X_test = _AutoMock('X_test')
y_test = _AutoMock('y_test')

Around 35,000 parameters, five epochs, 99% accuracy. A fully-connected network on the same task usually reaches about 98% at best, with ten times as many parameters. That difference is the advantage of CNNs.

What You Learned

A convolution slides a small filter across the image, using the same weights at every position. This makes CNNs efficient, and they detect a pattern wherever it appears.
The canonical pattern: Conv → Pool → Conv → Pool → Flatten → Dense.
Pooling shrinks feature maps and adds small-shift invariance.
Inputs must be (N, H, W, channels) and normalised to [0, 1].
Data augmentation and transfer learning are the two most useful techniques for real-world image problems.

Next: Working with Text: NLP Basics, the same approach applied to sequences of tokens.