Convolutional Neural Networks

A fully-connected layer on a 224×224 RGB image has 224·224·3 = 150,528 inputs. A single hidden layer of 1024 units needs 154 million parameters. That is very hard to train, and it would be a bad design anyway, because it ignores the most important fact about images: nearby pixels are correlated, and the same feature (an edge, a corner) can appear anywhere in the frame.

CNNs are designed to use both facts. A small filter slides across the image and computes the same operation at every location. The same weights are used at every position (parameter sharing), so a feature is detected wherever it appears (translation equivariance), without any extra parameters. From LeNet's 1998 demo on MNIST to ConvNeXt competing with vision transformers in 2022, the CNN has been the main architecture of modern computer vision. This lesson is a practical tour: the maths, the history of the main architectures, and the modern techniques that matter in production.

1. Convolution: The Maths, Shape by Shape

A 2D convolution takes an input feature map $X \in \mathbb{R}^{H \times W \times C_{\text{in}}}$ and a bank of $C_{\text{out}}$ filters, each of shape $k \times k \times C_{\text{in}}$. For each output channel $c$ and spatial position $(i, j)$:

$$
Y_{i,j,c} = \sum_{u=0}^{k-1} \sum_{v=0}^{k-1} \sum_{d=0}^{C_{\text{in}}-1} X_{i+u,\, j+v,\, d} \cdot K^{(c)}_{u,v,d} + b_c
$$

Concretely: every output channel is a learned linear combination of all input channels, computed by a $k \times k$ window sliding across the spatial dimensions.

Output shape formula

Given input spatial size $H_{\text{in}}$, kernel size $k$, padding $p$, stride $s$, dilation $d$:

$$
H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2p - d(k-1) - 1}{s} \right\rfloor + 1
$$

For the common case of $d = 1$:

$$
H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} - k + 2p}{s} \right\rfloor + 1
$$

Parameter count

A Conv2D layer with $C_{\text{in}}$ input channels, $C_{\text{out}}$ output channels, kernel size $k$:

$$
\text{params} = (k \cdot k \cdot C_{\text{in}} + 1) \cdot C_{\text{out}}
$$

That +1 is the bias per output channel. The count is independent of spatial size, because the same filter scans the whole image. A $3\times3$ conv from 64 to 128 channels is $(9 \cdot 64 + 1) \cdot 128 = 73,856$ parameters, no matter whether your input is $32\times32$ or $1024\times1024$.

2. Receptive Field: How Much Each Output Sees

A single $3\times3$ conv neuron sees a $3\times3$ patch of its input. Stack two of them and the second neuron sees a $5\times5$ patch of the original input. Stack three and it's $7\times7$. The receptive field grows linearly with depth for stride-1 stacks:

$$
\text{RF}_{\ell} = \text{RF}_{\ell-1} + (k_\ell - 1) \prod_{i < \ell} s_i
$$

When you add stride-2 layers, the receptive field grows much faster, because each stride-2 layer doubles the effective step size of every later layer.

Why this matters: a classification head at the top of the network needs to see the whole object. Three $3\times3$ convs see a $7\times7$ window, and four, where the second one has stride 2, see $13\times13$. To classify a 224×224 image, you need enough depth (or stride, or pooling) to grow the receptive field to roughly the size of the whole image. That is why classification CNNs are deep.

Two stacked $3\times3$ convs have the same receptive field as one $5\times5$ conv but use $2 \cdot 9 = 18$ params (per channel pair) instead of $25$, with two non-linearities instead of one. Replacing big kernels with stacks of small ones is the key idea of VGG, and most later CNNs do the same.

3. The 1×1 Convolution

A $1\times1$ conv is not "no convolution": it is a per-pixel linear projection across channels. It doesn't change the spatial dimensions, but it changes the channel count.

layers.Conv2D(filters=64, kernel_size=1)(x)    # (H, W, 256) becomes (H, W, 64)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

x = _AutoMock('x')
layers = _AutoMock('layers')

Used for:

• Channel reduction before an expensive conv ("bottleneck"). Inception and ResNet do this all the time: reduce the channels with a $1\times1$, do the expensive $3\times3$ on the reduced channels, and expand back with another $1\times1$.
• Channel mixing without spatial change. This is the same as a fully-connected layer applied separately at each spatial position.
• Network-in-Network: stack $1\times1$ convs to build per-pixel MLPs.

The $1\times1$ conv is easy to overlook, but almost every modern architecture uses it in many places to change the channel count cheaply.

4. Pooling, and Why Modern Networks Use Less of It

Pooling layers downsample the spatial dimensions with a fixed (non-learned) operation:

• Max pooling: take the max over a $k \times k$ window. The classic choice: it keeps the strongest activation and discards the rest.
• Average pooling: take the mean. Smoother, less aggressive.
• Global average pooling (GAP): take the mean over the entire spatial map → one scalar per channel. Replaces the Flatten + Dense(N) + Dense(num_classes) head with a single dense layer on top of GAP. This is the modern practice: fewer parameters, much less overfitting, and it is used in ResNet, EfficientNet, and ConvNeXt.

# Classic head: millions of parameters, and it overfits easily
x = layers.Flatten()(feature_map)
x = layers.Dense(4096, activation="relu")(x)         # 4096 * H * W * C params
x = layers.Dense(num_classes, activation="softmax")(x)

# Modern head: global average pooling (GAP), then one Dense layer
x = layers.GlobalAveragePooling2D()(feature_map)     # (B, C)
x = layers.Dense(num_classes, activation="softmax")(x)   # C * num_classes params

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

feature_map = _AutoMock('feature_map')
num_classes = _AutoMock('num_classes')
layers = _AutoMock('layers')

The GAP head also lets the network accept any input size, because the dense layer's input size depends only on the channel count, not on the feature map's spatial size.

Modern architectures often skip pooling layers completely and use stride-2 convolutions for downsampling. The effect is the same (the spatial size is halved), but the downsampling is learned.

5. The Main CNN Architectures

The history of CNN architectures is a series of simple ideas that each made a big difference.

LeNet-5 (1998)

LeCun's handwritten-digit classifier. Convolution and pooling layers, followed by fully connected layers. About 60k parameters. It showed that conv + pool + FC works. Trained on MNIST in the era before GPUs.

AlexNet (2012)

It won ImageNet 2012 by a huge margin and started the deep learning boom. 5 conv layers + 3 FC, ~60M params, ReLU activations (their first big success in a large network), dropout, GPU training. It used $11\times11$ and $5\times5$ kernels, which later architectures replaced with smaller ones.

VGG (2014)

Simonyan & Zisserman asked: "what if we stack many $3\times3$ convs?" 16 or 19 layers, all $3\times3$, all stride 1, with $2\times2$ max-pool between blocks. Roughly 138M params. The architecture is simple enough that VGG-16 is still used as a feature extractor for things like perceptual loss in image generation. The main lesson: many layers with small kernels work better than a few layers with big kernels.

Inception / GoogLeNet (2014)

Szegedy et al.: parallel branches of different kernel sizes ($1\times1$, $3\times3$, $5\times5$, pool) concatenated together: the "Inception module". It captures features at several scales in one layer. It used many $1\times1$ convs to reduce the number of channels. ~6M params despite 22 layers.

ResNet (2015)

He et al.: skip connections, $y = F(x) + x$. It trained 152-layer (and even 1000-layer) networks. With ResNet, adding many more layers improved accuracy instead of making it worse. Many later architectures use the residual idea: transformers, vision transformers, ConvNeXt, and the U-Nets used in diffusion models. ResNet is one of the most influential computer vision architectures.

DenseNet (2016)

Huang et al.: every layer takes inputs from all previous layers in the block, concatenated. Features are reused many times, so it needs relatively few parameters. Less common in production than ResNet, but the idea (dense connectivity) shows up in U-Net++ and other segmentation models.

EfficientNet (2019)

Tan & Le: "compound scaling". Instead of scaling depth, width, or input resolution separately, scale all three together with a single coefficient. EfficientNet-B0 through B7 give the best accuracy for each amount of computation (FLOPs) among the models they compared. Used Swish activations and depthwise-separable convs.

ConvNeXt (2022)

Liu et al. asked: "what if we apply the design choices that made ViT (Vision Transformer) successful to CNNs?" Larger kernels ($7\times7$), LayerNorm instead of BN, GELU instead of ReLU, inverted bottleneck, no max pool. The result is a pure CNN that matches Swin Transformer on ImageNet. It showed that CNNs can still compete with transformers.

6. A Residual Block in Code

This is the most useful architecture pattern to learn well. Two convs with batchnorm and ReLU, plus a skip connection:

from tensorflow.keras import layers, Input, Model

def residual_block(x, filters, stride=1):
    shortcut = x

    # 3x3 conv, then BN, then ReLU
    y = layers.Conv2D(filters, 3, strides=stride, padding="same", use_bias=False)(x)
    y = layers.BatchNormalization()(y)
    y = layers.ReLU()(y)

    # 3x3 conv, then BN (no activation yet)
    y = layers.Conv2D(filters, 3, strides=1, padding="same", use_bias=False)(y)
    y = layers.BatchNormalization()(y)

    # If the shortcut's shape doesn't match (different filters or stride),
    # project it with a 1x1 conv. Otherwise just add.
    if stride != 1 or shortcut.shape[-1] != filters:
        shortcut = layers.Conv2D(filters, 1, strides=stride, use_bias=False)(shortcut)
        shortcut = layers.BatchNormalization()(shortcut)

    y = layers.Add()([y, shortcut])
    y = layers.ReLU()(y)         # activation AFTER the addition
    return y

Key design points:

• use_bias=False on a conv before BN: BN has its own learnable shift, so the conv bias is not needed.
• ReLU after the add, not before. This is the "post-activation" ResNet design. The 2016 "pre-activation" variant (BN → ReLU → Conv) trains a bit better at very high depth but is otherwise equivalent.
• Shortcut projection when shapes don't match. It is required when stride != 1 (you've downsampled) or when channel count changes.

You stack these blocks. ResNet-18 has 8 of them, and ResNet-50 uses a deeper bottleneck variant (1×1 → 3×3 → 1×1) for parameter efficiency.

7. Depthwise-Separable Convolutions: The MobileNet Idea

A regular $3\times3$ conv with $C_{\text{in}} = C_{\text{out}} = C$ costs $9 C^2$ params and $9 C^2 H W$ multiply-adds. A depthwise-separable conv factors this into two cheaper steps:

1. Depthwise conv: a separate $3\times3$ kernel per input channel, no mixing across channels. Cost: $9 C$ params.
2. Pointwise conv ($1\times1$): mix channels. Cost: $C^2$ params.

Total: $9C + C^2$ vs $9 C^2$. For $C = 256$, that's $67{,}840$ vs $589{,}824$, about 9× cheaper. The loss in accuracy is small. Used in MobileNet, EfficientNet, Xception.

# Regular conv
x = layers.Conv2D(256, 3, padding="same")(x)

# Depthwise-separable equivalent
x = layers.DepthwiseConv2D(3, padding="same")(x)    # depthwise step
x = layers.Conv2D(256, 1)(x)                        # pointwise step

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

layers = _AutoMock('layers')

When you want a fast mobile model, depthwise-separable convolutions are the first optimisation to try.

8. Dilated (Atrous) Convolutions

A dilated conv with dilation $d$ inserts $d-1$ "holes" between weights. A $3\times3$ kernel with $d = 2$ has the same parameter count as a regular $3\times3$ but covers a $5\times5$ receptive field. If you stack dilated convs and double the dilation at each layer, the receptive field grows exponentially.

layers.Conv2D(64, 3, dilation_rate=2, padding="same")(x)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

x = _AutoMock('x')
layers = _AutoMock('layers')

Used in segmentation (DeepLab) and audio (WaveNet): you want a large receptive field without downsampling the spatial dimensions. Segmentation needs an output for every pixel, so you can't pool away much spatial detail. Dilation lets each output see more of the image without lowering the resolution.

9. Beyond Classification

CNNs are not only for "what's in this image". They are the backbone for many other vision tasks.

Object detection: "what's there and where"

• R-CNN family (Girshick, 2014 → Faster R-CNN, 2015): two-stage: propose regions, then classify each one. Accurate, but slower.
• YOLO (Redmon, 2016 → YOLOv8+, 2023): single-stage: predict boxes and classes in one forward pass. Fast enough for real time.
• SSD (Liu, 2016): single-shot multi-box detector. Between YOLO and Faster R-CNN.

All use a CNN backbone (ResNet, EfficientNet, ConvNeXt) followed by a detection head.

Segmentation: "label every pixel"

• U-Net (Ronneberger, 2015): encoder-decoder with skip connections from encoder to decoder. It is the most common architecture for medical imaging and the backbone of many diffusion models.
• Mask R-CNN (He, 2017): adds a segmentation head to Faster R-CNN for instance segmentation (separate masks per object).
• DeepLab (Chen, 2018): atrous convs for dense semantic segmentation.

Image generation

• GANs: a generator and a discriminator trained against each other. See GANs.
• Diffusion models: image generation by repeated denoising. They are currently the state of the art (SOTA) for image generation. U-Net is the standard denoiser backbone, but transformers (DiT) are replacing it in many newer models.

10. Data Augmentation: Better Generalisation at Low Cost

Data augmentation is one of the most effective ways to reduce overfitting. Keras has augmentation layers that can run on the GPU. Put them inside the model, so they don't slow down the CPU pipeline:

augment = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.05),
    layers.RandomZoom(0.1),
    layers.RandomContrast(0.1),
    layers.RandomBrightness(0.1),
])

# Apply only during training
inputs = keras.Input(shape=(224, 224, 3))
x = augment(inputs)                       # runs inside the model graph, so the GPU speeds it up
x = backbone(x)
outputs = head(x)
model = keras.Model(inputs, outputs)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def backbone(*_a, **_kw):
    print('-> backbone() called')
    return _AutoMock('backbone()')
def head(*_a, **_kw):
    print('-> head() called')
    return _AutoMock('head()')
keras = _AutoMock('keras')
layers = _AutoMock('layers')

These layers automatically do nothing at inference time (they only augment when training=True).

Heavier augmentations worth knowing:

• Mixup: linearly blend two images and their one-hot labels. A strong regulariser.
• CutMix: paste a random patch of one image onto another, and blend labels by the patch area.
• RandAugment / AutoAugment: AutoAugment searches for a good set of augmentations. RandAugment is a simpler version with only two settings. Both often beat hand-tuned augmentation.

For a small dataset (<10k images), augmentation often makes a large difference to test accuracy. Don't skip it.

11. Inference Optimisations

If you train in fp32 and also deploy in fp32, the model is slower and bigger than it needs to be. Three standard techniques:

Quantisation

Map fp32 weights and activations to int8 (or even int4). The model is 4× smaller, inference is often 2-4× faster, and the accuracy loss is usually small (often <1% on a well-trained model).

• Post-training quantisation: quantise after training, using a small calibration set. Quick, with no retraining.
• Quantisation-aware training (QAT): simulate quantisation noise during training so the model adapts to it. The accuracy loss is smaller.

TFLite handles both with a few lines of converter config.

Pruning

Set small-magnitude weights to zero and store the sparse model. With structured pruning (removing entire channels), the model really becomes faster. With unstructured pruning, you need hardware or software that supports sparse matrix multiplication to see a speedup.

Knowledge distillation

Train a small "student" network to mimic a large "teacher" network's outputs. The student is often better than the same network trained from scratch, because the teacher's soft targets (probabilities) carry information that hard labels don't. DistilBERT (~40% smaller, 60% faster, 97% of BERT's accuracy) is the standard example.

You can combine all three. A distilled, quantised and pruned ResNet can be many times smaller and faster than the original, with a loss of a few percent in accuracy. This can decide whether a model runs on a phone or needs a GPU.

Common Mistakes

1. BatchNorm momentum issue when fine-tuning

When you fine-tune a pretrained model with a small dataset and small batch, BN's running statistics can drift far from the pretrained values. Either freeze the BN layers (layer.trainable = False, but read the Keras docs, because the exact behaviour is subtle) or switch to GroupNorm. Many fine-tuning problems are really BN problems.

2. Very deep without residuals

A 30-layer plain CNN is very hard to train well. Without skip connections, the gradients vanish as they pass back through many layers. Add residual connections, use fewer layers, or use a normalisation scheme designed for deep networks (transformer-style pre-norm). Plain stacks deeper than about 15 layers are rarely used today.

3. No augmentation on a small dataset

If your training accuracy is 99% and validation is 70%, the fix is almost never a bigger model. Use augmentation, dropout, weight decay, and early stopping. Augmentation is the cheapest of these, so turn it on first.

4. Wrong padding mode for downsampling

padding="valid" with stride=2 and kernel_size=3 on a 224×224 input gives 111×111 (not 112×112), because $(224 - 3)/2 + 1 = 111.5 \to 111$. The lost half-pixel affects every later layer. Use "same" padding when you want predictable halving.

5. Mixing channels-first and channels-last formats

Keras defaults to channels_last (NHWC). Some pretrained models (especially imported from PyTorch) ship in channels_first (NCHW). If you mix them, the channel dimension is in the wrong place, and the outputs are wrong without any error. Always check data_format when loading external weights.

6. Forgetting to normalise inputs to match the pretrained model

ImageNet pretrained models expect inputs normalised with the ImageNet mean/std (or the model's specific preprocessing). Feeding raw [0, 1] images to a model that expects mean-subtracted inputs gives very poor accuracy. Keras has a preprocess_input function for each model family that does the right preprocessing.

🎯 Your Turn: Build a Residual Block as a Custom Keras Layer

Implement a ResidualBlock subclass of keras.layers.Layer. The block should:

• Accept filters and stride as constructor arguments.
• Apply: Conv2D(3×3) → BN → ReLU → Conv2D(3×3) → BN, add the skip connection, then ReLU.
• Project the skip with a 1×1 conv + BN when stride != 1 or when the channel count of the input doesn't match filters.
• Use use_bias=False on convs followed by BN.
• Use "same" padding so spatial size is preserved (except when stride > 1, which halves it).
• Override get_config for serialisation.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class ResidualBlock(layers.Layer):
    def __init__(self, filters, stride=1, **kwargs):
        super().__init__(**kwargs)
        # TODO 1: store filters and stride
        # TODO 2: create conv1 (3x3 stride=stride), bn1, conv2 (3x3 stride=1), bn2
        # TODO 3: prepare placeholders for the projection conv + bn (created in build)
        ...

    def build(self, input_shape):
        # TODO 4: if input channels != filters OR stride != 1, build a 1x1 projection
        #         with matching stride and bn; otherwise mark projection as None
        ...

    def call(self, x, training=False):
        # TODO 5: forward pass: conv1, bn1, relu, conv2, bn2 in that order
        #         project shortcut if needed, add, relu, return
        ...

    def get_config(self):
        return {**super().get_config(), "filters": self.filters, "stride": self.stride}


# Sanity check
inp = keras.Input(shape=(32, 32, 64))
x   = ResidualBlock(filters=128, stride=2)(inp)   # downsample + channel change
x   = ResidualBlock(filters=128, stride=1)(x)     # same shape
model = keras.Model(inp, x)
model.summary()                                   # expect (None, 16, 16, 128)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

kwargs = _AutoMock('kwargs')

Run in Colab or locally with pip install tensorflow. Expected outputs in comments.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class ResidualBlock(layers.Layer):
    """Standard ResNet basic block: Conv-BN-ReLU-Conv-BN, add the skip, then ReLU."""

    def __init__(self, filters, stride=1, **kwargs):
        super().__init__(**kwargs)
        self.filters = filters
        self.stride  = stride

        # Main path
        self.conv1 = layers.Conv2D(filters, 3, strides=stride,
                                   padding="same", use_bias=False)
        self.bn1   = layers.BatchNormalization()
        self.conv2 = layers.Conv2D(filters, 3, strides=1,
                                   padding="same", use_bias=False)
        self.bn2   = layers.BatchNormalization()

        # Will be created in build() if needed
        self.proj_conv = None
        self.proj_bn   = None

    def build(self, input_shape):
        in_channels = input_shape[-1]
        if self.stride != 1 or in_channels != self.filters:
            self.proj_conv = layers.Conv2D(self.filters, 1,
                                           strides=self.stride, use_bias=False)
            self.proj_bn   = layers.BatchNormalization()
        super().build(input_shape)

    def call(self, x, training=False):
        shortcut = x

        y = self.conv1(x)
        y = self.bn1(y, training=training)
        y = tf.nn.relu(y)

        y = self.conv2(y)
        y = self.bn2(y, training=training)

        if self.proj_conv is not None:
            shortcut = self.proj_conv(shortcut)
            shortcut = self.proj_bn(shortcut, training=training)

        y = y + shortcut
        y = tf.nn.relu(y)               # activation AFTER addition
        return y

    def get_config(self):
        return {**super().get_config(),
                "filters": self.filters, "stride": self.stride}


# Sanity check
inp = keras.Input(shape=(32, 32, 64))
x   = ResidualBlock(filters=128, stride=2)(inp)     # (None, 16, 16, 128)
x   = ResidualBlock(filters=128, stride=1)(x)       # (None, 16, 16, 128)
model = keras.Model(inp, x)
model.summary()
# Layer (type)                Output Shape              Param #
# input (InputLayer)          (None, 32, 32, 64)        0
# residual_block_0            (None, 16, 16, 128)       ~230k
# residual_block_1            (None, 16, 16, 128)       ~296k

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

kwargs = _AutoMock('kwargs')

What You Learned

• A conv layer's output shape is $\lfloor (H - k + 2p) / s \rfloor + 1$. Params are $(k^2 C_{\text{in}} + 1) C_{\text{out}}$, independent of spatial size.
• Receptive field grows with depth and stride. Stacked $3\times3$ convs beat single $5\times5$ on both params and expressiveness.
• $1\times1$ convs mix channels per-pixel. Used everywhere for cheap channel manipulation and bottlenecks.
• Global average pooling + one Dense replaces the old Flatten + huge Dense head. This is modern practice: fewer params, less overfitting, and any input size works.
• The CNN family tree: LeNet → AlexNet → VGG (stack small kernels) → Inception (parallel paths) → ResNet (skip connections) → DenseNet → MobileNet (depthwise-separable) → EfficientNet (compound scaling) → ConvNeXt (transformer design ideas applied to CNNs).
• Residual blocks are the basic building block. Remember the shape: two convs + BN + ReLU + skip + ReLU.
• Depthwise-separable convs are ~9× cheaper than regular convs for similar accuracy, which makes them the standard choice for mobile models.
• Dilated convs grow the receptive field without downsampling. They are used in segmentation and audio.
• Beyond classification: detection (YOLO, R-CNN), segmentation (U-Net, Mask R-CNN), generation (GANs, diffusion U-Net).
• Augmentation is essential on small datasets. Keras layers that run on the GPU (RandomFlip, RandomRotation...) are the easy option, but Mixup/CutMix/RandAugment are stronger options.
• Inference optimisations: quantisation (int8, 4× smaller), pruning (sparse), distillation (small student). Combine them for large speedups with a small loss in accuracy.

Next: RNNs & LSTMs: Sequential Architectures: for inputs where order matters, the architecture that reads a sequence one token at a time.