Transfer Learning: Standing on Pretrained Shoulders

Runtime note — code in this lesson uses TensorFlow/Keras and Hugging Face Transformers. Neither runs inside the in-browser Pyodide sandbox. Execute these snippets in a local Python environment (Colab, a CUDA workstation, or a CPU-only laptop for the smaller models).

Almost nobody trains a serious model from scratch any more. They take a network that was already trained on hundreds of millions of images, or hundreds of billions of tokens, and bolt their tiny dataset onto the end of it. The result is a model that converges in minutes instead of weeks and beats from-scratch baselines by double-digit margins — especially when your dataset is small.

That trick is transfer learning, and in 2026 it is the default starting point for almost every applied vision, language, and audio project. This lesson covers the why, the two operating modes, the canonical workflow, the hyperparameters that matter, and the failure modes that will burn you if you treat it as a magic spell.

1. Why It Works — A Hierarchy of Features

A deep network doesn't learn one giant feature detector; it learns a stack of progressively more abstract ones. Visualise the filters of an ImageNet-trained CNN and you see a clean gradient:

• Early layers — edges, colour blobs, simple textures. Universal. A cat photo and an X-ray both contain edges.
• Middle layers — corners, repeating patterns, simple shapes. Still mostly generic.
• Late layers — task-specific compositions. "Pointy ear", "tabby stripe", "the part of the image where the dog's face usually sits".
• Classifier head — projects the final feature vector onto the 1,000 ImageNet classes.

When you move to a new task with a small dataset, the early and middle layers are already doing useful work for free. Throwing them away to retrain from a random initialisation is wasteful — you'd just relearn edges and textures with less data and worse results. Transfer learning keeps those layers and replaces only the parts that are genuinely task-specific.

The same hierarchy holds in NLP. A language model trained on a trillion tokens has internalised grammar, common-sense, factual associations, and discourse structure. Your sentiment classifier doesn't need to relearn what a noun is.

2. The Two Modes

You almost always do both, in sequence. That's the canonical workflow.

3. The Canonical Workflow

1. Load a pretrained backbone with the classification head removed.
2. Replace the head with one sized to your task (e.g. a Dense(2) for binary classification).
3. Freeze the backbone. Train only the new head for a few epochs at normal LR. This stabilises the random head before its gradients are allowed to disturb the backbone's good weights.
4. Unfreeze the backbone (or just the top half). Recompile with a very small LR.
5. Train end-to-end for a small number of additional epochs.

Skipping step 3 — unfreezing immediately — is the single most common mistake. The fresh, randomly-initialised head produces enormous gradients on the first batches; backpropagating those into a perfectly-tuned backbone can wipe out months of pretraining in a few steps.

4. A Complete Image Example — MobileNetV2

We'll fine-tune MobileNetV2 (pretrained on ImageNet) onto a binary cats-vs-dogs task. MobileNetV2 is small enough to train on a laptop yet representative of the workflow you'd use with a ResNet, EfficientNet, or ViT.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

IMG_SIZE = (160, 160)
BATCH = 32

# 1. Load data (any tf.data pipeline; here a placeholder)
train_ds, val_ds = ...    # yields (image, label) batches in [0, 255]

# 2. Build the backbone — exclude the original 1000-class head
base = keras.applications.MobileNetV2(
    input_shape=IMG_SIZE + (3,),
    include_top=False,                          # drop the ImageNet classifier
    weights="imagenet",
)
base.trainable = False                          # freeze for stage 1

# 3. Build the new model on top
inputs = keras.Input(shape=IMG_SIZE + (3,))
x = keras.applications.mobilenet_v2.preprocess_input(inputs)   # [-1, 1] scaling
x = base(x, training=False)                     # training=False keeps BN frozen
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)

# 4. Stage 1 — train only the head
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

# 5. Stage 2 — unfreeze and fine-tune with a tiny LR
base.trainable = True
# Optionally keep the very early layers frozen — they're the most generic
for layer in base.layers[:100]:
    layer.trainable = False

model.compile(
    optimizer=keras.optimizers.Adam(1e-5),      # 100× smaller
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

A few details that matter:

• base(x, training=False) keeps BatchNorm layers in inference mode. BN's running mean/variance were tuned on ImageNet; updating them on a 200-image cats-vs-dogs batch is catastrophic. Even when you unfreeze in stage 2, you usually keep BN frozen — see Common Mistakes.
• preprocess_input does the same scaling the model saw during pretraining. Skip it and your inputs are in the wrong range — the network is solving a different problem than it was trained on.
• GlobalAveragePooling2D collapses the spatial dimensions to one vector per image — far fewer parameters than a Flatten followed by a huge Dense.

5. The Same Pattern in Text — Hugging Face Transformers

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=2,            # replaces the head automatically
)

# Tokenise your dataset, then:
args = TrainingArguments(
    output_dir="out",
    learning_rate=2e-5,                  # the LR that just works for BERT-style fine-tuning
    per_device_train_batch_size=16,
    num_train_epochs=3,
    eval_strategy="epoch",
)
Trainer(model=model, args=args, train_dataset=..., eval_dataset=...).train()

from_pretrained(..., num_labels=2) is the text equivalent of "load backbone, replace head". The Trainer handles the freeze/unfreeze choreography for you — by default it fine-tunes the whole stack at a small LR. For BERT-family models, 2e-5 is the default that has shipped to production at most companies that fine-tune transformers.

Audio works the same way with Wav2Vec2ForSequenceClassification. The pattern is universal: load a backbone, attach a task-appropriate head, fine-tune at a small LR.

6. The Hyperparameters That Actually Matter

In rough order of impact:

1. Learning rate. 10×–100× smaller than from-scratch. The most common LRs you'll see in published fine-tunes: 2e-5 for transformer NLP, 1e-4 to 1e-5 for vision CNNs/ViTs, 1e-3 for the head-only stage.
2. Number of frozen layers. All-frozen is fastest and safest. Unfreezing the top half is the common sweet spot. Unfreezing everything works when your dataset is large.
3. Batch size. Bigger batches stabilise gradients but reduce stochasticity. For fine-tuning, 16–64 is the typical range — small because you usually don't have enough data to justify more.
4. LR schedule. A linear warmup followed by linear decay (the "BERT schedule") is a no-brainer for transformer fine-tuning. For vision, cosine decay is common.
5. Number of epochs. Fine-tuning converges fast. 3–10 epochs is normal. If you're running 100 epochs you're either overfitting wildly or your LR is too small.

The single best debugging move when transfer learning underperforms: lower the learning rate. Then lower it again.

7. When Transfer Learning Fails

The pretraining task and your task need to share enough underlying structure for the features to transfer. Some calibration:

• Works well — natural images → medical X-rays. Both are 2D continuous-tone images; edges and textures are universal. Even the gap from "everyday photos" to "chest X-rays" is small enough that ImageNet weights consistently beat from-scratch.
• Iffy — natural images → satellite imagery. Different scale, different colour statistics, different object priors. ImageNet weights help, but task-specific pretraining (e.g. on aerial imagery) helps more.
• Iffy — natural images → spectrograms or scientific imaging where pixels don't represent "stuff with edges".
• Often fails — ImageNet → tabular data, time series, or graph data. There is no shared visual structure. Use a domain-appropriate pretrained model or skip transfer learning.
• NLP equivalent — English LLM → a low-resource language with a different script. Tokenisation breaks first; semantics may never transfer. Use a multilingual model (mBERT, XLM-R) or domain-pretrain.

Domain adaptation is the structured fix when the gap is too large: pretrain → intermediate pretrain on unlabelled in-domain data → task fine-tune. Standard for medical NLP (BioBERT, ClinicalBERT) and chemistry models.

8. Why Even Big Teams Use It

The folklore says transfer learning is for small-data startups. In practice the biggest labs still start from pretrained checkpoints, for two reasons:

• Compute economics. A from-scratch run of a serious vision or language model is hundreds of GPU-days. A fine-tune is hours. Even when budget allows the former, the latter ships sooner.
• Convergence quality. Starting from a good initialisation lets you train at lower LRs without getting stuck in poor local minima. Final test accuracy is often higher than from-scratch even when you have plenty of data.

The only places people still do serious from-scratch training are (a) the foundation-model labs whose product is the pretrained checkpoint, and (b) tasks where no suitable pretrained model exists.

Common Mistakes

1. Same learning rate as from-scratch.

The pretrained weights are close to a good solution; large gradients shove them away from it. Drop the LR by 10×–100× when fine-tuning. If your loss spikes on the first epoch after unfreezing, that's the symptom.

2. Forgetting training=False on the frozen backbone.

BatchNorm layers have two states: training (uses batch statistics, updates running averages) and inference (uses the running averages). When you "freeze" a layer by setting trainable=False, BN's weights freeze but its running averages still update unless you also pass training=False at call time. The fix is either base(x, training=False) as shown above, or base.trainable = False plus keeping training=False even after unfreezing the rest. Skipping this on a tiny batch is enough to destroy a model.

3. Wrong preprocessing.

Each pretrained model expects specific input scaling — [-1, 1] for MobileNet, [0, 1] for some ResNets, ImageNet-normalised for ViTs. Use the bundled preprocess_input rather than guessing. The same applies in NLP: always use the tokenizer that ships with the model.

4. Picking a pretrained model from a wildly different domain.

Fine-tuning a clip-art model on dermatology images, or an English BERT on Tamil, or a music model on speech — when the gap is too large, transfer learning underperforms from-scratch. Spend the five minutes to check whether a domain-specific checkpoint exists before committing.

5. Skipping the head-only warmup.

Unfreezing the backbone before the new head has stabilised pushes huge first-batch gradients backwards through the pretrained layers, undoing the pretraining. Always train the head alone for a few epochs first.

6. Re-shuffling the labels.

If your binary classifier outputs probability of "dog" but you train on a dataset where label 0 is dog and label 1 is cat, you're optimising the inverse problem. Check label encoding before you start a long fine-tune run.

🎯 Your Turn — Fine-Tune MobileNetV2 on a Two-Class Task

Build the full transfer-learning workflow on a small binary image classification task (the cats-vs-dogs subset from tensorflow_datasets is a clean target, but any two-class folder of images works).

Requirements:

• Use keras.applications.MobileNetV2(weights="imagenet", include_top=False).
• Add a GlobalAveragePooling2D → Dropout(0.2) → Dense(1, sigmoid) head.
• Stage 1: freeze the backbone, train the head for 5 epochs at LR 1e-3.
• Stage 2: unfreeze the top half of the backbone, recompile at LR 1e-5, train 5 more epochs.
• Use the model's preprocess_input and pass training=False to the frozen backbone.
• Print final validation accuracy.

Skeleton:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

IMG_SIZE = (160, 160)

# TODO 1: load train_ds, val_ds (any tf.data pipeline producing (image, label) pairs)

# TODO 2: build the backbone with include_top=False, weights="imagenet"
#         freeze it

# TODO 3: stack the new head — preprocess_input, base(x, training=False),
#         GlobalAveragePooling2D, Dropout, Dense(1, sigmoid)

# TODO 4: compile with Adam(1e-3), binary_crossentropy; fit 5 epochs

# TODO 5: unfreeze the top half of base.layers; freeze BN layers explicitly
# TODO 6: recompile with Adam(1e-5); fit 5 more epochs

# TODO 7: print final val accuracy

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow_datasets as tfds

IMG_SIZE = (160, 160)
BATCH = 32

# 1. Load cats_vs_dogs from TFDS; resize and batch
(train_raw, val_raw), info = tfds.load(
    "cats_vs_dogs", split=["train[:80%]", "train[80%:]"],
    as_supervised=True, with_info=True,
)
def resize(image, label):
    return tf.image.resize(image, IMG_SIZE), label
train_ds = train_raw.map(resize).batch(BATCH).prefetch(tf.data.AUTOTUNE)
val_ds = val_raw.map(resize).batch(BATCH).prefetch(tf.data.AUTOTUNE)

# 2. Backbone, frozen
base = keras.applications.MobileNetV2(
    input_shape=IMG_SIZE + (3,),
    include_top=False,
    weights="imagenet",
)
base.trainable = False

# 3. Head
inputs = keras.Input(shape=IMG_SIZE + (3,))
x = keras.applications.mobilenet_v2.preprocess_input(inputs)
x = base(x, training=False)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)

# 4. Stage 1 — head only
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

# 5. Unfreeze top half, keep BN frozen
base.trainable = True
cutoff = len(base.layers) // 2
for layer in base.layers[:cutoff]:
    layer.trainable = False
for layer in base.layers:
    if isinstance(layer, layers.BatchNormalization):
        layer.trainable = False

# 6. Stage 2 — fine-tune
model.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
history = model.fit(train_ds, validation_data=val_ds, epochs=5)

print(f"Final val accuracy: {history.history['val_accuracy'][-1]:.4f}")

What You Learned

• A deep network learns a hierarchy of features — early layers are generic, late layers are task-specific. Transfer learning reuses the generic part.
• Feature extraction freezes the backbone and trains only a new head. Fine-tuning then unfreezes some/all layers and trains end-to-end at a tiny LR.
• The canonical workflow is head-warmup → unfreeze → fine-tune. Skipping the warmup destroys the pretrained weights.
• Fine-tuning LRs are 10×–100× smaller than from-scratch. The most common values: 2e-5 for transformer NLP, 1e-4–1e-5 for vision.
• Always use the model's own preprocess_input (or its tokenizer). Wrong input scaling silently breaks everything.
• Keep BatchNorm in inference mode when fine-tuning small batches. Pass training=False or set the BN layers trainable=False explicitly.
• Transfer learning fails when the source and target domains share too little structure. Pick a pretrained model from a related domain, or domain-adapt with intermediate pretraining.
• Even teams with abundant data start from pretrained checkpoints — it converges faster and usually scores higher.

Next: Natural Language Processing — applying transfer learning across the full NLP task taxonomy, from classification to retrieval, using the Hugging Face ecosystem.