Transfer Learning: Standing on Pretrained Shoulders

Runtime note: the code in this lesson uses TensorFlow/Keras and Hugging Face Transformers. Neither runs inside the in-browser Pyodide sandbox. Execute these snippets in a local Python environment (Colab, a CUDA workstation, or a CPU-only laptop for the smaller models).

Almost nobody trains a serious model from scratch any more. They take a network that was already trained on hundreds of millions of images, or hundreds of billions of tokens, and train it further on their own small dataset. The result is a model that converges in minutes instead of weeks, and usually beats a model trained from scratch, especially when your dataset is small.

This is called transfer learning, and in 2026 it is the default starting point for almost every applied vision, language, and audio project. This lesson covers the why, the two operating modes, the canonical workflow, the hyperparameters that matter, and the cases where it fails.

1. Why It Works: A Hierarchy of Features

A deep network learns a stack of feature detectors, each one more abstract than the one before it. If you visualise the filters of an ImageNet-trained CNN, you see a clear progression from simple to complex:

• Early layers: edges, colour blobs, simple textures. These are universal. A cat photo and an X-ray both contain edges.
• Middle layers: corners, repeating patterns, simple shapes. Still mostly generic.
• Late layers: task-specific combinations. "Pointy ear", "tabby stripe", "the part of the image where the dog's face usually sits".
• Classifier head: maps the final feature vector onto the 1,000 ImageNet classes.

When you move to a new task with a small dataset, the early and middle layers already do useful work, and you don't have to train them. Throwing them away to retrain from a random initialisation is wasteful: you would relearn edges and textures with less data and get worse results. Transfer learning keeps those layers and replaces only the parts that are truly task-specific.

The same hierarchy holds in NLP. A language model trained on a trillion tokens has learned grammar, common sense, factual associations and how sentences connect into longer text. Your sentiment classifier doesn't need to relearn what a noun is.

2. The Two Modes

You almost always do both, one after the other. That is the standard workflow.

3. The Standard Workflow

1. Load a pretrained backbone with the classification head removed.
2. Replace the head with one sized to your task (e.g. a Dense(2) for binary classification).
3. Freeze the backbone. Train only the new head for a few epochs at normal LR. This stabilises the random head before its gradients are allowed to disturb the backbone's good weights.
4. Unfreeze the backbone (or just the top half). Recompile with a very small LR.
5. Train end-to-end for a small number of additional epochs.

Skipping step 3 (unfreezing immediately) is a very common mistake. The new, randomly initialised head produces very large gradients on the first batches. Backpropagating those into a well-trained backbone can destroy months of pretraining in a few steps.

4. A Complete Image Example: MobileNetV2

We'll fine-tune MobileNetV2 (pretrained on ImageNet) onto a binary cats-vs-dogs task. MobileNetV2 is small enough to train on a laptop, and the workflow is the same one you'd use with a ResNet, EfficientNet, or ViT.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

IMG_SIZE = (160, 160)
BATCH = 32

# 1. Load data (any tf.data pipeline; here a placeholder)
train_ds, val_ds = ...    # yields (image, label) batches in [0, 255]

# 2. Build the backbone without the original 1000-class head
base = keras.applications.MobileNetV2(
    input_shape=IMG_SIZE + (3,),
    include_top=False,                          # drop the ImageNet classifier
    weights="imagenet",
)
base.trainable = False                          # freeze for stage 1

# 3. Build the new model on top
inputs = keras.Input(shape=IMG_SIZE + (3,))
x = keras.applications.mobilenet_v2.preprocess_input(inputs)   # [-1, 1] scaling
x = base(x, training=False)                     # training=False keeps BN frozen
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)

# 4. Stage 1: train only the head
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

# 5. Stage 2: unfreeze and fine-tune with a tiny learning rate
base.trainable = True
# Optionally keep the very early layers frozen, because they are the most generic
for layer in base.layers[:100]:
    layer.trainable = False

model.compile(
    optimizer=keras.optimizers.Adam(1e-5),      # 100× smaller
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

A few details that matter:

• base(x, training=False) keeps BatchNorm layers in inference mode. BN's running mean/variance were tuned on ImageNet. Updating them on a 200-image cats-vs-dogs batch is catastrophic. Even when you unfreeze in stage 2, you usually keep BN frozen (see Common Mistakes).
• preprocess_input does the same scaling the model saw during pretraining. If you skip it, your inputs are in the wrong range, and the network sees data that looks different from what it was trained on.
• GlobalAveragePooling2D averages over the spatial dimensions to give one vector per image. That needs far fewer parameters than a Flatten followed by a huge Dense.

5. The Same Pattern in Text: Hugging Face Transformers

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=2,            # replaces the head automatically
)

# Tokenise your dataset, then:
args = TrainingArguments(
    output_dir="out",
    learning_rate=2e-5,                  # a learning rate that works well for BERT-style fine-tuning
    per_device_train_batch_size=16,
    num_train_epochs=3,
    eval_strategy="epoch",
)
Trainer(model=model, args=args, train_dataset=..., eval_dataset=...).train()

from_pretrained(..., num_labels=2) is the text equivalent of "load backbone, replace head". The Trainer does not use separate freeze and unfreeze stages: by default it fine-tunes the whole model at a small LR. For BERT-family models, 2e-5 is a very common default.

Audio works the same way with Wav2Vec2ForSequenceClassification. The pattern is always the same: load a backbone, attach a head for your task, and fine-tune at a small LR.

6. The Hyperparameters That Matter Most

In rough order of impact:

1. Learning rate. 10×–100× smaller than from-scratch. The most common LRs you'll see in published fine-tunes: 2e-5 for transformer NLP, 1e-4 to 1e-5 for vision CNNs/ViTs, 1e-3 for the head-only stage.
2. Number of frozen layers. All-frozen is fastest and safest. Unfreezing the top half is a common middle ground. Unfreezing everything works when your dataset is large.
3. Batch size. Bigger batches stabilise gradients but reduce stochasticity. For fine-tuning, 16–64 is the typical range. It is small because you usually don't have enough data to justify more.
4. LR schedule. A linear warmup followed by linear decay (the "BERT schedule") is the standard choice for transformer fine-tuning. For vision, cosine decay is common.
5. Number of epochs. Fine-tuning converges quickly. 3–10 epochs is normal. If you're running 100 epochs you're either overfitting wildly or your LR is too small.

When transfer learning performs badly, the first thing to try is a lower learning rate. If that doesn't help, lower it again.

7. When Transfer Learning Fails

The pretraining task and your task need to share enough underlying structure for the features to transfer. Some examples:

• Works well: natural images → medical X-rays. Both are 2D continuous-tone images. Edges and textures are universal. Even the gap from "everyday photos" to "chest X-rays" is small enough that ImageNet weights consistently beat from-scratch.
• Mixed results: natural images → satellite imagery. Different scale, different colour statistics, different object priors. ImageNet weights help, but task-specific pretraining (e.g. on aerial imagery) helps more.
• Mixed results: natural images → spectrograms or scientific imaging where pixels don't represent "stuff with edges".
• Often fails: ImageNet → tabular data, time series, or graph data. There is no shared visual structure. Use a domain-appropriate pretrained model or skip transfer learning.
• NLP equivalent: English LLM → a low-resource language with a different script. Tokenisation fails first, and the meaning may never transfer. Use a multilingual model (mBERT, XLM-R) or continue pretraining on text from your domain.

Domain adaptation is the usual fix when the gap is too large: pretrain → intermediate pretrain on unlabelled in-domain data → task fine-tune. Standard for medical NLP (BioBERT, ClinicalBERT) and chemistry models.

8. Why Even Big Teams Use It

A common belief is that transfer learning is only for small-data startups. In practice the biggest labs still start from pretrained checkpoints, for two reasons:

• Compute economics. A from-scratch run of a serious vision or language model is hundreds of GPU-days. A fine-tune takes hours. Even when the budget allows a from-scratch run, a fine-tuned model is ready sooner.
• Convergence quality. Starting from a good initialisation lets you train at lower LRs without getting stuck in poor local minima. Final test accuracy is often higher than from-scratch even when you have plenty of data.

The only places people still do serious from-scratch training are (a) the foundation-model labs whose product is the pretrained checkpoint, and (b) tasks where no suitable pretrained model exists.

Common Mistakes

1. Same learning rate as from-scratch.

The pretrained weights are close to a good solution, and large updates push them away from it. Drop the LR by 10×–100× when fine-tuning. If your loss jumps up in the first epoch after unfreezing, this is the likely cause.

2. Forgetting training=False on the frozen backbone.

BatchNorm layers have two states: training (uses batch statistics, updates running averages) and inference (uses the running averages). In Keras, setting trainable=False on a BN layer also switches it to inference mode, so its running averages stop updating. The problem comes when you unfreeze for fine-tuning: BN would go back to training mode unless the base was called with training=False. The fix is either base(x, training=False) as shown above, or base.trainable = False plus keeping training=False even after unfreezing the rest. If you skip this and train with small batches, the BN statistics can be ruined, and the model with them.

3. Wrong preprocessing.

Each pretrained model expects specific input scaling: [-1, 1] for MobileNet, [0, 1] for some ResNets, ImageNet-normalised for ViTs. Use the bundled preprocess_input rather than guessing. The same applies in NLP: always use the tokenizer that comes with the model.

4. Picking a pretrained model from a very different domain.

Fine-tuning a clip-art model on dermatology images, or an English BERT on Tamil, or a music model on speech: when the gap is too large, transfer learning does worse than training from scratch. Take five minutes to check whether a domain-specific checkpoint exists before committing.

5. Skipping the head-only warmup.

Unfreezing the backbone before the new head has stabilised sends very large first-batch gradients backwards through the pretrained layers, undoing the pretraining. Always train the head alone for a few epochs first.

6. Mismatched label encoding.

If your binary classifier outputs probability of "dog" but you train on a dataset where label 0 is dog and label 1 is cat, you're optimising the inverse problem. Check label encoding before you start a long fine-tune run.

🎯 Your Turn: Fine-Tune MobileNetV2 on a Two-Class Task

Build the full transfer-learning workflow on a small binary image classification task (the cats-vs-dogs subset from tensorflow_datasets is a good choice, but any two-class folder of images works).

Requirements:

• Use keras.applications.MobileNetV2(weights="imagenet", include_top=False).
• Add a GlobalAveragePooling2D → Dropout(0.2) → Dense(1, sigmoid) head.
• Stage 1: freeze the backbone, train the head for 5 epochs at LR 1e-3.
• Stage 2: unfreeze the top half of the backbone, recompile at LR 1e-5, train 5 more epochs.
• Use the model's preprocess_input and pass training=False to the frozen backbone.
• Print final validation accuracy.

Skeleton:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

IMG_SIZE = (160, 160)

# TODO 1: load train_ds, val_ds (any tf.data pipeline producing (image, label) pairs)

# TODO 2: build the backbone with include_top=False, weights="imagenet"
#         freeze it

# TODO 3: stack the new head: preprocess_input, base(x, training=False),
#         GlobalAveragePooling2D, Dropout, Dense(1, sigmoid)

# TODO 4: compile with Adam(1e-3), binary_crossentropy; fit 5 epochs

# TODO 5: unfreeze the top half of base.layers; freeze BN layers explicitly
# TODO 6: recompile with Adam(1e-5); fit 5 more epochs

# TODO 7: print final val accuracy

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow_datasets as tfds

IMG_SIZE = (160, 160)
BATCH = 32

# 1. Load cats_vs_dogs from TFDS; resize and batch
(train_raw, val_raw), info = tfds.load(
    "cats_vs_dogs", split=["train[:80%]", "train[80%:]"],
    as_supervised=True, with_info=True,
)
def resize(image, label):
    return tf.image.resize(image, IMG_SIZE), label
train_ds = train_raw.map(resize).batch(BATCH).prefetch(tf.data.AUTOTUNE)
val_ds = val_raw.map(resize).batch(BATCH).prefetch(tf.data.AUTOTUNE)

# 2. Backbone, frozen
base = keras.applications.MobileNetV2(
    input_shape=IMG_SIZE + (3,),
    include_top=False,
    weights="imagenet",
)
base.trainable = False

# 3. Head
inputs = keras.Input(shape=IMG_SIZE + (3,))
x = keras.applications.mobilenet_v2.preprocess_input(inputs)
x = base(x, training=False)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)

# 4. Stage 1: train the head only
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=5)

# 5. Unfreeze top half, keep BN frozen
base.trainable = True
cutoff = len(base.layers) // 2
for layer in base.layers[:cutoff]:
    layer.trainable = False
for layer in base.layers:
    if isinstance(layer, layers.BatchNormalization):
        layer.trainable = False

# 6. Stage 2: fine-tune
model.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
history = model.fit(train_ds, validation_data=val_ds, epochs=5)

print(f"Final val accuracy: {history.history['val_accuracy'][-1]:.4f}")

What You Learned

• A deep network learns a hierarchy of features: early layers are generic, late layers are task-specific. Transfer learning reuses the generic part.
• Feature extraction freezes the backbone and trains only a new head. Fine-tuning then unfreezes some/all layers and trains end-to-end at a tiny LR.
• The standard workflow is head warmup → unfreeze → fine-tune. Skipping the warmup can destroy the pretrained weights.
• Fine-tuning LRs are 10×–100× smaller than from-scratch. The most common values: 2e-5 for transformer NLP, 1e-4–1e-5 for vision.
• Always use the model's own preprocess_input (or its tokenizer). Wrong input scaling breaks the model without any error message.
• Keep BatchNorm in inference mode when fine-tuning with small batches. Pass training=False or set the BN layers trainable=False explicitly.
• Transfer learning fails when the source and target domains share too little structure. Pick a pretrained model from a related domain, or domain-adapt with intermediate pretraining.
• Even teams with abundant data start from pretrained checkpoints, because training converges faster and usually scores higher.

Next: Natural Language Processing, which applies transfer learning across the full NLP task taxonomy, from classification to retrieval, using the Hugging Face ecosystem.