Transfer Learning & Applications

1.The problem: too many knobs, too few examples

Here is the trap, in numbers. ResNet-18 has about 11.7 million parameters. If you train all of them from scratch on a few hundred images, the model has far more free knobs than facts to constrain them. It fits the training set perfectly and the test set badly. That is textbook overfitting, and no amount of dropout fully saves you when the parameter-to-example ratio is that lopsided.

But notice something. The reason ImageNet models took weeks to train was not the classifier on top. It was learning good features: the convolutional filters that turn raw pixels into useful descriptions. Those filters are not specific to ImageNet's 1000 classes. An edge is an edge whether it borders a cat or a tumour.

2.Early layers are generic, late layers are specific

Why can the body transfer at all? Because of what the layers learn, in order. Yosinski and colleagues studied this directly in 2014. The first convolutional layer of almost any vision network, no matter the dataset, converges to the same kind of filters: oriented edge detectors and colour blobs, close cousins of Gabor filters in the visual cortex. These are general. Move deeper and the filters combine edges into textures, textures into parts, parts into object-like patterns. By the last layers, the features are tuned to separate the exact classes the model was trained on. Those are specific.

So transferability falls off with depth. An edge detector helps any vision task. A "this looks like an ImageNet terrier" detector helps only tasks that care about terriers. The same paper found a second, less obvious effect: layers in the middle are co-adapted, meaning a layer's features depend on the precise features below it. Cut the network in the middle and the halves no longer fit, which is its own source of trouble when you freeze part-way. The clean takeaway holds: freeze the generic early layers, and reconsider the specific late ones for your task.

3.Two strategies: feature extraction and fine-tuning

That split gives you exactly two moves, plus a dial between them.

Feature extraction. Freeze the entire pretrained body. Forward-pass your images through it to get feature vectors, then train only a fresh classifier head on top. In code this is one line of intent: set $\text{requires\_grad} = \text{False}$ on every backbone parameter so no gradients are computed for them in the backward pass. The body still runs forward and still produces features. It just never updates. Because gradients skip most of the network, this also trains in roughly half the wall-clock time.

Fine-tuning. Unfreeze some of the later layers and let them keep training, alongside the new head. You are not starting those layers from random weights, you are nudging already-good weights toward your task. This adapts the specific features when your task differs from ImageNet, but it needs more data to be safe, because you have unlocked more parameters.

4.The 2x2 rule: dataset size by domain similarity

How do you choose? CS231n's guidance reduces to two questions. Is your dataset small or large? Is your domain similar to or different from ImageNet's natural photos? That makes four cells.

Start with the diagonal. Small and similar, the handout's 300 flowers for 5 species, calls for feature extraction: 300 images is too little to fine-tune 11.7 million weights, and flowers are close enough to ImageNet that the frozen features already work. Large and different lets you fine-tune aggressively because the data can support it.

The off-diagonal "small and different" cell is the genuinely awkward one, an X-ray model built from a photo backbone. Frozen photo features are not ideal for X-rays, but you lack the data to fine-tune safely. The trick is to pull features from an earlier, more generic layer, where photos and X-rays still share edges, and accept that a frozen pretrained model on a far domain may simply underperform. When a fine-tuned ImageNet model does poorly on X-rays, the phenomenon is called domain shift, and the simplest fix is to fine-tune on even a small labelled set from the target domain.

5.See the trade-off: freeze or fine-tune

The simulator below is a pretrained backbone drawn as a vertical stack, generic early layers at the bottom, task-specific layers and the new head at the top. Click a layer to lock or unlock it. Locked layers are frozen, unlocked layers train. Set the dataset size and the domain similarity, then read the recommendation, the count of trainable parameters, and the overfitting-risk note. Try to make the strategy panel agree with the 2x2 table above.

Notice the tension the readout makes concrete. Unlocking more layers adds millions of trainable parameters. With a small dataset that pushes the overfitting risk straight to high, which is why the recommendation flips to feature extraction. The numbers below the canvas are the whole argument of this lesson in one panel.

6.In NumPy: a frozen backbone and a trained head

You will not implement ResNet by hand, but the mechanics of freezing are short enough to see exactly. The point of "freeze" is that frozen layers run forward and produce features, but their weights get no gradient and never update. Only the head learns. Here is the loop, stripped to its bones.

import numpy as np

# A "pretrained" backbone: fixed weights we will NOT update.
# Maps a flattened image x (D-dim) to a feature vector (F-dim).
rng = np.random.default_rng(0)
W_backbone = rng.standard_normal((D, F)) * 0.05   # frozen: requires_grad = False

def backbone(x):
    # forward pass through frozen body; ReLU features
    return np.maximum(0.0, x @ W_backbone)        # shape (N, F)

# A fresh classifier head we DO train (F features -> C classes).
W_head = np.zeros((F, C))
b_head = np.zeros(C)

def softmax(z):
    z = z - z.max(axis=1, keepdims=True)
    e = np.exp(z); return e / e.sum(axis=1, keepdims=True)

def train_head(X, y_onehot, lr=0.1, epochs=200):
    global W_head, b_head
    feats = backbone(X)                            # backbone is FROZEN: computed once
    N = X.shape[0]
    for _ in range(epochs):
        probs = softmax(feats @ W_head + b_head)   # forward through head
        dz = (probs - y_onehot) / N                # softmax + cross-entropy gradient
        # gradients ONLY for the head; the backbone gets none
        W_head -= lr * (feats.T @ dz)
        b_head -= lr * dz.sum(axis=0)
    return W_head, b_head

Two things to read off the code. First, because the backbone is frozen, feats is computed once outside the loop, never recomputed, never updated: that is the speedup feature extraction buys you. Second, only W_head and b_head receive a gradient. For a 512-dimensional feature vector and 5 flower classes, the head has $512 \times 5 + 5 = 2565$ trainable parameters, against 11.7 million if you fine-tuned everything. That gap is why 300 images is plenty for the head and nowhere near enough for the full network.

7.Quick check

Four questions, rising difficulty. Answer, hit Check, read the why. The full bank lives in the Exam Center.