Convolutions

1.Images break fully-connected layers

A fully-connected (FC) layer wires every input to every output. That is fine for a 100-feature tabular row. It falls apart on an image, for two separate reasons.

Reason one: the parameter count explodes. Flatten a $128\times128\times3$ image and you get $49{,}152$ inputs. Connect those to a modest layer of 256 neurons and the weight count is

$$128 \times 128 \times 3 \times 256 = 12{,}582{,}912 \text{ weights}.$$

That is one layer. The full handout number to remember: a $128\times128\times3$ image into 256 FC neurons gives 12,582,912 weights, which overfits small datasets and costs a fortune to train. A $224\times224\times3$ image into 1000 neurons would need over 150 million.

Reason two: flattening throws away geometry. Once you flatten, the network has no idea that two pixels were neighbours. It has to relearn, from scratch, that adjacent pixels relate to each other, and it has to relearn that fact separately for every position in the image.

2.The convolution is slide, multiply, sum

Take a small grid of weights, the filter (or kernel), say $3\times3$. Place it over the top-left corner of the image. Multiply each filter weight by the pixel underneath it, add up all 9 products, and write that single number into the output. Then slide the filter one step to the right and repeat. When you reach the edge, drop down a row and start again.

Formally, for filter $K$ over an input patch, each output value is

$$y_{i,j} = \sum_{m}\sum_{n} K_{m,n}\,\cdot\,x_{\,i+m,\,j+n}.$$

That inner double sum is just a dot product between the filter and the patch of image it currently covers. A convolution is a dot product, repeated at every location.

A note on naming. What deep-learning libraries call "convolution" is technically cross-correlation: the real mathematical convolution first flips the kernel. The flip does not matter for learning, because the network just learns the weights in whatever orientation it needs. As Goodfellow's text puts it, learning with cross-correlation instead of true convolution simply gives you a flipped kernel, and the result is identical. (Deep Learning, ch. 9.)

Different filters detect different things. Below you can step a filter across a $7\times7$ input and watch the output feature map fill in, one position at a time. Try the preset edge, blur, and sharpen kernels.

3.One formula sets every output size

How big is the output? Count how many positions the filter can occupy as it slides. With an input of width $W$, a filter of size $F$, padding $P$ on each side, and stride $S$:

$$\text{out} = \left\lfloor \frac{W - F + 2P}{S} \right\rfloor + 1.$$

Read it from the pieces. The filter's left edge can start at position 0 and slide until its right edge hits the (padded) boundary, which leaves $W - F + 2P$ pixels of travel. Divide by the stride to count the steps, and add 1 for the starting position. The same formula runs independently for height. (Source: Stanford CS231n.)

One trap: the numerator must be divisible by the stride, or the filter does not tile the input cleanly. CS231n's example: $W=10,\,F=3,\,P=0,\,S=2$ gives $(10-3)/2 + 1 = 4.5$, which is not a whole number, so that configuration is invalid. The floor in the formula is what frameworks use to round it down, but you should treat a non-integer result as a sign your hyperparameters do not fit.

4.Padding and stride are your two dials

Without padding, every convolution shrinks the image. A $3\times3$ filter on a $7\times7$ input (stride 1) gives $(7-3)/1 + 1 = 5$, so you lose a ring of pixels. Stack a few layers and the picture vanishes. The fix is zero-padding: ring the input with zeros so the filter can sit over the border pixels too.

The padding that keeps the output the same size as the input is called "same" padding. For stride 1, set

$$P = \frac{F - 1}{2}.$$

For a $3\times3$ filter that is $P=1$; for $5\times5$ it is $P=2$. Check it: $7\times7$ input, $3\times3$ filter, $P=1$, stride 1 gives $(7-3+2)/1 + 1 = 7$. Same size in, same size out. (Source: CS231n.)

Stride is how far the filter jumps each step. Stride 1 looks at every position; stride 2 skips every other one, roughly halving each output dimension and downsampling the feature map. Larger strides mean a smaller, cheaper output but coarser spatial detail.

5.Parameter sharing and the receptive field

Here is the move that makes convolutions cheap. The same filter slides over the whole image, so a $3\times3$ filter has exactly 9 weights (plus 1 bias) no matter whether the image is $32\times32$ or $4000\times4000$. This is parameter sharing: one set of weights, reused at every location.

Why is sharing legitimate for images but not for tabular data? A vertical edge is a vertical edge wherever it sits, so a detector that works in the top-left works in the bottom-right too. Reusing weights across positions is exactly right. In a tabular row, feature 1 might be "age" and feature 5 "income"; those have different meanings, so forcing them to share a weight would be nonsense. Sharing only makes sense when the same pattern can appear in many places, which is the defining property of image data.

Parameter sharing also buys a property called translation equivariance: shift the input, and the feature map shifts the same way. The detector follows the pattern around the image (Goodfellow et al., 2016).

The receptive field is the region of the original input that one output value depends on. A single $3\times3$ conv has a $3\times3$ receptive field. Stack two such layers and the second layer's output depends on a $5\times5$ patch of the input; stack three and it is $7\times7$. The handout's formula:

$$\text{RF} = 1 + L\,(F - 1),$$

where $L$ is the number of stacked layers and $F$ the filter size. This is the trick behind VGG: two stacked $3\times3$ convs cover the same $5\times5$ receptive field as one $5\times5$ conv, but use only $2\times9 = 18$ weights per channel pair instead of 25, a 28% saving, and add an extra nonlinearity in between (on VGG's 3x3 design).

6.In NumPy: convolution from scratch

The whole operation is a few lines. Compute the output size with the formula, then for each output position take the dot product of the filter with the patch underneath it. This matches the post-lecture exercise: implement conv2d with NumPy only and check it against scipy.signal.correlate2d.

import numpy as np

def conv2d(image, kernel, stride=1, pad=0):
    # zero-pad the border so the filter can reach edge pixels
    if pad:
        image = np.pad(image, pad, mode='constant')
    H, W = image.shape
    F = kernel.shape[0]
    # output size: floor((W - F + 2P)/S) + 1, here pad already applied
    out_h = (H - F) // stride + 1
    out_w = (W - F) // stride + 1
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            r, c = i * stride, j * stride
            patch = image[r:r+F, c:c+F]      # the F x F window
            out[i, j] = np.sum(patch * kernel)  # element-wise multiply, then sum
    return out

# vertical-edge (Sobel-style) filter
K = np.array([[1, 0, -1],
              [2, 0, -2],
              [1, 0, -1]])
img = np.eye(7)                  # a diagonal pattern
print(conv2d(img, K).shape)      # -> (5, 5):  (7 - 3)//1 + 1 = 5

The np.sum(patch * kernel) line is the dot product from section 2. Everything else is bookkeeping for where the window sits. Swap in a different kernel and the same loop detects a different pattern.

7.Quick check

Four questions, rising difficulty. Answer, hit Check, read the why. The full bank lives in the Exam Center.