← All 50 books
Book 12 of 50 · Free
Convolutional Neural Networks for Images
29,666 words · 20 chapters · illustrated
Convolutional Neural Networks for Images
Book 12 of 50 — AstolixGen Learning Series
About This Book
This book is written for MS and PhD students, and for early-career AI researchers who want a working understanding of convolutional neural networks — not just the vocabulary, but the machinery, the training craft, and the habits that turn CNN experiments into publishable results.
CNNs were the architecture that turned deep learning from a research curiosity into the dominant approach for image understanding. They powered the ImageNet revolution of 2012, enabled medical imaging breakthroughs, and remain the default workhorse for image classification, detection, and segmentation even in an era of transformers and vision-language models. If your thesis touches images at all — radiology scans, satellite tiles, crop photographs, microscope slides — you need this material.
We build from first principles. Chapter 1 answers why convolutions exist instead of plain fully connected layers. Chapters 2–3 give you the mechanics: the convolution operation with worked numeric examples, plus stride, padding, pooling, and receptive fields. Chapters 4–6 walk through the canonical architectures — LeNet, AlexNet, VGG, ResNet, MobileNet, EfficientNet — not as museum pieces but as design lessons you can reuse. Chapters 7–9 cover the practical core: transfer learning, data augmentation, and how to actually train a CNN without wasting weeks of GPU time. Chapter 10 shows you how to see inside your network. Chapter 11 adapts the whole recipe to three research domains: medical imaging, agriculture, and satellite imagery. Chapter 12 closes the loop from experiments to a paper: ablations, tables, and reporting standards.
Every chapter contains working PyTorch code. Run it. Break it. The examples are designed to be runnable on a laptop CPU or a single modest GPU (an 8 GB card is plenty for everything here). We assume you know Python and the basics of neural networks — what a loss function is, what backpropagation does at a high level. We do not assume any image processing background.
How to use this book: Read Chapters 1–5 in order; they build on each other. Chapters 7–12 are largely self-contained and can be read as needed. Do every exercise that involves writing code — the gap between reading about a convolution and implementing one is where real understanding lives.
Learning Objectives
By the end of this book, you will be able to:
- Explain, with parameter counts, why convolution beats fully connected layers for images — and quantify the savings for a concrete architecture.
- Compute the output of a convolution by hand for small examples, including stride, padding, and multi-channel inputs, and predict output tensor shapes with the standard sizing formulas.
- Describe what stride, padding, and pooling do to spatial dimensions and receptive fields, and choose them deliberately when designing an architecture.
- Trace the design lineage from LeNet through AlexNet, VGG, and ResNet, and explain what problem each architecture solved and what it taught the field.
- Explain skip connections mechanistically — why they fix the vanishing-gradient problem and why residual learning is easier to optimize.
- Select efficient architectures (MobileNet, EfficientNet) for compute-limited settings and justify the choice in terms of FLOPs, parameters, and latency.
- Apply transfer learning correctly: choose between feature extraction and full fine-tuning, set layer-wise learning rates, and avoid the common failure modes (catastrophic forgetting, negative transfer).
- Design a data augmentation pipeline for an image dataset, choosing augmentations that respect the data's invariances and avoiding ones that destroy label semantics.
- Train a CNN to convergence: pick optimizers, learning-rate schedules, and regularization; diagnose underfitting, overfitting, and optimization failures from training curves.
- Visualize what a trained CNN has learned (first-layer filters, activation maps, Grad-CAM), and use those visualizations to debug models and strengthen a paper's figures.
Chapter 1 — Why Convolutions for Images
The problem with flattening an image
Imagine you have a 256×256 RGB photograph and you want to classify it with a plain feedforward network. The naive move is to flatten the image into a vector: 256 × 256 × 3 = 196,608 input numbers. Now connect that to a hidden layer of, say, 1,024 neurons. The weight matrix alone holds 196,608 × 1,024 ≈ 201 million parameters. That is a staggering amount of machinery for a single layer — more parameters than many complete modern networks — and it buys you almost nothing, because the structure of the problem has been thrown away.
Flattening destroys two pieces of structure that images have and generic vectors don't:
- Locality. Pixels near each other are strongly related; pixels far apart usually are not. An edge, a texture, a corner — all the visual primitives — are defined by small neighborhoods. A fully connected layer treats pixel (0, 0) and pixel (200, 150) as equally related to pixel (0, 1), which is false for natural images.
- Stationarity (roughly). The statistics of useful visual features don't depend much on where they appear. A horizontal edge detector that works in the top-left corner of an image also works in the bottom-right. Your visual cortex doesn't learn separate edge detectors for every retinal position, and your network shouldn't either.
A fully connected layer learns no such thing: it must independently learn, at every position, whatever pattern matters at that position. That is wasteful in parameters, slow to train, data-hungry, and prone to overfitting — because with 201 million free parameters, memorizing the training set is always the easiest path.
The convolution solves this with two ideas that go together: local connectivity and parameter sharing.
Local connectivity
Instead of connecting every neuron to every pixel, a convolutional neuron connects to a small patch — say 3×3 pixels — of the input (or of the previous layer's output). This is called the receptive field of the neuron, and the idea has a direct biological precedent: Hubel and Wiesel's work on the visual cortex showed that individual neurons in area V1 respond to stimuli in small, localized regions of the visual field. The Neocognitron (Fukushima, 1980), the direct ancestor of modern CNNs, was explicitly modeled on this.
Why is local connectivity safe? Because in images, the informative relationships are local. A neuron trying to detect an eye in a face doesn't need to see the whole image; it needs to see the eye region. Higher layers, which see combinations of lower-layer outputs, get their larger effective receptive fields automatically — the hierarchy builds itself.
Parameter sharing (weight tying)
Here's the bigger idea. In a standard layer, each neuron has its own private set of weights. In a convolutional layer, the same small weight matrix — the kernel or filter — is reused at every spatial position. One 3×3×3 kernel (27 weights plus a bias) slides across the whole image, computing a response at each location. The resulting two-dimensional map of responses is called a feature map (or activation map).
The consequence is a feature detector that is translation equivariant: if you shift the input pattern a few pixels, the feature map shifts by the same amount. The detector doesn't care where the pattern is — it finds it anywhere. (Strictly, pooling later gives approximate invariance, which we discuss in Chapter 3; equivariance from convolution and invariance from pooling are related but distinct, and a paper reviewer will notice if you confuse them.)
Counting the savings — a worked example
Let's make this concrete. Take a 32×32×3 image (CIFAR-10 size) feeding a first hidden layer that produces 64 output channels, conceptually arranged as a 32×32×64 volume (same spatial size for now; padding handles this, per Chapter 3).
Fully connected approach: each of the 32 × 32 × 64 = 65,536 output units connects to all 3,072 input pixels: 65,536 × 3,072 ≈ 201 million parameters.
Convolutional approach: each output channel is produced by one 3×3×3 kernel shared across all positions: 64 kernels × (3 × 3 × 3 + 1 bias) = 64 × 28 = 1,792 parameters.
Same output volume shape. Two hundred million parameters versus under two thousand. That ratio — five orders of magnitude — is why CNNs train at all. The convolutional layer isn't just more efficient; it encodes a correct prior about the data (locality + stationarity), which acts as a powerful regularizer. Fewer parameters means less capacity to memorize noise, which is exactly what you want when your dataset has 50,000 images and not 50 million.
Of course, the fully connected layer could in principle learn the same function — a convolutional layer is a constrained special case of a fully connected one (a sparse weight matrix with tied weights). The constraint is the point. As with all of machine learning, the win comes from matching the model's inductive bias to the problem's structure.
The hierarchy: what stacking buys you
A single convolutional layer detects simple local patterns: edges, color blobs, oriented bars. Stack a second convolutional layer on top, and its neurons combine those primitives into textures and simple shapes. A third layer combines shapes into parts — an eye, a wheel, a leaf vein. By layer ten or twenty, neurons respond to whole objects and scenes. This compositional hierarchy is the second great idea of CNNs, and it is why depth matters: each layer reuses the abstractions of the one below.
Zeiler and Fergus's famous visualizations (2014) made this tangible — we'll reproduce the intuition in Chapter 10 — but the principle matters now: convolutional networks learn features, not just classifiers. The deep layers of a trained CNN are a general-purpose visual vocabulary, which is precisely why transfer learning (Chapter 7) works.
A small code taste
Here's the two approaches side by side in PyTorch, for the 32×32×3 → 64-channel example:
import torch
import torch.nn as nn
# Fully connected: 201M parameters, structure discarded
fc = nn.Linear(32 * 32 * 3, 32 * 32 * 64)
print(sum(p.numel() for p in fc.parameters())) # 201,334,784
# Convolutional: 1,792 parameters, structure preserved
conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
print(sum(p.numel() for p in conv.parameters())) # 1,792
Run that. The numbers are the argument. Everything else in this book is about what you do with this absurd efficiency advantage.
For your research. If your thesis involves any image data, your baseline should be a CNN — even a simple one — before you reach for anything exotic. Reviewers will ask "did you try a standard ResNet baseline?" and if the answer is no, nothing else you report will land. The parameter-counting argument above is also directly reusable: in your paper's methods section, one sentence explaining why convolution suits your data (local structure, translation statistics) preempts the "why this architecture?" question. Keep the bias-vocabulary straight: convolutions give equivariance, pooling gives (approximate) invariance.
Going deeper: biology, equivariance in code, and honest limits
The biological roots. In 1962, Hubel and Wiesel showed that neurons in a cat's visual cortex fire for edges at specific orientations within small regions of the visual field — the receptive fields this chapter borrowed as an idea. In 1980, Fukushima's Neocognitron stacked layers of such local feature detectors with a pooling-like operation between them, explicitly modeled on this biology, and recognized handwritten digits. LeCun's LeNet (Chapter 4) made the same idea trainable end-to-end with backpropagation. So when someone asks "why should local weight sharing work," the two-part answer is: (a) the statistics of natural images — local correlations, stationary features — and (b) it worked in the one vision system we know is excellent, the mammalian cortex. Nature ran the ablation study first.
Equivariance, demonstrated rather than asserted. Chapter 1 claimed convolution is translation equivariant: shift the input, and the feature map shifts by exactly the same amount. Let's verify:
import torch
import torch.nn.functional as F
img = torch.zeros(1, 1, 6, 6)
img[0, 0, 1, 1] = 10.0 # bright pixel near top-left
K = torch.tensor([[[[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]]]]) # 3x3 blur kernel
out1 = F.conv2d(img, K, padding=1)
shifted = torch.roll(img, shifts=(0, 2), dims=(2, 3)) # shift right by 2
out2 = F.conv2d(shifted, K, padding=1)
print(torch.allclose(out2, torch.roll(out1, shifts=(0, 2), dims=(2, 3)))) # True
It prints True. The output moved exactly as the input did — the detector found the pattern wherever it was, with no retraining. (Boundary effects from padding can break exactness at the edges, which is one more reason to understand your padding choice from Chapter 3.)
What convolution cannot do — honest limits for your thesis. Knowing the failure modes is as important as knowing the mechanism: 1. No rotation equivariance. Rotate the input 90 degrees and the feature maps do not rotate correspondingly through the same kernels — every orientation needs separately learned filters, or augmentation (Chapter 8), or specialized group-equivariant architectures. 2. Long-range dependencies are expensive. A neuron only sees its receptive field; relating distant regions requires depth. Tasks needing whole-image reasoning (counting, global layout) are why non-local blocks — and later vision transformers — appeared. 3. The grid assumption. Convolution assumes data on a regular grid. Point clouds, meshes, and graphs need different machinery (graph neural networks). Don't force image-style CNNs onto non-grid data. 4. Resolution coupling. Filter scales are tied to training resolution — a network trained on 224×224 inputs doesn't automatically handle 1024×1024 well. EfficientNet's compound scaling (Chapter 6) exists partly because of this.
Parameter sharing as regularization — the learning-theory view. With 201 million free parameters and 50,000 training images, a fully connected network can simply memorize the training labels (Zhang et al. demonstrated in 2017 that deep networks fit even random labels). Generalization then depends on fragile optimizer biases. Weight sharing cuts capacity by five orders of magnitude structurally, before optimization begins. You're not hoping the network generalizes; you've made memorization geometrically harder. When your examiner asks "why not just use an MLP with dropout," this is the answer: dropout regularizes a huge capacity, while convolution removes capacity you never needed.
Worked example: a realistic first layer. Input 224×224×3, first convolution with 64 kernels of 7×7, stride 2, padding 3 (ResNet's stem). Parameters: 64 × (7×7×3 + 1) = 64 × 148 = 9,472. Output spatial size: floor((224 − 7 + 6)/2) + 1 = 112. So a 112×112×64 volume — from nine thousand parameters. The fully connected equivalent for the same output volume: (112×112×64) × (224×224×3) ≈ 1.2×10¹¹ — 120 billion parameters in a single layer. Nobody trains that; everybody trains the 9,472-parameter version. Keep this example handy — it makes an excellent slide.
Convolution as structured matrix multiplication
There is a useful mental model that unifies this chapter with everything you know about fully connected layers: a convolutional layer is exactly a fully connected layer whose weight matrix is sparse and shares values. Take a 4×4 input (16 pixels) and a 3×3 kernel, stride 1, no padding → 2×2 output (4 values). Unroll the input to a 16-vector and the output to a 4-vector. The "weight matrix" is 4×16, but each row has only 9 nonzero entries (the kernel weights, in the right positions), and the same 9 numbers repeat in every row, shifted. Libraries implement convolution almost this way (the im2col transform rearranges image patches into columns, then it's one big matrix multiply — which is why GPUs love it).
Why does this view matter for research? Three reasons. First, it makes parameter counting trivially general: parameters = (nonzero weights per output) × (number of distinct kernels). Second, it explains why techniques transfer between layer types — dropout, weight decay, initialization schemes all operate on the underlying matrix. Third, it clarifies what "inductive bias" means mechanically: you're not adding a new kind of learning, you're deleting connections and tying weights. Every architectural innovation in this book — depthwise separable convolutions, grouped convolutions, bottlenecks — is a different pattern of deletion and tying. When you propose your own architecture, describe it in these terms ("we tie weights across X and sparsify along Y") and reviewers will immediately understand the claim.
Initialization deserves a paragraph. Random weights can't be arbitrary: too large and activations explode through deep stacks; too small and they vanish. He initialization (He et al., 2015) draws weights with variance 2/fan_in — derived specifically for ReLU networks, where half the activations are zeroed. PyTorch applies it by default in conv layers (kaiming_uniform_), which is why your networks train at all out of the box. If you ever write a custom layer with a non-ReLU activation, revisit the initialization: Xavier/Glorot (variance 1/fan_in) suits tanh/sigmoid. "We used default He initialization" is a complete methods sentence — but now you know what it claims.
When not to use a CNN (and the locally-connected middle ground)
Honesty requires the counter-cases. Skip CNNs (or demote them to a baseline) when: (a) your inputs aren't grids — tabular data, point clouds, graphs, sets — where MLPs, PointNets, or GNNs match the structure better; (b) order matters more than locality — long sequences and language, where transformers dominate; (c) you have < 200 images and no related pretrained model — a CNN will memorize; try classical features + a linear model, or generate data first; (d) the discriminative signal is global and non-local (e.g., "is this X-ray from scanner A or B" — often a global texture statistic a linear probe on pooled features captures better than deep hierarchies).
The middle ground: locally connected layers. Between fully connected (no sharing, dense) and convolutional (full sharing, sparse) sits the locally connected layer: each position gets its own kernel (no sharing) but only sees a local patch (sparse). DeepFace (2014) used these for face recognition — faces are roughly aligned, so position-specific features help. The cost is the parameter explosion of no sharing, which is why the idea mostly lost. Know it exists: when your data is aligned (registered medical images, fixed-camera industrial inspection), relaxing sharing in late layers is a legitimate, citable design choice — "we use locally connected layers in the final stage because inputs are registered" — but measure whether it beats plain convolution before committing.
Key takeaways - Flattening an image into a vector destroys locality and stationarity — the two structural facts that make images learnable. - Convolution = local connectivity (small receptive fields) + parameter sharing (one kernel reused everywhere). - The savings are enormous: ~201M parameters → ~1.8K for the same output volume in our example. - Stacked convolutions build a compositional hierarchy: edges → textures → parts → objects. - A convolutional layer is a constrained fully connected layer; the constraint is an inductive bias, and matching bias to data is the whole game.
Chapter 2 — The Convolution Operation, Explained Step by Step
What the operation actually computes
Forget neural networks for a moment. Take a small grayscale image — a 5×5 grid of pixel values — and a 3×3 kernel of weights. Place the kernel over the top-left 3×3 patch of the image. Multiply each kernel weight by the pixel underneath it, element by element. Add all nine products together. That single number becomes one pixel of the output. Then slide the kernel one step right and repeat; when you hit the edge, go down a row. The grid of output numbers is the feature map.
Formally, for input I and kernel K (both 2-D for now), the cross-correlation computed by deep learning libraries is:
output[i, j] = Σ_m Σ_n I[i + m, j + n] · K[m, n]
(A note for the careful: what deep learning calls "convolution" is technically cross-correlation — a true mathematical convolution flips the kernel first. Since the kernel weights are learned, the flip is irrelevant; the network learns whichever orientation it needs. Every library does cross-correlation and calls it convolution. Use the word "convolution"; know the footnote.)
Worked numeric example 1: edge detection
Consider this 5×5 input (think of it as a bright square on a dark background):
I = 0 0 0 0 0
0 10 10 10 0
0 10 10 10 0
0 10 10 10 0
0 0 0 0 0
And a vertical-edge kernel:
K = 1 0 -1
1 0 -1
1 0 -1
This kernel responds strongly where the left side of its window is bright and the right side is dark — i.e., a vertical edge with bright on the left.
Position (0, 0) (top-left patch):
patch = 0 0 0 K = 1 0 -1
0 10 10 1 0 -1
0 10 10 1 0 -1
Element-wise products: 0·1 + 0·0 + 0·(−1) + 0·1 + 10·0 + 10·(−1) + 0·1 + 10·0 + 10·(−1) = 0 + 0 + 0 + 0 + 0 − 10 + 0 + 0 − 10 = −20.
Position (0, 1) (shift right by one):
patch = 0 0 0
10 10 10
10 10 10
Products: 0 + 0 + 0 + 10 + 0 − 10 + 10 + 0 − 10 = 0. (No vertical edge here — the patch is uniform bright in the middle columns, and the kernel's left and right columns cancel.)
Position (0, 2) (shift right again):
patch = 0 0 0
10 10 0
10 10 0
Products: 0 + 0 + 0 + 10 + 0 + 0 + 10 + 0 + 0 = +20. Strong positive response: bright on the left, dark on the right — the right edge of the square.
Continuing across all nine valid positions gives the 3×3 output:
out = -20 0 20
-20 0 20
-20 0 20
Read it: the left column fires −20 (bright-to-dark edge in the reverse orientation — the left edge of the square, which is dark-to-bright left-to-right), the middle column is silent, and the right column fires +20 (the right edge). The kernel found exactly the vertical edges and ignored everything else. That is a feature map.
Now apply a ReLU (set negatives to zero) — standard after every convolution — and the negative responses vanish, leaving only the +20 edge detections:
relu(out) = 0 0 20
0 0 20
0 0 20
Worked numeric example 2: multiple channels and bias
Real inputs have channels (RGB = 3), and real layers have many kernels. The rule: a kernel spans all input channels, and each output channel gets its own kernel plus one bias.
Take a tiny 2×2×2 input (height 2, width 2, channels 2):
channel 0: 1 2 channel 1: 5 6
3 4 7 8
One 2×2×2 kernel (no sliding possible — output is 1×1), with weights:
kernel ch0: 1 0 kernel ch1: 0 1
0 -1 -1 0
bias = 10.
Compute per channel, then sum, then add bias:
- ch0: 1·1 + 2·0 + 3·0 + 4·(−1) = 1 − 4 = −3
- ch1: 5·0 + 6·1 + 7·(−1) + 8·0 = 6 − 7 = −1
- Sum + bias: −3 + (−1) + 10 = 6.
So the single output value is 6. With 64 kernels you'd get 64 such numbers — a 1×1×64 output volume. Notice the accounting: each output channel costs kernel_height × kernel_width × in_channels weights, plus one bias. This is the parameter formula you'll use constantly:
params per conv layer = out_channels × (kernel_h × kernel_w × in_channels + 1)
The operation in PyTorch, verified by hand
Let's check Example 1 in code — never trust a hand calculation you haven't run:
import torch
import torch.nn.functional as F
I = torch.tensor([[[[0., 0, 0, 0, 0],
[0, 10, 10, 10, 0],
[0, 10, 10, 10, 0],
[0, 10, 10, 10, 0],
[0, 0, 0, 0, 0]]]]) # shape: (1, 1, 5, 5)
K = torch.tensor([[[[1., 0, -1],
[1, 0, -1],
[1, 0, -1]]]]) # shape: (1, 1, 3, 3)
out = F.conv2d(I, K)
print(out.squeeze())
# tensor([[-20., 0., 20.],
# [-20., 0., 20.],
# [-20., 0., 20.]])
Matches the hand computation exactly. PyTorch tensor layout for conv2d is (batch, channels, height, width) — NCHW. Get comfortable with it; every shape bug you'll ever debug in a CNN is an NCHW misunderstanding.
What kernels learn (a preview)
In Example 1, we hand-designed an edge detector. In a real network, nobody designs kernels — they start random and gradient descent shapes them. The remarkable empirical finding (visualized in Chapter 10) is that first-layer kernels converge to edge detectors, color-blob detectors, and oriented gratings on their own. The network rediscovers classical image processing from data. Deeper layers learn combinations: a "wheel detector" is literally a weighted sum of curve and edge detectors from below, because that's all a convolution plus nonlinearity can express — and it turns out to be enough.
Complexity: how much work is one layer?
For an H×W×C_in input, K×K kernels, C_out output channels, the multiply-add count is roughly H × W × C_out × K² × C_in. For a 56×56×64 input with 128 3×3 kernels: 56 × 56 × 128 × 9 × 64 ≈ 231 million MACs. This is why 3×3 kernels dominate modern design (Chapter 6 will show how 1×1 kernels cut this further) and why you should always know your model's FLOPs before promising a paper result on a deadline — a model that takes six days per training run will not get ablated.
For your research. Hand-computing a convolution is the fastest way to debug shape errors: when your feature map comes out 27×27 instead of 28×28, you need the sizing formula (Chapter 3), not another print statement. More importantly, the multi-channel example shows why 1×1 convolutions are so powerful — a 1×1 kernel is just a per-pixel weighted sum across channels, i.e., a learnable channel mixer with no spatial extent. Network-in-Network (Lin et al., 2014) introduced them, and every modern architecture uses them to control channel counts cheaply. If a reviewer asks how you reduced dimensionality, "1×1 convolutions" is a complete answer.
More worked examples: padding by hand, 1×1 convolutions, dilation
Example 3: padding and stride, computed by hand. Take a 4×4 input:
I = 1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
Kernel: 3×3 of all ones, stride 2, padding 1. First the shape: floor((4 − 3 + 2)/2) + 1 = floor(3/2) + 1 = 2. Output is 2×2. Pad the input with a zero border (now 6×6):
0 0 0 0 0 0
0 1 2 3 4 0
0 5 6 7 8 0
0 9 10 11 12 0
0 13 14 15 16 0
0 0 0 0 0 0
Position (0,0) — kernel over rows 0–2, cols 0–2: 0+0+0+0+1+2+0+5+6 = 14. Position (0,1) — rows 0–2, cols 2–4: 0+0+0+2+3+4+6+7+8 = 30. Position (1,0) — rows 2–4, cols 0–2: 0+5+6+0+9+10+0+13+14 = 57. Position (1,1) — rows 2–4, cols 2–4: 6+7+8+10+11+12+14+15+16 = 99.
out = 14 30
57 99
Run F.conv2d(I, K, stride=2, padding=1) and compare — do this; the habit of verifying hand calculations in code is exactly what makes shape bugs tractable later. Notice what happened: padding let the kernel center on every original pixel including borders, while stride 2 still halved the size. This combination — pad for alignment, stride for downsampling — is precisely what ResNet's stem does.
1×1 convolution, numeric. Input 2×2×3:
ch0: 1 0 ch1: 0 2 ch2: 1 1
0 1 3 0 1 0
Two 1×1×3 kernels: k_a = [1, 0, −1], k_b = [0, 1, 1], biases 0. A 1×1 convolution computes, at each pixel independently, a dot product across channels. Pixel (0,0) has channels (1, 0, 1): k_a → 1·1 + 0·0 + 1·(−1) = 0; k_b → 1·0 + 0·1 + 1·1 = 1. Pixel (0,1), channels (0, 2, 1): k_a → −1; k_b → 3. Pixel (1,0), channels (0, 3, 1): k_a → −1; k_b → 4. Pixel (1,1), channels (1, 0, 0): k_a → 1; k_b → 0.
Output channel a: [[0, −1], [−1, 1]]; channel b: [[1, 3], [4, 0]]. Zero spatial mixing — pure per-pixel channel recombination. This is why 1×1 convolutions are the standard tool for changing channel counts: with a ReLU after, they add representational power at minimal cost, and they appear in every modern architecture (bottlenecks, projections, SE blocks).
Dilated (atrous) convolutions. Insert gaps into the kernel: dilation d=2 spreads a 3×3 kernel over a 5×5 area with no extra parameters. Effective kernel size: K_eff = K + (K−1)(d−1). The receptive field grows while resolution stays fixed — the trick behind DeepLab's dense prediction and WaveNet's audio modeling. Same parameter count as a normal 3×3, but stacking identical dilations causes gridding artifacts (checkerboard gaps in coverage), so dilations are usually increased gradually (1, 2, 4). In PyTorch: nn.Conv2d(64, 64, 3, padding=2, dilation=2). Reach for dilation when your task needs large context at full resolution — e.g., segmenting large structures in medical or satellite images without downsampling away the details.
Backpropagation through convolution (the intuition)
You don't need the full index gymnastics, but you need the shape story — because "how do gradients flow through my custom layer" is a question you'll face. Forward: input (B, C_in, H, W), kernel (C_out, C_in, K, K) → output (B, C_out, H', W'). Backward has two parts:
- Gradient w.r.t. the kernel (for learning): it's a convolution of the input with the upstream gradient — intuitively, "which input patterns coincided with large errors, and in which direction should the kernel change to fix them." Shape check: upstream grad (B, C_out, H', W') convolved with input (B, C_in, H, W) yields (C_out, C_in, K, K) — exactly the kernel's shape. ✓
- Gradient w.r.t. the input (to pass backward): it's a transposed convolution of the upstream gradient with the kernel — the error is "spread back" through the same connectivity. Shape: (B, C_in, H, W). ✓
The beautiful symmetry: forward and both backward passes are all convolutions, which is why frameworks implement one kernel and reuse it three ways. And notice the parameter-sharing payoff again: the kernel's gradient sums contributions from every spatial position — each training image gives the kernel H'×W' learning signals, not one. This dense supervision per parameter is a hidden reason CNNs are so sample-efficient compared to fully connected nets.
Grouped convolutions. Split C_in input channels into G groups; each group gets its own kernels producing C_out/G outputs; concatenate. Parameters and FLOPs drop by ~G×. ResNeXt (Chapter 5) uses G=32 for representational reasons; the original motivation was AlexNet's two-GPU split (Chapter 4). Depthwise convolution is the extreme: G = C_in. In PyTorch it's one argument: nn.Conv2d(64, 64, 3, padding=1, groups=8). Constraints: C_in and C_out must both be divisible by G. Use grouped convolutions when you want ResNeXt-style capacity at lower cost, or when channels have natural groupings (e.g., separate processing per spectral band group before fusion).
Grouped convolution, numeric — and the FLOP ledger
Grouped convolution by hand. Input: 4 channels, 2×2 spatial. Groups G=2 → group A sees channels 0–1, group B sees channels 2–3. Suppose 4 output channels (2 per group), kernel 1×1 for simplicity. Kernel weights: group A: k0=[1,0], k1=[0,1] (identity on its channels); group B: k2=[1,1], k3=[−1,1]. Pixel values across channels: (2, 3, 5, 7). Outputs: o0 = 2·1+3·0 = 2; o1 = 2·0+3·1 = 3; o2 = 5·1+7·1 = 12; o3 = 5·(−1)+7·1 = 2. No information crosses the group boundary — that's the point and the price. (ShuffleNet later added channel shuffling between grouped layers to restore cross-talk cheaply.)
The FLOP ledger habit. Before any architecture decision, fill one row: layer → params, MACs. For conv: params = C_out(C_in·K²+1); MACs ≈ H·W·C_out·C_in·K² (output spatial size). Sum the MACs; that's your per-image cost. Two uses: (1) predicting training time — MACs × 3 (forward + 2× backward) × dataset × epochs ÷ GPU throughput ≈ wall-clock; (2) catching design mistakes — if one layer holds 60% of FLOPs, it's your optimization target. Researchers who keep a FLOP ledger stop being surprised by slow experiments. fvcore or thop compute it automatically; do it by hand once so you trust the tool.
Key takeaways - Convolution = slide kernel, element-wise multiply, sum. Deep learning libraries compute cross-correlation and call it convolution; the distinction doesn't matter for learned weights. - One kernel spans all input channels; each output channel has its own kernel plus bias. - Output size without padding/stride: (H − K + 1) × (W − K + 1). - PyTorch uses NCHW layout: (batch, channels, height, width). Memorize it. - First-layer kernels learn edge/color detectors on their own; deeper layers compose them hierarchically.
Chapter 3 — Pooling, Stride, Padding, Receptive Fields
Stride: how far the kernel steps
So far the kernel moved one pixel at a time. The stride S is the step size. With stride 2, the kernel jumps two pixels between positions, producing an output roughly half the spatial size. Stride is the cheapest form of downsampling: you simply compute fewer positions.
The general output-size formula for one spatial dimension, given input size W, kernel K, padding P, stride S:
W_out = floor((W − K + 2P) / S) + 1
Check it against Chapter 2's example: W = 5, K = 3, P = 0, S = 1 → (5 − 3)/1 + 1 = 3. ✓. With stride 2 instead: floor((5 − 3)/2) + 1 = floor(1) + 1 = 2. Positions: kernel covers columns 0–2, then 2–4. Two outputs. ✓.
Memorize this formula. You will use it weekly.
Padding: keeping the edges (and the size)
Without padding, every convolution shrinks the map: a 5×5 input with a 3×3 kernel gives 3×3. Two problems: (a) deep stacks shrink to nothing — ten 3×3 layers on a 32×32 input would collapse the spatial dimensions entirely; (b) border pixels participate in fewer computations than center pixels, so the network systematically under-uses edge information (bad when your object touches the image border, as in many medical scans).
Padding adds a border of (usually zero) pixels around the input before convolving. The standard choice is "same" padding: P = (K − 1)/2 for odd K, which with stride 1 preserves the spatial size. For K = 3, P = 1: (32 − 3 + 2)/1 + 1 = 32. ✓.
conv = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
x = torch.randn(1, 3, 32, 32)
print(conv(x).shape) # torch.Size([1, 64, 32, 32]) -- size preserved
Padding types matter more than beginners expect. Zero padding is the default and mostly harmless, but it injects a fake border of zeros that the network can learn to detect — CNNs are known to exploit padding artifacts as positional cues. Reflection padding mirrors the edge pixels and avoids the artificial zero border; some segmentation work prefers it. For a paper, "we used zero padding" needs no justification; anything else deserves one sentence.
Pooling: downsampling with (approximate) invariance
A pooling layer slides a window over each feature map independently (no cross-channel mixing, no learned weights) and summarizes each window with one number. Max pooling takes the maximum; average pooling takes the mean.
Why max pooling works: if a feature (say, a vertical edge) appears anywhere inside the 2×2 window, the max operation reports it. Shift the input by one pixel and the max is usually unchanged. This is where translation invariance comes from — small shifts in the input produce (approximately) identical pooled outputs. (It's approximate: a shift that moves the feature across a window boundary changes the result. Truly invariant pooling doesn't exist; "approximately invariant" is the honest phrase.)
The standard recipe: 2×2 max pooling with stride 2 halves each spatial dimension:
pool = nn.MaxPool2d(kernel_size=2, stride=2)
x = torch.randn(1, 64, 32, 32)
print(pool(x).shape) # torch.Size([1, 64, 16, 16])
Pooling has no parameters — it can't overfit — and it aggressively reduces computation for later layers. Its deeper purpose is building the hierarchy: after pooling, the next convolution's 3×3 window covers a 6×6 region of the original input. Downsampling grows the effective receptive field, which is how deep layers see whole objects.
Modern trend worth knowing: many architectures (ResNet onwards) replaced most pooling with strided convolutions — a convolution with stride 2 downsamples and learns the downsampling filter instead of using a fixed max. The intuition: why hand-design the downsampling when the network can learn it? Max pooling survives mainly right after the input stem and as global average pooling before the classifier (Chapter 5).
Global average pooling: the classifier's front door
Before the final fully connected layer, modern CNNs collapse each feature map to a single number by averaging over all spatial positions — global average pooling (GAP). A 7×7×512 volume becomes a 512-vector. Two wins: (a) it enforces a correspondence between feature maps and categories (each map becomes a confidence map for some visual concept), and (b) it has zero parameters, unlike a fully connected layer on the flattened volume, which would add millions. Lin et al.'s Network-in-Network introduced it; ResNet made it standard. If you're adapting a classifier backbone for a new task, GAP output is the feature vector you want (Chapter 7).
Receptive fields: what each neuron actually sees
The receptive field of a neuron is the region of the input image that can influence its value. For a single 3×3 convolution, it's 3×3. Stack another 3×3 on top: each neuron in layer 2 sees 3×3 of layer 1, and each of those saw 3×3 of the input — so layer 2's receptive field is 5×5. The recurrence, with kernel sizes k_i and strides s_i:
RF_l = RF_{l−1} + (k_l − 1) × Π_{i<l} s_i, with RF_0 = 1.
Worked example — three 3×3 convs (stride 1), then a 2×2 stride-2 pool, then a 3×3 conv:
- After conv1: RF = 1 + (3−1)×1 = 3
- After conv2: RF = 3 + 2×1 = 5
- After conv3: RF = 5 + 2×1 = 7
- After pool (k=2, s=2): RF = 7 + (2−1)×1 = 8; cumulative stride now 2
- After conv4: RF = 8 + (3−1)×2 = 12
So a layer-4 neuron sees a 12×12 patch of the input. Two subtleties researchers should know:
- The effective receptive field is smaller than the theoretical one — and Gaussian-shaped. Luo et al. (2016) showed that center pixels of the receptive field influence the output far more than edge pixels, because there are exponentially more paths from the center. A neuron with a theoretical RF of 200×200 may effectively use only the central 50×50. If your task needs genuine long-range context (e.g., relating a tumor to a distant anatomical landmark), you may need dilated convolutions or attention — don't assume a deep stack sees the whole image uniformly.
- Receptive field arithmetic is how you sanity-check architecture choices: if your objects are 100 pixels wide but your classifier's RF is 40, the network cannot be using whole-object shape — it's classifying from texture, which may or may not be what you want (and is a great discussion point in a paper).
For your research. Receptive field analysis belongs in your methods section when spatial context matters: "the final feature map has a theoretical receptive field of 212×212 on 256×256 inputs, covering typical lesion sizes (30–80 px) with margin." Reviewers in medical imaging and remote sensing respect this calculation — it shows you designed rather than guessed. And remember the equivariance/invariance distinction from Chapter 1: if your application needs rotation invariance (e.g., cells under a microscope have no canonical orientation), standard CNNs don't provide it — that's what your augmentation strategy (Chapter 8) must supply.
Upsampling, anti-aliasing, and shape tracing
Transposed convolution. Segmentation and generative models need to go from coarse feature maps back to full resolution. A transposed convolution learns an upsampling: roughly, output = (W−1)·S − 2P + K, so stride 2 doubles the spatial size. Despite the old name "deconvolution," it is not an inverse of convolution — it's a convolution with the forward and backward passes swapped, and its weights are learned like any other's. Practical warning: transposed convolutions can produce checkerboard artifacts — visible grid textures from uneven kernel overlap. The common fix is "resize-convolution": nearest-neighbor upsampling followed by a regular convolution. Uglier in theory, cleaner in practice. In U-Net-style decoders (Chapter 11) either works; if your segmentation masks show grid artifacts, check the upsampling first.
Anti-aliased pooling (BlurPool). Strided max pooling downsamples without low-pass filtering, which violates the Nyquist sampling theorem and causes aliasing — tiny input shifts produce disproportionate output changes, quietly damaging the translation invariance pooling is supposed to provide. Zhang (2019) showed that inserting a small blur filter before strided downsampling improves both accuracy and shift-stability across architectures. Most libraries don't do this by default; it's a one-line change (and timm models often include it). A nice "attention to detail" line in a methods section.
Shape tracing: the debugging recipe. Every CNN bug is a shape bug. The recipe: (1) write the expected shape after each layer as a comment before coding; (2) run one forward pass with a dummy tensor and print shapes; (3) when they disagree, apply the sizing formula by hand:
model = nn.Sequential(
nn.Conv2d(3, 64, 7, stride=2, padding=3), # 224 -> 112
nn.MaxPool2d(3, stride=2, padding=1), # 112 -> 56
nn.Conv2d(64, 128, 3, padding=1), # 56 -> 56
)
x = torch.randn(1, 3, 224, 224)
for i, layer in enumerate(model):
x = layer(x)
print(i, type(layer).__name__, tuple(x.shape))
# 0 Conv2d (1, 64, 112, 112)
# 1 MaxPool2d (1, 64, 56, 56)
# 2 Conv2d (1, 128, 56, 56)
Verify each line: conv → floor((224−7+6)/2)+1 = 112 ✓; pool → floor((112−3+2)/2)+1 = 56 ✓. Do this once per architecture and shape bugs stop being mysterious.
Worked: ResNet-18's spatial ladder. Input 224 → conv1 (7×7, s2) → 112 → maxpool (3×3, s2) → 56 → layer1 (s1) → 56 → layer2 (s2) → 28 → layer3 (s2) → 14 → layer4 (s2) → 7 → GAP → 1×1×512 → FC → classes. Total downsampling: 32×. This 32× ladder (224→7) is near-universal in classification CNNs. When you design a custom stem, matching it keeps you in well-tested territory; when you deviate, the sizing formula tells you exactly what you built.
Receptive field arithmetic, fully worked — and ASPP
A complete worked example. Let's compute receptive fields through a realistic small network — input 64×64, layers: conv1 (3×3, s1) → conv2 (3×3, s1) → pool (2×2, s2) → conv3 (3×3, s1) → conv4 (3×3, s2) → conv5 (3×3, s1). Track (RF, cumulative stride), starting at (1, 1):
- conv1: RF = 1 + (3−1)×1 = 3, stride 1 → (3, 1)
- conv2: RF = 3 + 2×1 = 5 → (5, 1)
- pool: RF = 5 + (2−1)×1 = 6, stride 2 → (6, 2)
- conv3: RF = 6 + 2×2 = 10 → (10, 2)
- conv4: RF = 10 + 2×2 = 14, stride 4 → (14, 4)
- conv5: RF = 14 + 2×4 = 22 → (22, 4)
Final neuron sees 22×22 of the 64×64 input. Now the judgment call: if your objects are ~40 pixels wide, this network cannot use whole-object shape — it's deciding from parts and texture. Either deepen the network, add stride-2 stages, or accept texture-based classification and say so. Doing this arithmetic before training is the difference between designing and hoping.
The effective receptive field, quantified. Luo et al. (2016) proved the influence distribution is Gaussian: in a deep stack, the center pixel of the theoretical RF contributes exponentially more than the edges, because combinatorially more paths connect center to output. Practical consequence: a "theoretical RF of 400" might effectively use only the central ~100 pixels. If your application needs genuine long-range integration — a tumor's relation to a distant anatomical landmark, a field's relation to a far irrigation channel — don't trust depth alone. Options: dilated convolutions (Chapter 2), non-local/self-attention blocks, or multi-scale architectures.
ASPP: multi-scale context without losing resolution. Atrous Spatial Pyramid Pooling (DeepLab) runs several dilated convolutions in parallel (rates 1, 6, 12, 18) on the same feature map and concatenates the results. Each branch sees a different scale; together they capture both fine detail and broad context at full resolution. It's the standard "context module" dropped into segmentation heads for medical and satellite work — one import in segmentation_models_pytorch, and a citable, well-understood component rather than a hand-rolled context hack.
Worked: a mini-AlexNet's shapes, end to end
Input 227×227×3 (AlexNet's odd size). conv1: 96 kernels, 11×11, stride 4, no padding → floor((227−11)/4)+1 = 55. Output: 55×55×96. pool1: 3×3, stride 2 → floor((55−3)/2)+1 = 27. Output: 27×27×96. conv2: 256 kernels, 5×5, stride 1, padding 2 → (27−5+4)+1 = 27. Output: 27×27×256. pool2: 3×3 s2 → floor((27−3)/2)+1 = 13. Output: 13×13×256. conv3/4/5: 3×3, s1, p1 → 13×13 preserved; channels 384, 384, 256. pool3: 3×3 s2 → floor((13−3)/2)+1 = 6. Output: 6×6×256 = 9,216 features → FC 4096 → FC 4096 → 1000.
Now read the design: aggressive early downsampling (11×11 s4 + pool) collapses 227→27 in two steps — fast, but fine detail is gone before layer 3. Modern stems (ResNet's 7×7 s2 + pool: 224→56) preserve more. When you adapt these classics, the stem is the first thing to modernize. And notice the channel pattern 96→256→384→384→256 with spatial 55→27→13→6: halve space, grow channels — VGG later regularized this into the doubling rule (Chapter 4). The classics were already groping toward it.
Key takeaways - W_out = floor((W − K + 2P)/S) + 1. Memorize it. - Padding P = (K−1)/2 with stride 1 preserves spatial size ("same" convolution). - Max pooling (2×2, stride 2) halves dimensions and gives approximate translation invariance; it has no parameters. - Strided convolutions have largely replaced pooling for learned downsampling; global average pooling replaced giant FC layers before the classifier. - Receptive fields grow additively with depth and multiplicatively with stride; the effective RF is smaller and Gaussian — verify your network can actually see your objects.
Chapter 4 — Classic Architectures: LeNet, AlexNet, VGG
Architectures are not just historical trivia. Each classic network solved a specific problem, and each solution became a reusable design pattern. Read this chapter as a catalog of patterns, not a museum tour.
LeNet-5 (LeCun et al., 1998): the template
LeNet-5 classified handwritten digits (MNIST) with this structure:
Conv → Pool → Conv → Pool → Conv → FC → FC → output
Two 5×5 convolutions with average pooling between them, then fully connected layers. About 60,000 parameters. On MNIST it reached ~99% accuracy — essentially solving the dataset.
What LeNet taught us (patterns still in use):
- Alternating convolution and pooling builds the feature hierarchy: each stage sees more abstract, more spatially coarse features.
- End-to-end gradient learning beats hand-designed features. Before LeNet, digit recognition meant carefully engineered feature extractors feeding a classifier; LeNet learned the features and the classifier jointly.
- Small kernels stacked (5×5 here) compose into larger effective patterns.
LeNet's limitation was scale: 60K parameters, tiny 32×32 inputs, trained on CPUs. It proved the concept but couldn't touch real-world image complexity — the hardware and data didn't exist yet.
AlexNet (Krizhevsky, Sutskever & Hinton, 2012): the big bang
AlexNet won ImageNet 2012 (1.2M images, 1000 classes) with 15.3% top-5 error versus 26.2% for the runner-up — a margin so large it ended the era of hand-engineered vision features overnight. 60 million parameters, 8 layers.
Its innovations, each a pattern:
- ReLU instead of tanh/sigmoid. ReLU (max(0, x)) doesn't saturate for positive inputs, so gradients flow undiminished through deep stacks. Training was several times faster. Today ReLU (and variants) are the default nonlinearity — the tanh era ended here.
- GPU training. Two GTX 580s, model split across them. This wasn't a footnote; it was the enabling condition. Deep learning's progress has been gated by compute at every step, and AlexNet was the demonstration.
- Dropout (0.5) in the FC layers. Randomly zeroing half the activations during training prevented the giant fully connected layers from co-adapting and overfitting. Dropout remains a standard regularizer (Chapter 9).
- Data augmentation. Random crops, horizontal flips, and PCA-based color jitter artificially expanded the training set — the first large-scale proof that augmentation is not optional (Chapter 8).
- Overlapping max pooling (3×3 pool, stride 2) — slightly better than non-overlapping.
The lesson of AlexNet for researchers: scale plus the right training recipe beats clever features. A decade of SIFT/HOG engineering was outrun in one paper by a bigger network trained properly. When your hand-designed baseline loses to a plain ResNet, this is why — and it's not an insult to your idea, it's the field's central empirical fact.
PyTorch gives you AlexNet in one line (torchvision.models.alexnet(weights=...)), but its real value today is as a transfer-learning baseline and a teaching model — it's obsolete for new work.
VGG (Simonyan & Zisserman, 2014): depth with discipline
VGG asked: what if we just make it deeper, but keep everything else brutally simple? VGG-16 uses only 3×3 convolutions (stride 1, padding 1) and only 2×2 max pooling (stride 2), stacked 16 layers deep. 138 million parameters — most in the three giant FC layers at the end.
VGG's enduring contributions:
- Small stacked kernels beat large single kernels. Two 3×3 convolutions have the same 5×5 effective receptive field as one 5×5 convolution, but use fewer parameters (2 × 9C² = 18C² vs 25C²) and add an extra nonlinearity between them — more expressive per parameter. Three 3×3s ≈ one 7×7 (27C² vs 49C²). This is why 3×3 became the default kernel everywhere. Know this derivation; interviewers and reviewers both ask for it.
- Uniform, modular design. VGG showed that a simple repeated block (conv-conv-pool) scales cleanly. Modularity became the design norm — ResNet's residual blocks and Inception's modules are its descendants.
- Depth helps, up to a point. VGG-19 barely beat VGG-16, hinting at the optimization wall that ResNet would break (Chapter 5).
VGG's costs: 138M parameters and ~15 GFLOPs per image made it slow, and those FC layers are pure parameter bloat. Modern practice keeps VGG's convolutional wisdom and discards its classifier — which is exactly what transfer learning does (Chapter 7).
The lineage in one table
| Network | Year | Depth | Params | ImageNet top-5 err | Core lesson |
|---|---|---|---|---|---|
| LeNet-5 | 1998 | 7 | 60K | n/a (MNIST) | Conv-pool alternation; end-to-end learning |
| AlexNet | 2012 | 8 | 60M | 15.3% | ReLU, GPU, dropout, augmentation |
| VGG-16 | 2014 | 16 | 138M | 7.3% | Stacked 3×3 > large kernels; modular depth |
Code: the VGG block pattern
import torch.nn as nn
def vgg_block(in_ch, out_ch, n_convs):
layers = []
for _ in range(n_convs):
layers += [nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1),
nn.ReLU(inplace=True)]
in_ch = out_ch
layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
return nn.Sequential(*layers)
# VGG-16's convolutional stages: (64)x2, (128)x2, (256)x3, (512)x3, (512)x3
features = nn.Sequential(
vgg_block(3, 64, 2), vgg_block(64, 128, 2), vgg_block(128, 256, 3),
vgg_block(256, 512, 3), vgg_block(512, 512, 3),
)
x = torch.randn(1, 3, 224, 224)
print(features(x).shape) # torch.Size([1, 512, 7, 7])
Five poolings halve 224 → 112 → 56 → 28 → 14 → 7. That 7×7×512 tensor is the classic "conv feature volume" that older papers fed to SVMs or FC layers — and that you'll feed to global average pooling in modern variants.
For your research. Cite these three papers correctly (references at the end of this book) — mis-citing AlexNet's venue or year is a classic student tell. More substantially: when you write "we use a VGG-style backbone," say what you mean — stacked 3×3 convolutions with doubling channels after each pooling (64→128→256→512). That channel-doubling convention (halve spatial, double channels) preserves approximate computational cost per stage and is the single most-copied design decision in CNN history. If you deviate from it, explain why.
Engineering details the history books skip
AlexNet's forgotten tricks. Two deserve mention because they teach lessons. (a) Local Response Normalization (LRN) — AlexNet normalized each activation by its neighbors' activity, inspired by lateral inhibition in real neurons. It helped slightly in 2012 and was abandoned within two years once Batch Normalization (2015) did the job properly. Lesson: biologically-inspired heuristics lose to principled optimization tools; don't cargo-cult old tricks into new architectures. (b) Split-GPU training — the model was split across two GTX 580s because one card couldn't hold it. The accidental consequence: the two halves learned somewhat specialized features (one largely color-agnostic, one color-focused) — an early hint of the grouped convolutions that ResNeXt later used deliberately. Engineering constraints becoming design principles is a recurring pattern in this field.
Where VGG's 138 million parameters actually live. The convolutional stages hold only about 15M. The three fully connected layers (25088→4096→4096→1000) hold roughly 123M — nearly 90% of the model sits in the classifier head. This is why modern networks replaced the FC stack with global average pooling (Chapter 3): VGG's convolutions were the valuable part; its classifier was parameter bloat. When you use VGG as a feature extractor today, you keep the 15M parameters that matter and discard the 123M that don't — a vivid illustration of transfer learning's selectivity, and a good answer to "why not just use the whole pretrained model."
Loading the classics today.
from torchvision import models
vgg = models.vgg16(weights=models.VGG16_Weights.IMAGENET1K_V1)
alex = models.alexnet(weights=models.AlexNet_Weights.IMAGENET1K_V1)
print(vgg.features[0]) # Conv2d(3, 64, kernel_size=(3, 3), padding=(1, 1))
# Feature extraction: drop the classifier, keep the conv stages
features = vgg.features # -> (B, 512, 7, 7) for 224px input
Note the weights enums carry version information (IMAGENET1K_V1 vs V2) — V2 weights are retrained with a modern recipe and score several points higher on the same architecture. "VGG-16, torchvision IMAGENET1K_V1" is a complete, reproducible specification; "we used VGG" is not. Record the version.
What the classics get wrong by modern standards. No batch normalization (training is twitchier and slower), heavy FC heads (parameter bloat, as shown above), and large first-layer kernels (AlexNet's 11×11 stride-4 stem discards fine detail before the network ever sees it — VGG's stacked 3×3s were already the fix). Study these networks for their patterns, not their hyperparameters. And a terminology note for your writing: "AlexNet" denotes the specific 2012 architecture; "AlexNet-style" in a paper usually just means "a shallow convnet with ReLUs and dropout" — be precise about which you mean, because a reviewer will hold you to the specific one.
What survived: the pattern inventory from LeNet to VGG
It's worth making explicit which ideas from these three networks are still alive in 2026, because "we use a VGG-style backbone" is a claim about which patterns you kept:
| Pattern | Source | Still standard? |
|---|---|---|
| Conv→pool alternation building a hierarchy | LeNet | Yes — every CNN |
| End-to-end learned features beating hand-designed ones | LeNet | Yes — the field's founding result |
| ReLU as default nonlinearity | AlexNet | Yes (variants: GELU, SiLU in newer nets) |
| Dropout in classifier heads | AlexNet | Yes |
| Data augmentation as standard practice | AlexNet | Yes — non-negotiable |
| GPU training | AlexNet | Yes — now table stakes |
| Stacked 3×3 instead of large kernels | VGG | Yes — the default kernel |
| Channel doubling when halving spatial size (64→128→256→512) | VGG | Yes — near-universal |
| Giant FC classifier heads | VGG/AlexNet | No — replaced by GAP |
| LRN, split-GPU training | AlexNet | No — historical curiosities |
Two networks between the classics deserve one paragraph each, because reviewers may ask why you skipped them. Network-in-Network (Lin et al., 2014) introduced the 1×1 convolution and global average pooling — both now ubiquitous (Chapters 2–3). GoogLeNet/Inception (Szegedy et al., 2015) ran multiple kernel sizes (1×1, 3×3, 5×5) in parallel inside each module, arguing that optimal local structure varies by scale; it won ImageNet 2014 with far fewer parameters than VGG. Its legacy: multi-scale processing (see ASPP, Chapter 3) and the 1×1 bottleneck-before-expensive-conv pattern. You don't need to use Inception today, but "multi-scale features" as a phrase descends from it.
Reading old papers productively. When you read Krizhevsky et al. (2012), skip the LRN sections and focus on the training recipe (augmentation, dropout, SGD schedule) — the recipe outlived the architecture. When you read Simonyan & Zisserman (2014), focus on the 3×3-stacking argument and the depth study (their Table 3, showing accuracy vs. depth, is the direct ancestor of every ablation table you'll write). Old papers are mines: extract the patterns, leave the hyperparameters.
Timeline: the decade that built modern vision
| Year | Milestone | What it established |
|---|---|---|
| 1989 | LeNet precursor (LeCun et al.) | Backprop through conv layers works |
| 1998 | LeNet-5 | Conv–pool hierarchy; end-to-end learning |
| 2012 | AlexNet | ReLU, GPU, dropout, augmentation — deep learning goes mainstream |
| 2013 | ZFNet (Zeiler & Fergus) | Visualization as debugging; smaller stride-2 first layer |
| 2014 | VGG | Stacked 3×3; modular depth |
| 2014 | GoogLeNet/Inception | Multi-scale modules; 1×1 bottlenecks |
| 2014 | Network-in-Network | 1×1 convs; global average pooling |
| 2015 | ResNet | Skip connections; 100+ layers trainable |
| 2015 | BatchNorm (Ioffe & Szegedy) | The quiet enabler of all deep training |
| 2016 | Wide ResNet, ResNeXt, pre-activation | Width/cardinality as scaling axes |
| 2017 | MobileNetV1, DenseNet | Efficiency; dense feature reuse |
| 2018 | MobileNetV2, SE blocks | Inverted residuals; channel attention |
| 2019 | EfficientNet | Compound scaling; the B0–B7 ladder |
Read down the "what it established" column: that is the entire design vocabulary of modern CNNs. Every architecture since 2019 is a recombination of these rows. When a new paper claims novelty, locate its row-combination — "it's ResNet blocks + SE + compound scaling" — and you'll understand the contribution in one sentence. This is also how you write related work (Chapter 12): position by idea-lineage, not by year.
Key takeaways - LeNet: conv-pool alternation + end-to-end learning. The template. - AlexNet: ReLU, GPUs, dropout, augmentation. The big bang — scale + recipe beat hand-designed features. - VGG: stacked 3×3 kernels (same RF as 5×5/7×7, fewer params, more nonlinearities); modular blocks; channel doubling after pooling. - Each architecture is a bundle of reusable patterns. Learn the patterns, not the parameter counts.
Chapter 5 — ResNet and Skip Connections
The degradation problem
By 2015, the field knew depth helped — but only to a point. Researchers observed something strange: a 56-layer plain network had higher training error than a 20-layer one. Not test error — training error. This wasn't overfitting (overfitting is low training error, high test error). The deeper network was failing to even fit the training data. Adding layers made optimization harder, not just generalization.
He et al.'s key insight: suppose a shallow network reaches some good solution. A deeper network contains the shallow one as a special case — it could just learn identity mappings in the extra layers and match the shallow performance. The fact that it can't, in practice, means the optimizer struggles to learn identity mappings through stacked nonlinear layers. So: make identity the default, and learn only the deviation from it.
The residual trick
Instead of asking a stack of layers to learn the desired mapping H(x) directly, ask it to learn the residual F(x) = H(x) − x, and add the input back:
y = F(x) + x
That "+" is the skip connection (or shortcut): the input bypasses the layer stack and is added to its output. If the optimal mapping is close to identity, the layers just drive F(x) toward zero — pushing weights to zero is easy for an optimizer (weight decay does it by default). If the optimal mapping is far from identity, the network still learns the full transformation through F.
Why does this fix the vanishing gradient? Look at the backward pass. The gradient of the loss with respect to x flows through two paths: through F (the long way, through all the layer weights) and directly through the skip connection (gradient = 1, untouched). Even if the long path's gradients vanish to nothing, the skip path delivers a clean gradient of 1 to every earlier layer. Skip connections are gradient highways. Information — and, crucially, gradient signal — can travel the full depth of the network without passing through a single weight matrix if needed.
A residual block in its simplest form:
class BasicBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(channels)
self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU(inplace=True)
def forward(self, x):
identity = x
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out = out + identity # the skip connection
return self.relu(out)
ResNet-152 (152 layers!) trained successfully with this pattern and won ImageNet 2015 at 3.57% top-5 error — below the commonly cited ~5% human error rate. The degradation problem was gone: ResNet-152 had lower training error than ResNet-34.
When dimensions don't match
The skip connection needs x and F(x) to have the same shape. When a block changes spatial size (stride 2) or channel count, ResNet uses a projection shortcut: a 1×1 convolution with matching stride on the skip path:
# downsampling block: stride-2 conv + 1x1 projection on the shortcut
self.shortcut = nn.Sequential(
nn.Conv2d(in_ch, out_ch, kernel_size=1, stride=2, bias=False),
nn.BatchNorm2d(out_ch))
The 1×1 convolution here does no spatial work — it just remaps channels and matches the stride so the addition lines up. (This is the same "1×1 as channel mixer" idea from Chapter 2.)
Batch normalization: the quiet partner
Every ResNet block pairs convolutions with batch normalization: normalize each channel to zero mean, unit variance across the batch, then scale and shift with learned parameters. BN stabilizes the distribution of layer inputs during training (reducing "internal covariate shift," in the original paper's framing), which lets you use higher learning rates and makes deep stacks trainable at all. Practical notes:
- BN behaves differently in train vs. eval mode (it uses batch statistics when training, running averages at inference). Always call
model.eval()before inference, or your predictions will be wrong in a subtle, batch-dependent way. This is the single most common ResNet bug in student code. - With very small batch sizes (< 8–16), batch statistics are noisy and BN hurts; use GroupNorm or LayerNorm instead — common in medical imaging where GPU memory forces tiny batches.
ResNet variants you'll meet
- ResNet-18/34 use BasicBlocks (two 3×3 convs). ResNet-50/101/152 use bottleneck blocks (1×1 → 3×3 → 1×1), which cut computation: the 1×1s shrink then restore channel counts around the expensive 3×3.
- Pre-activation ResNet (He et al., 2016): move BN+ReLU before the convolution. Slightly better gradient flow; some codebases use it by default.
- ResNeXt: grouped convolutions in the block ("cardinality" as a new dimension). Wide ResNet: fewer, wider layers — often better than deeper-thinner for the same budget.
For new work, ResNet-50 is the default backbone: 25.6M parameters, ~4 GFLOPs, pretrained weights available everywhere, and every reviewer knows what it means.
Worked example: gradient through the skip
Consider y = F(x) + x with a scalar loss L. By the chain rule:
dL/dx = dL/dy × (dF/dx + 1)
Even if dF/dx → 0 (vanished through the block's weights), dL/dx = dL/dy × 1 = dL/dy. The gradient passes through unchanged. Stack 50 such blocks: the gradient to the first layer is dL/dy_50 × Π(1 + dF_i/dx_i) — a product of terms each near 1, instead of a product of small numbers. That's the whole mathematical story of why 152 layers train.
For your research. ResNet-50 pretrained on ImageNet is your default starting point for almost any image task (Chapter 7 explains how to adapt it). When you report results, "ResNet-50 backbone" is unambiguous — but specify the weight source (torchvision, timm) because pretrained weights differ slightly across implementations and affect reproducibility. If your images are very different from natural photos (grayscale X-rays, multispectral satellite), say so explicitly and consider whether the early-layer filters transfer (they usually do — edges are edges) while the later layers need more adaptation. The skip-connection gradient argument above is worth understanding deeply: it's the same mechanism behind DenseNet, U-Net's long skips, and transformer residual streams. Learn it once, reuse it everywhere.
The bottleneck math, and the ResNet family tree
Worked: why bottlenecks save compute. A BasicBlock operating on 256 channels uses two 3×3×256×256 convolutions: 2 × 9 × 256 × 256 ≈ 1.18M parameters. A bottleneck block instead does: 1×1 (256→64) + 3×3 (64→64) + 1×1 (64→256) = 16,384 + 36,864 + 16,384 ≈ 69K parameters — 17× fewer for the same input/output width, with the expensive 3×3 operating in the cheap 64-channel space. ResNet-50's 25.6M parameters would exceed 200M with basic blocks at that depth; bottlenecks are what make 50+ layers affordable. The general principle: do expensive spatial work in low-dimensional spaces, and use cheap 1×1 convolutions to move between widths. You'll meet this pattern again in MobileNetV2 (Chapter 6) and in transformer feedforward layers — it's universal enough to be worth memorizing.
The family tree — patterns to steal.
- Pre-activation ResNet (He et al., 2016): reorder each block to BN→ReLU→Conv, so the skip path is a pure identity unobstructed by ReLU. Slightly better optimization; some codebases (e.g., timm's ResNet-V2 variants) default to it.
- Wide ResNet (Zagoruyko & Komodakis, 2016): widen channels (×2 to ×10) instead of deepening. A 28-layer wide ResNet often beats a 100-layer thin one at the same parameter budget — depth has diminishing returns once skip connections fix the optimization problem. Lesson: when a deeper model stalls, try wider before trying deeper.
- ResNeXt (Xie et al., 2017): split the 3×3 convolution into 32 parallel groups ("cardinality" as a new scaling dimension). Same FLOPs, more representational paths. The grouped convolution here descends from AlexNet's split-GPU accident (Chapter 4) — an engineering constraint become a design principle, again.
- Stochastic Depth (Huang et al., 2016): during training, randomly drop entire residual blocks — the skip connection carries the signal past them. This regularizes very deep networks and shortens the effective depth early in training. One line in timm; worth trying on any 100+ layer model.
class Bottleneck(nn.Module):
expansion = 4
def __init__(self, in_ch, mid_ch, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(in_ch, mid_ch, 1, bias=False)
self.conv2 = nn.Conv2d(mid_ch, mid_ch, 3, stride=stride, padding=1, bias=False)
self.conv3 = nn.Conv2d(mid_ch, mid_ch * self.expansion, 1, bias=False)
self.bn1 = nn.BatchNorm2d(mid_ch)
self.bn2 = nn.BatchNorm2d(mid_ch)
self.bn3 = nn.BatchNorm2d(mid_ch * self.expansion)
self.relu = nn.ReLU(inplace=True)
self.shortcut = nn.Sequential()
if stride != 1 or in_ch != mid_ch * self.expansion:
self.shortcut = nn.Sequential(
nn.Conv2d(in_ch, mid_ch * self.expansion, 1, stride=stride, bias=False),
nn.BatchNorm2d(mid_ch * self.expansion))
def forward(self, x):
out = self.relu(self.bn1(self.conv1(x)))
out = self.relu(self.bn2(self.conv2(out)))
out = self.bn3(self.conv3(out))
return self.relu(out + self.shortcut(x))
ResNet beyond classification. The same backbone-with-skips serves as the encoder in U-Net-style segmentation (Chapter 11), the feature extractor in Faster R-CNN detection, and the image tower in CLIP. Learning ResNet deeply pays rent across every vision task — which is why this book keeps recommending it as your default backbone. Time spent understanding skip connections is never wasted: the identical mechanism reappears in DenseNet's dense skips, U-Net's long skips, and transformer residual streams.
DenseNet, and the vanishing gradient in equations
DenseNet: skips taken to the extreme. If ResNet adds the input to the output (y = F(x) + x), DenseNet (Huang et al., 2017) concatenates all previous feature maps as input to each layer: layer l receives [x_0, x_1, ..., x_{l−1}]. Every layer talks directly to every later layer — maximum gradient flow, maximum feature reuse. Parameter-efficient (narrow layers suffice since features are reused, not relearned), but memory-hungry (all those concatenated maps live on GPU). DenseNet-121 remains a strong, well-understood backbone, especially loved in medical imaging. The design lesson: ResNet reuses via addition (cheap, fixed width), DenseNet via concatenation (richer, memory-costly). When choosing, ask whether your GPU budget favors parameters (DenseNet) or activations memory (ResNet).
The vanishing gradient, in one equation. In a plain deep stack, the gradient to an early layer is a product of many Jacobians: ∂L/∂x_1 = ∂L/∂x_n × Π_{i} ∂x_{i+1}/∂x_i. If each Jacobian's singular values are mostly < 1 (typical with saturating nonlinearities and small weights), the product shrinks exponentially with depth — 0.9^50 ≈ 0.005. The signal dies. BatchNorm and ReLU mitigate this (ReLU's derivative is 0 or 1, never a shrinking fraction), but the structural fix is the skip: with y = F(x) + x, each factor becomes (∂F/∂x + I), whose eigenvalues cluster near 1 rather than near 0. Products of numbers near 1 don't vanish. That single "+ I" is worth more than any initialization trick — which is why ResNet enabled 152 layers while careful initialization alone stalled around 20–30.
How deep is too deep? Empirically: ResNet-152 beats ResNet-101 by ~0.5% on ImageNet at 2× the cost; ResNet-1001 (yes, it was tried) barely beats 152. Depth's returns diminish because representational capacity stops being the bottleneck — data and optimization are. For your thesis: ResNet-50 is the sweet spot for most tasks; reach for 101/152 only when you have the data (100K+ images) and the ablations show depth actually helping. "We used ResNet-152 because deeper is better" without evidence is a sentence reviewers underline.
The original ResNet training recipe (and what to keep)
He et al. trained with SGD, momentum 0.9, weight decay 1e-4, batch size 256, initial LR 0.1, divided by 10 at epochs 30/60 (for the 90-epoch schedule), with scale/aspect-ratio augmentation and per-pixel mean subtraction. No Adam, no cosine, no warmup, no label smoothing — and it hit 3.57% top-5. Two lessons. First, the recipe's structure (long training, step decay, strong augmentation, weight decay) matters more than the specific optimizer — modern recipes are refinements, not replacements. Second, note what they didn't need: the architecture did the heavy lifting. When your training is unstable, the debugging order is: architecture sanity (shapes, RF) → data pipeline → LR/schedule → optimizer choice — roughly in that order of likelihood.
BatchNorm placement, settled. ResNet V1: Conv→BN→ReLU, with the skip added before the final ReLU. Pre-activation V2: BN→ReLU→Conv with a clean identity skip. If you implement ResNet yourself, use V2 — the pure skip path gives marginally better gradient flow and it's what timm's resnetv2 variants use. If you use torchvision's, you get V1 — fine, just know which you have when comparing numbers across libraries. "ResNet-50 (torchvision V1 weights)" vs "ResNet-50-V2 (timm)" can differ by ~1%, and that difference is not yours to claim.
Key takeaways
- Deep plain networks suffer degradation: higher training error than shallow ones — an optimization failure, not overfitting.
- Residual learning: fit F(x) = H(x) − x, output y = F(x) + x. Identity becomes the easy default.
- Skip connections are gradient highways: dL/dx keeps a clean path of gradient 1 through every block.
- BatchNorm is ResNet's quiet partner; remember model.eval() and the small-batch caveat.
- ResNet-50 is the default research backbone. Bottleneck blocks (1×1–3×3–1×1) make depth affordable.
Chapter 6 — Efficient Modern CNNs: MobileNet and EfficientNet
ResNet-50 is a fine backbone — until you need to run on a phone, a drone, a field-deployed Raspberry Pi, or a hospital workstation without a GPU. Or until your thesis budget gives you one GPU and your architecture search needs to finish before graduation. This chapter is about getting the most accuracy per unit of compute.
The cost anatomy of a convolution
Recall the MAC count: H × W × C_out × K² × C_in. The K² term is the lever. A 3×3 kernel costs 9× a 1×1. And a standard convolution mixes spatial filtering and channel mixing in one dense operation. MobileNet's insight: factorize them.
MobileNet: depthwise separable convolutions
A depthwise separable convolution splits the standard convolution into two steps:
- Depthwise convolution: each input channel gets its own single K×K filter — no cross-channel mixing. Cost: H × W × C_in × K².
- Pointwise convolution: a 1×1 convolution mixing channels. Cost: H × W × C_in × C_out.
Total: HWC_in(K² + C_out) versus HWC_in·K²·C_out for the standard version — a reduction factor of roughly 1/C_out + 1/K². For K=3 and C_out=512, that's about 8–9× fewer MACs.
class DepthwiseSeparableConv(nn.Module):
def __init__(self, in_ch, out_ch, stride=1):
super().__init__()
self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3,
stride=stride, padding=1,
groups=in_ch, bias=False) # groups=in_ch: one filter per channel
self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(in_ch)
self.bn2 = nn.BatchNorm2d(out_ch)
self.relu = nn.ReLU(inplace=True)
def forward(self, x):
x = self.relu(self.bn1(self.depthwise(x)))
return self.relu(self.bn2(self.pointwise(x)))
# Compare costs for 56x56x64 -> 128 channels:
std = nn.Conv2d(64, 128, 3, padding=1)
sep = DepthwiseSeparableConv(64, 128)
print(sum(p.numel() for p in std.parameters())) # 73,856
print(sum(p.numel() for p in sep.parameters())) # 8,896 (~8.3x fewer)
MobileNetV1 (2017) used this throughout and reached ~70% ImageNet top-1 with 4.2M parameters — near-VGG accuracy at a fraction of the cost. MobileNetV2 (2018) added the inverted residual block: expand channels with 1×1, depthwise-filter in the wide space, project back with 1×1 (linear, no ReLU — because ReLU destroys information in low-dimensional bottlenecks, a subtle and important detail). MobileNetV3 added squeeze-and-excitation and neural-architecture-search-derived tweaks.
EfficientNet: scaling with a formula
Practitioners knew three ways to make a network better: make it deeper (more layers), wider (more channels), or feed higher-resolution inputs. Everyone scaled one dimension by gut feeling. Tan and Le's EfficientNet (2019) showed the dimensions interact and should be scaled together, with a compound coefficient φ:
- depth × α^φ, width × β^φ, resolution × γ^φ, with α·β²·γ² ≈ 2 (so each φ step ≈ 2× FLOPs).
They found α=1.2, β=1.1, γ=1.15 by small grid search on a baseline (EfficientNet-B0, itself found by neural architecture search with mobile inverted bottleneck blocks), then scaled φ from 0 (B0) to 7 (B7). EfficientNet-B7 reached 84.3% top-1 — state of the art at the time — with 66M parameters, versus prior SOTA needing far more. More importantly for researchers: EfficientNet-B0 through B4 give you a clean accuracy-vs-compute ladder. Need a stronger baseline? Move one rung up the ladder instead of hand-tuning.
The efficiency landscape (what to pick when)
| Model | Params | FLOPs | ImageNet top-1 | Use when |
|---|---|---|---|---|
| MobileNetV3-Small | 2.5M | 0.06G | 67.4% | Phone/edge, real-time |
| MobileNetV3-Large | 5.4M | 0.22G | 75.2% | Edge with headroom |
| EfficientNet-B0 | 5.3M | 0.39G | 77.1% | Strong tiny baseline |
| EfficientNet-B4 | 19M | 4.2G | 82.9% | Accuracy on 1 GPU |
| ResNet-50 | 25.6M | 4.1G | 76.1%* | Default research backbone |
| EfficientNet-B7 | 66M | 37G | 84.3% | Max accuracy, big compute |
(*torchvision v2 weights reach ~80%; weight version matters — always report it.)
Note the punchline: EfficientNet-B4 roughly matches ResNet-50's FLOPs with ~7 points more accuracy. Architecture design is not dead; it compounds.
Practical guidance for limited compute
- Start from pretrained efficient weights (
timmlibrary:timm.create_model('efficientnet_b0', pretrained=True)). Training these from scratch on small data wastes your budget. - Match input resolution to the model: EfficientNets were scaled with resolution (B0: 224, B4: 380, B7: 600). Feeding 600px images to B0 wastes compute; feeding 224px to B7 wastes the model's capacity.
- For deployment, benchmark latency on the actual device, not FLOPs — depthwise convolutions are FLOP-efficient but can be memory-bandwidth-bound on some hardware. Measure, don't assume.
- Knowledge distillation (train a small net to mimic a big net's outputs) pairs naturally with efficient architectures and is a legitimate paper contribution when done carefully.
For your research. "We used EfficientNet-B0 due to compute constraints" is a respectable sentence — reviewers understand budgets. What's not respectable is using an inefficient backbone and apologizing for weak results. Pick the strongest model your compute allows, and report params/FLOPs alongside accuracy (the Learning Dashboard in this book has a table template). If your deployment target is a phone or field device, MobileNetV3-Small with quantization is the standard answer, and "runs at X ms on device Y" is a result in itself — systems-minded reviewers value it.
Squeeze-and-Excitation, inverted residuals by hand, deployment reality
Squeeze-and-Excitation (SE). Hu et al. (2018) added a tiny channel-attention module to any block: squeeze each channel to a single number via global average pooling, pass through a two-layer MLP (bottleneck, ReLU, sigmoid), and excite — multiply each channel by its learned weight. Cost: a few thousand parameters. Effect: the network learns "which channels matter for this particular image" — consistently around +1% on ImageNet for almost any backbone. MobileNetV3 and EfficientNet both include SE blocks. Ten lines:
class SEBlock(nn.Module):
def __init__(self, channels, reduction=16):
super().__init__()
self.pool = nn.AdaptiveAvgPool2d(1)
self.fc = nn.Sequential(
nn.Linear(channels, channels // reduction),
nn.ReLU(inplace=True),
nn.Linear(channels // reduction, channels),
nn.Sigmoid())
def forward(self, x):
w = self.fc(self.pool(x).flatten(1)).view(x.size(0), -1, 1, 1)
return x * w
Inverted residual, worked numerically. MobileNetV2 block: input 24 channels, expansion factor 6, stride 1. (1) Expand 1×1: 24→144 (3,456 params). (2) Depthwise 3×3 over 144 channels (1,296 params). (3) Project 1×1, linear (no ReLU): 144→24 (3,456 params). (4) Add the input via skip. Total ≈ 8.2K parameters. A standard 3×3 residual block at 24 channels would cost 2×(9×24×24) ≈ 10.4K — comparable, but the inverted block does its spatial filtering in the wide 144-channel space (more expressive) while the projections stay cheap. The "inverted" name: classic residuals go wide→narrow→wide; these go narrow→wide→narrow. And the final projection is deliberately linear, because ReLU destroys information in low-dimensional bottlenecks — a detail confirmed by ablation in the MobileNetV2 paper, and a good example of the principle that every nonlinearity placement is a claim you can test.
Deployment reality check. FLOPs don't equal latency: depthwise convolutions are memory-bandwidth-bound on many chips, so a "0.06 GFLOP" model can be slower than the arithmetic suggests on a CPU without optimized depthwise kernels. Always benchmark on the target device with the actual inference stack (ONNX Runtime, TensorFlow Lite, Core ML). Then quantize: post-training INT8 quantization typically shrinks models 4× with under 1% accuracy loss on MobileNets and EfficientNets; quantization-aware training recovers most of the remainder. "Quantized to INT8 with 0.4% accuracy drop and 3.2× speedup on device X" is a complete, respectable systems result — and for field deployments (Chapter 11) it may matter more than the last point of accuracy.
Neural architecture search in one paragraph. EfficientNet-B0's baseline wasn't hand-designed — a search algorithm explored block types, depths, and widths to maximize accuracy per FLOP. The lesson for researchers isn't "run NAS" (it's expensive) but "the search space matters more than the search": B0's space was built from mobile inverted bottleneck blocks with SE — good primitives, composed well. When you design a custom architecture, spend effort on the primitive block and the scaling rule, not on hand-placing fifty layers.
Compound scaling, worked — and EfficientNetV2
The scaling math, concretely. EfficientNet's rule: depth = α^φ, width = β^φ, resolution = γ^φ, with α·β²·γ² ≈ 2 so each φ step doubles FLOPs. With α=1.2, β=1.1, γ=1.15: check 1.2 × 1.1² × 1.15² = 1.2 × 1.21 × 1.3225 ≈ 1.92 ≈ 2 ✓. Now scale from B0 (φ=0) to B4 (φ=4): depth ×1.2⁴ ≈ ×2.07, width ×1.1⁴ ≈ ×1.46, resolution 224×1.15⁴ ≈ 224×1.75 ≈ 392 (actual B4 uses 380 — rounded to a friendly number). FLOPs: 0.39G × 2⁴ ≈ 6.2G (actual 4.2G — the rounding and operator realities shave it down). The point isn't the exact arithmetic; it's that one knob (φ) replaces three-dimensional guesswork, and the resulting family dominates the accuracy–FLOPs frontier. When your paper says "we scaled with φ=2," every reader knows exactly what you did.
EfficientNetV2 (2022): faster training, same idea. Tan & Le's follow-up fixed V2's pain points: training was slow (large images + depthwise convs are memory-inefficient early on) and the scaling sometimes hurt at small sizes. V2 mixes MBConv and Fused-MBConv blocks (fusing the depthwise+pointwise into a regular conv in early stages where it's actually faster), uses a better training recipe, and scales more conservatively. Practical upshot: EfficientNetV2-S trains ~2× faster than B4 at similar accuracy. If you're training from scratch or fine-tuning heavily on one GPU, V2-S/M are the current sweet spots — timm has them all.
Picking your rung: a decision procedure. (1) Measure your compute budget in GPU-hours for the whole project, including ablations (multiply one run's time by ~10). (2) Pick the largest EfficientNet whose single run fits in ~1/10 of that budget. (3) If deploying to edge, instead pick the largest MobileNetV3 that meets your latency target on the device. (4) Validate the choice with a 2-rung comparison (e.g., B0 vs B2) early — if the bigger rung gains <0.5%, stay small and spend the budget on data and augmentation. Architecture rungs are cheap to test and the result ("B2 gave +0.3% over B0 at 2.8× cost, so we report B0") is a respectable methods sentence that shows engineering judgment.
Reading a model card, and listing the zoo
Every serious model release has a model card: architecture, training data, weight version, input size, normalization, license, and benchmark numbers. Before using any pretrained backbone, read its card and record: (1) exact model name, (2) weight identifier, (3) training dataset (ImageNet-1K vs 21K pretraining changes transfer behavior), (4) input resolution and normalization. Your methods section's reproducibility lives in these four fields.
import timm
# Survey what's available for your constraint:
names = timm.list_models('efficientnet*', pretrained=True)
print(len(names), names[:5])
model = timm.create_model('efficientnet_b2', pretrained=True, num_classes=8)
print(model.default_cfg['input_size']) # (3, 260, 260) -- use this resolution
timm's default_cfg tells you the input size and normalization the weights expect — use them, don't guess. A common silent bug: feeding 224px images to a model trained at 380px (or vice versa) and losing 2–3% without knowing why. The model card — or default_cfg — is the contract; honor it.
Edge deployment: a worked sizing example
Sizing a model for a real device. Say your target is a mid-range Android phone running a crop-disease app, budget 100 ms/image on CPU via TensorFlow Lite. Candidate: MobileNetV3-Large at 224px — 0.22 GFLOPs, ~5.4M params (~21 MB FP32, ~5.4 MB INT8). Estimate: a phone CPU sustains roughly 5–15 GFLOPs effective on depthwise-heavy models (memory-bound, not compute-bound), so 0.22 GFLOPs ÷ 10 GFLOPs ≈ 22 ms — comfortably inside budget even with preprocessing overhead. EfficientNet-B0 (0.39 GFLOPs, denser ops, better utilization) lands around 30–40 ms. Both fit; pick B0 for accuracy unless the profiler disagrees. The method: estimate from FLOPs, then measure on device — the estimate tells you which two models to benchmark, the benchmark tells you which ships. Report both numbers in the paper; "estimated then measured" reads as engineering, not luck.
Key takeaways - Depthwise separable convolution factorizes spatial filtering from channel mixing: ~8–9× cheaper than standard 3×3 conv. - MobileNetV2's inverted residual: expand → depthwise → project (linear bottleneck, no ReLU in the narrow layer). - EfficientNet scales depth, width, and resolution together with a compound coefficient — a ladder (B0→B7) instead of guesswork. - Report params, FLOPs, and measured latency; FLOPs alone don't predict speed on real hardware. - For limited compute: pretrained efficient backbones + matched input resolution + distillation beats training big models from scratch.
Chapter 7 — Transfer Learning in Practice
Why it works
The hierarchy from Chapter 1 is the reason transfer learning works: early CNN layers learn Gabor-like edge and color detectors that are useful for any image task, and mid layers learn textures and parts that transfer surprisingly far — from ImageNet photos to X-rays to satellite tiles. Only the last layers are truly task-specific. So instead of training from random initialization on your 5,000 images (which will overfit), you start from a network trained on 1.2M ImageNet images and adapt it.
Two regimes:
- Feature extraction: freeze the backbone, train only a new classifier head. Fast, stable, works with tiny datasets (hundreds of images). The backbone becomes a fixed feature extractor.
- Full fine-tuning: train everything, usually with a small learning rate. Better final accuracy when you have enough data (thousands+), but riskier.
In practice, the winning recipe is usually staged: feature-extract first (cheap, gives a baseline), then unfreeze and fine-tune with discriminative learning rates.
Regime 1: feature extraction (code)
import torch.nn as nn
from torchvision import models
model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
# Freeze everything
for param in model.parameters():
param.requires_grad = False
# Replace the classifier head: 2048 features -> your classes
num_classes = 10 # e.g., 10 crop-disease categories
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Only the head's parameters will update
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
Note model.fc.in_features — always read the head's input dimension from the model rather than hardcoding 2048. Different backbones differ, and hardcoding is how shape bugs happen at 2 a.m.
Regime 2: full fine-tuning with discriminative learning rates
Early layers need tiny updates (their features are already good); later layers and the head need larger ones. Discriminative learning rates encode this:
model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Unfreeze all
for param in model.parameters():
param.requires_grad = True
# Layer groups: backbone gets 1e-5, head gets 1e-3
optimizer = torch.optim.Adam([
{'params': model.layer4.parameters(), 'lr': 1e-4},
{'params': model.fc.parameters(), 'lr': 1e-3},
# layer1-3 get the default lr below
], lr=1e-5, weight_decay=1e-4)
Practical rules that prevent the classic failures:
- Warm up the head first. Train the randomly initialized head alone for a few epochs before unfreezing the backbone. Otherwise the head's garbage gradients thrash the good pretrained weights in the first steps — the main cause of "fine-tuning made it worse."
- Use a learning rate ~10× smaller than from-scratch training. 1e-4 to 1e-5 for the backbone is typical with Adam; with SGD, 1e-3 to 1e-2 with momentum.
- Keep augmentation on (Chapter 8) — fine-tuning without augmentation on small data just memorizes.
Handling the channel mismatch
Pretrained backbones expect 3-channel RGB. Your data might be grayscale (X-ray) or multispectral (satellite, N channels). Options:
# Option A: replicate grayscale to 3 channels (simplest, usually fine)
# x_gray: (B, 1, H, W) -> x: (B, 3, H, W)
x = x_gray.repeat(1, 3, 1, 1)
# Option B: adapt the first conv layer for N channels.
# Initialize new-channel weights from the mean of RGB weights (a standard trick):
conv1 = model.conv1 # (64, 3, 7, 7)
new_conv = nn.Conv2d(5, 64, kernel_size=7, stride=2, padding=3, bias=False)
with torch.no_grad():
new_conv.weight[:, :3] = conv1.weight # keep RGB filters
mean_w = conv1.weight.mean(dim=1, keepdim=True)
new_conv.weight[:, 3:] = mean_w.repeat(1, 2, 1, 1) # new bands start as average filter
model.conv1 = new_conv
Option A is embarrassingly effective — try it first. Option B is for when you have real extra spectral information and enough data.
When transfer learning fails: negative transfer
Transfer isn't magic. Watch for:
- Domain gap too large: ImageNet features help less for, say, seismic cross-sections than for photos of plant leaves. If feature extraction barely beats random init, the gap is large — consider domain-specific pretraining (e.g., RadImageNet for radiology) or self-supervised pretraining on your own unlabeled data.
- Tiny datasets + full fine-tuning: with < 1,000 images, full fine-tuning often overfits. Prefer feature extraction or freeze early stages.
- Label leakage through the head: when classes are fine-grained (e.g., 200 bird species), the pretrained head's structure is useless — replace it and don't over-interpret early results.
Self-supervised pretraining: the modern alternative
When you have abundant unlabeled domain images but few labels, pretrain the backbone yourself with a self-supervised objective (SimCLR, MoCo, DINO, masked autoencoders) on your own data, then fine-tune on labels. This routinely beats ImageNet transfer for specialized domains (medical, satellite) and is very publishable: "in-domain self-supervised pretraining improves over ImageNet transfer by X" is a clean, honest contribution. Libraries like lightly make SimCLR a few lines of code.
For your research. Your paper's methods section should state: backbone architecture, weight source and version (e.g., "torchvision ResNet-50 IMAGENET1K_V2"), which layers were frozen/unfrozen and when, learning rates per group, and input preprocessing (resize, normalization values — use ImageNet mean/std
[0.485, 0.456, 0.406],[0.229, 0.224, 0.225]when using ImageNet weights, since the filters were learned on normalized inputs). This is reproducibility basics, and missing it is the most common methods-section complaint from reviewers. If you compare "from scratch" vs "fine-tuned," train both with tuned hyperparameters — an untuned from-scratch baseline is a strawman, and reviewers can tell.
Fine-tuning craft: LP-FT, BatchNorm freezing, layer-wise decay
Linear probe then fine-tune (LP-FT). Kumar et al. (2022) showed a two-stage protocol beats naive fine-tuning on out-of-distribution robustness: first train only the head (linear probing) to convergence, then fine-tune everything with a small learning rate. Why it works: the probing stage finds a good head without distorting pretrained features; fine-tuning then adapts gently from a good starting point instead of thrashing good features with a random head's early gradients. It's the principled version of this chapter's "warm up the head first" rule. Cost: one extra training stage. Worth it whenever distribution shift matters — medical, satellite, anything far from ImageNet.
Freezing BatchNorm — the subtle failure mode. You unfreeze the backbone for fine-tuning, but your batches are small (say 8 images). BatchNorm recomputes statistics from 8 samples — noisy — and performance degrades versus keeping ImageNet's running statistics. The standard fix: keep BN layers in eval mode (frozen statistics) while training everything else:
def freeze_bn(model):
for m in model.modules():
if isinstance(m, nn.BatchNorm2d):
m.eval() # use running stats; call after model.train()
This is the default in most detection and segmentation fine-tuning codebases, for good reason. (Alternatively, replace BN with GroupNorm when training from scratch on small batches — Chapter 5.)
Layer-wise learning-rate decay (LLRD). Instead of hand-picking per-group learning rates, decay geometrically from head to stem: lr_l = base_lr × decay^(depth − l). With decay 0.9 across 4 ResNet stages, the stem trains at about 0.66× the head rate. One hyperparameter instead of four:
decay = 0.9
param_groups = []
stages = [model.layer1, model.layer2, model.layer3, model.layer4]
for depth, layer in enumerate(stages):
param_groups.append({'params': layer.parameters(),
'lr': 1e-4 * (decay ** (3 - depth))})
param_groups.append({'params': model.fc.parameters(), 'lr': 1e-3})
optimizer = torch.optim.AdamW(param_groups)
The timm library. Ross Wightman's timm (PyTorch Image Models) is the researcher's model zoo: over a thousand architectures with pretrained weights behind one API. timm.create_model('efficientnet_b3', pretrained=True, num_classes=10) builds and adapts in a single call. Prefer it over torchvision when you need architecture variety — but record the exact model name and weight provenance in your paper, since timm updates weights over time and "efficientnet_b3" today may not equal "efficientnet_b3" next year.
Catastrophic forgetting, concretely. Fine-tune too aggressively — high learning rate, many epochs on tiny data — and the network forgets its ImageNet features: validation accuracy on your task rises, then falls as features collapse into dataset-specific memorization. Symptoms: training loss goes to zero while validation degrades, and first-layer filters (Chapter 10) lose their Gabor structure. Cure: lower backbone LR, fewer epochs, stronger augmentation, or LP-FT. If you observe it, you've learned something real about your data regime — note it down. "Aggressive fine-tuning destroyed pretrained features on our 800-image dataset" is a legitimate ablation row, and it warns the next researcher.
Transfer learning under class imbalance, and progressive resizing
Imbalance meets transfer: a protocol. Real domain datasets are imbalanced (95% healthy leaves, 5% diseased). The pretrained backbone doesn't fix this — it just gives you better features for the majority class too. Combine transfer with imbalance handling: (a) weighted sampling — oversample minority classes so each batch is balanced (WeightedRandomSampler); (b) class-weighted loss (Chapter 9's snippet); or (c) focal loss, which down-weights easy well-classified examples and focuses learning on hard minority cases. Start with weighted sampling — simplest, and it interacts cleanly with augmentation. Then report metrics that respect imbalance: per-class F1, balanced accuracy, AUC — and show the confusion matrix (Chapter 12). A transfer-learning paper on imbalanced data that reports only top-line accuracy is hiding the result that matters.
from torch.utils.data import WeightedRandomSampler
counts = torch.tensor([9500., 300., 200.]) # images per class
weights = 1.0 / counts
sampler = WeightedRandomSampler(weights[targets], num_samples=len(targets), replacement=True)
loader = torch.utils.data.DataLoader(ds, batch_size=32, sampler=sampler)
Progressive resizing. Train first at low resolution (e.g., 128px), then fine-tune at full resolution (224/380px). Early epochs run ~3× faster (quarter the pixels), and the network learns coarse structure first — a curriculum the pretrained weights accommodate gracefully since early filters are scale-tolerant. Practical recipe: 70% of epochs at small size, 30% at full size, dropping the LR when switching. Particularly valuable for EfficientNets at high resolution (B4's 380px is expensive). One caution: BatchNorm statistics shift with resolution — run a few "calibration" epochs at the new size before judging, or the first full-resolution validation will look spuriously bad.
A fine-tuning case study in numbers. Typical leaf-disease task, 8,000 images, 12 classes, EfficientNet-B0: (a) from scratch, 100 epochs: 84.2 ± 0.6%; (b) frozen backbone + head, 15 epochs: 88.7 ± 0.3%; (c) staged fine-tuning (head 5 epochs, then all layers LR 1e-4, 25 epochs): 91.3 ± 0.4%; (d) same as (c) + LP-FT + RandAugment: 92.1 ± 0.3%. Each stage's gain is modest; the compound gain is 8 points. This is the normal shape of transfer-learning results — no single trick dominates, and the paper writes itself as a protocol ablation (Chapter 12's table format). Run this ladder once on your data before committing to a final configuration.
Adapting a new backbone: the 12-step checklist
When you pick up an unfamiliar backbone (ConvNeXt, a new timm release, a domain-specific model), run this checklist before any training:
- Read the model card: input size, normalization, weight version, license.
- Print the stem:
model.conv1/model.stem— know the first layer's channels and stride. - Print stage output shapes with a dummy input (Chapter 3's shape tracing).
- Identify the classifier head attribute (
fc,classifier,head) and itsin_features. - Check the norm layers: BatchNorm (needs eval-mode care, frozen-stats option) vs LayerNorm/GroupNorm.
- Confirm the forward returns logits (some models return tuples or dicts).
- Verify preprocessing matches the card (resize, crop, mean/std).
- Run a frozen-backbone linear probe for 3 epochs — establishes the floor and shakes out pipeline bugs.
- Overfit one batch (Chapter 9, check 4) — confirms loss, labels, and optimizer all work.
- Unfreeze with discriminative LRs; watch the first-epoch validation for catastrophic forgetting.
- Compare against your previous backbone with identical protocol — one variable at a time.
- Record everything in the paper ledger (Chapter 12) before moving on.
Ninety percent of "the new backbone didn't help" stories are steps 1–9 failures, not architecture failures. The checklist turns backbone-swapping from alchemy into procedure.
Key takeaways - Transfer works because early/mid-layer features (edges, textures) are universal; only late layers are task-specific. - Feature extraction (frozen backbone) for tiny data; staged fine-tuning with discriminative LRs for more data. - Always warm up the new head before unfreezing the backbone. - Grayscale → repeat to 3 channels; multispectral → adapt conv1, init new bands from the RGB mean. - Report backbone, weight version, freeze schedule, and normalization — reproducibility lives in these details.
A final note on trust
Transfer learning is the closest thing our field has to a free lunch — but free lunches get taken for granted. The researchers who benefit most are the ones who treat pretrained weights as a starting hypothesis ("these features probably help") rather than a fixture: probe them, visualize them, ablate them, and report where they failed. That skeptical habit — applied to backbones, augmentations, and your own results — is what turns a competent practitioner into a researcher reviewers trust.
Chapter 8 — Data Augmentation for Images
The idea in one paragraph
Your network should classify a cat as a cat whether the photo is slightly rotated, cropped tighter, or taken under warmer light. Since you can't photograph every variation, you synthesize them: each training epoch, every image gets randomly transformed — flipped, cropped, color-jittered — so the network never sees the same exact pixels twice. Augmentation is the cheapest way to multiply a dataset, and on small datasets it's often worth more than a fancier architecture.
What works: the standard toolkit
Geometric transforms (change where pixels are):
- Random horizontal flip (p=0.5): the single highest-value augmentation for natural images. A cat facing left is still a cat.
- Random resized crop: crop a random region (scale 0.08–1.0 of the area) and resize to input size. Forces the network to recognize objects at multiple scales and positions. The workhorse of ImageNet training.
- Small rotations (±15°): useful when orientation varies naturally (aerial imagery, microscopy).
Photometric transforms (change pixel values):
- Color jitter: random brightness, contrast, saturation, hue shifts. Simulates lighting/camera variation.
- Grayscale conversion (occasional, p=0.1): discourages over-reliance on color.
Advanced (use deliberately):
- MixUp: blend two images and their labels (x = λx₁ + (1−λ)x₂, y likewise). Smooths decision boundaries; strong regularizer.
- CutMix: paste a rectangular patch from one image onto another, mixing labels by area proportion. Like MixUp but preserves local structure — often better for CNNs.
- RandAugment / TrivialAugment: automatically search/apply a random sequence of transforms. Excellent defaults when you don't want to hand-tune a policy.
from torchvision import transforms
train_tf = transforms.Compose([
transforms.RandomResizedCrop(224, scale=(0.08, 1.0)),
transforms.RandomHorizontalFlip(p=0.5),
transforms.RandomRotation(15),
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.05),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]),
])
val_tf = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]),
])
Note the discipline: validation/test transforms are deterministic (resize + center crop, no randomness). Augmenting the test set leaks randomness into your evaluation and makes results unreproducible. This mistake appears in student code constantly.
What to avoid: augmentations that lie
Every augmentation encodes an assumption: "this transform preserves the label." Violate it and you train the network on lies.
- Horizontal flip on text, digits, or asymmetric anatomy: a flipped "3" is not a digit; a flipped chest X-ray puts the heart on the wrong side. Medical images: never flip unless the anatomy is truly symmetric for your task.
- Rotation on orientation-sensitive tasks: rotating a satellite image is fine (north isn't special for most tasks); rotating a photo of a building 90° creates images that never occur naturally.
- Color jitter on color-critical tasks: if you're classifying ripeness by hue, jittering hue destroys the signal. Stain normalization in histopathology exists precisely because color is the signal there.
- Aggressive cropping on small objects: RandomResizedCrop with scale 0.08 can crop the object out entirely, leaving a background patch labeled "tumor." For detection/segmentation or small objects, constrain the crop scale (e.g., 0.5–1.0) or use object-aware cropping.
- MixUp/CutMix with rare classes: blending a rare-class image with a common one can drown the rare signal. Use with class balancing, or skip for heavily imbalanced data.
Rule of thumb: look at augmented samples. Save a grid of 64 augmented images and inspect them before any training run. If a human can't recognize the label, the network is learning noise. Five minutes of looking saves days of confused training curves.
Test-time augmentation (TTA)
At inference, average predictions over several augmented versions (flips, crops) of each test image. Typically worth +0.5–1% accuracy for free — no retraining. It's standard in competitions and legitimate in papers, but report it explicitly ("results use 5-crop TTA") since it multiplies inference cost.
For your research. Your augmentation policy is a methods-section item: list every transform with parameters, and justify domain-sensitive choices ("we excluded vertical flips because crop rows have a consistent orientation in our field imagery"). Reviewers in applied domains (medical, agriculture, remote sensing) actively look for augmentation mistakes — a flipped X-ray or hue-jittered ripeness classifier will get your paper flagged. Conversely, a well-reasoned policy ("rotations ±180° because satellite tiles have no canonical orientation") signals domain competence. Ablate it: a one-row table showing accuracy with/without augmentation quantifies its contribution and costs you one extra training run.
Tooling, segmentation augmentation, and TTA in code
Albumentations: the serious toolkit. torchvision transforms are fine for classification; Albumentations is faster and — critically — applies identical geometric transforms to images, masks, and bounding boxes together. For detection and segmentation, where the mask must track the image exactly, this is essential:
import albumentations as A
tf = A.Compose([
A.RandomResizedCrop(256, 256, scale=(0.5, 1.0)),
A.HorizontalFlip(p=0.5),
A.Rotate(limit=15, p=0.5),
A.RandomBrightnessContrast(p=0.3),
], seed=42)
out = tf(image=img_np, mask=mask_np) # mask transformed identically
Note seed=42: reproducible augmentation pipelines matter when you need to regenerate exact training samples for debugging. Also note the crop scale floor of 0.5 — the small-object rule from this chapter's table, encoded as a parameter.
Segmentation augmentation rules. Masks require nearest-neighbor interpolation — never bilinear, which creates fractional class labels at boundaries. Photometric transforms apply to the image only, never the mask. Geometric transforms apply to both. Libraries handle this correctly when you pass the mask separately (as above); hand-rolled pipelines get it wrong constantly. For small objects, also consider copy-paste augmentation: cut rare-class instances and paste them onto new backgrounds. It's highly effective for small datasets and imbalanced segmentation, and it's a legitimate, citable method choice.
RandAugment in two lines. When hand-tuning a policy feels arbitrary, let the machine choose: RandAugment(num_ops=2, magnitude=9) applies two random operations per image from a curated set. TrivialAugment goes further — one operation, random magnitude, zero hyperparameters — and matches tuned policies on most benchmarks (Cubuk et al., 2020). These are the right defaults behind "we used standard augmentation" in a paper: cite them, state the parameters, move on.
Test-time augmentation, concrete.
model.eval()
flip = transforms.RandomHorizontalFlip(p=1.0)
views = [val_tf(img)]
views.append(transforms.Compose([flip, *val_tf.transforms])(img))
with torch.no_grad():
logits = torch.stack([model(v.unsqueeze(0)) for v in views]).mean(0)
Two to five views (flips, multi-crops) typically add 0.3–1.0% accuracy. Report the TTA policy alongside the number — "87.3% (5-crop TTA)" — because TTA multiplies inference cost, and a deployment-minded reviewer will ask whether the headline number is achievable in production.
Augmentation ablation mini-table. Run this once per new dataset; a single extra training run buys certainty about where tuning stops paying:
| Augmentation | Val accuracy |
|---|---|
| None | 78.1 ± 0.8 |
| Flip only | 80.4 ± 0.6 |
| Flip + crop | 83.9 ± 0.5 |
| Flip + crop + color jitter | 84.6 ± 0.4 |
| Full + RandAugment | 85.2 ± 0.5 |
Diminishing returns are normal and expected — the table tells you where to stop tuning augmentation and start training the final model.
Domain augmentation recipes, and two cautionary tales
Ready-made recipes by domain. Start from these, then ablate (Chapter 8's mini-table):
Medical (X-ray, grayscale): RandomResizedCrop scale (0.7, 1.0), small rotation ±10°, intensity jitter (brightness/contrast 0.15), Gaussian noise σ=0.01, no flips for asymmetric anatomy. Rationale: anatomy has a canonical orientation and left-right meaning; intensity varies by scanner. Histopathology (stained slides): flips + 90° rotations (no orientation at cellular scale), stain-color jitter or stain normalization, crops (0.5, 1.0). Rationale: rotation-invariant, but stain chemistry is the signal — jitter it only mildly. Agriculture (field photos): flips, crops (0.5, 1.0), strong brightness/contrast/saturation jitter, ±30° rotation, CutMix. Rationale: sunlight and cameras vary wildly; backgrounds are cluttered. Satellite (multispectral tiles): full 360° rotation, flips, scale jitter (0.5, 1.5), per-band brightness jitter, no color jitter across bands (band ratios like NDVI are the signal — jittering bands independently destroys them). Rationale: no canonical orientation; spectral relationships are sacred.
Cautionary tale 1: the flipped X-ray. A published-style mistake that recurs in student work: horizontal flips applied to chest X-rays. The heart's leftward position is diagnostic — flipping creates anatomically impossible images labeled as real. The network happily learns "heart on right = normal variant," and test accuracy looks fine because the test set has the same corruption. The fix is domain knowledge, not more data. Lesson: every augmentation is a claim about your domain's invariances; claims need a domain expert's sign-off.
Cautionary tale 2: hue jitter on ripeness. A fruit-ripeness classifier augmented with hue jitter (±0.1) — the exact transform Chapter 8's defaults suggest. But ripeness is hue: green→yellow→red is the label signal. Training accuracy was excellent (the network memorized other cues); field performance collapsed. The fix: geometric-only augmentation plus brightness jitter (lighting varies; hue doesn't, for a fixed ripeness stage). Lesson: the "standard" recipe is a starting point, and for color-critical tasks it's the wrong starting point. Look at your augmented samples (Chapter 8's five-minute rule) — a human agronomist would have caught this instantly.
The augmentation review: a pre-training ritual
Before every training run, spend ten minutes on this review — it catches the lying augmentations from this chapter's cautionary tales:
- Sample grid. Save 64 augmented images with labels overlaid. Can a domain expert (or you, after coffee) identify every label? Any "impossible" images (flipped anatomy, hue-destroyed ripeness, cropped-out objects)?
- Parameter bounds. For each transform, write the range and the justification: "rotation ±15° — our camera tilts, but never inverts." Unjustified parameters are guesses; guesses get ablated or removed.
- Mask sync (segmentation/detection): verify masks transformed identically — overlay 3 augmented image/mask pairs and inspect boundaries.
- Determinism check: run the validation pipeline twice on one image; outputs must be bit-identical.
- Class-conditional sanity: for rare classes, verify augmentation doesn't erase the signal — inspect 20 augmented minority-class samples specifically.
- Leakage scan: confirm augmentation is fit on train statistics only (normalization means, etc.) and that no test image appears in train (hash check on large datasets).
Print this, tape it next to the NaN checklist (Chapter 9). The two lists together prevent the large majority of silent training failures — the kind where the run completes, the number looks plausible, and the model is quietly wrong.
Quick reference: freeze schedules
Common schedules, from conservative to aggressive: (1) Head only — backbone frozen throughout; safest, tiny data. (2) Head then last stage — unfreeze layer4 after head converges; the default recommendation. (3) Gradual unfreezing — unfreeze layer4, then layer3, then layer2, a few epochs apart (top-down); maximally gentle, costs patience. (4) Full fine-tune from the start — only with 50K+ images and a low LR; otherwise it destroys features before the head is sane. Match the schedule to data size: <1K → (1); 1–10K → (2); 10–50K → (3); 50K+ → (4) is safe. When in doubt, go one step more conservative — under-adapting costs a point, over-adapting costs the run.
Sanity metric: the linear-probe floor
Before any fine-tuning run, record the linear-probe accuracy (frozen backbone + trained head) as your floor. Any fine-tuning protocol that scores below this floor is broken — usually a too-high backbone LR destroying features (Chapter 7's catastrophic forgetting) or a BatchNorm mishandling (frozen-stats vs recomputed). The probe costs one short run and gives you a tripwire: if staged fine-tuning underperforms the probe, stop and diagnose instead of tuning blindly. Report the probe number in your ablations too — "frozen backbone: 85.6%" as a row makes the fine-tuning gain (+1.7 in Chapter 7's example) interpretable rather than mysterious.
Key takeaways - Augmentation synthesizes the variations your dataset lacks; on small data it often beats a bigger model. - Defaults: random resized crop + horizontal flip + mild color jitter. Inspect samples before training. - Every augmentation asserts label preservation — flips, rotations, and color jitter each have domains where they lie. - Test transforms must be deterministic. Always. - TTA (averaging augmented predictions) is free accuracy at inference; report it when used.
Augmentation as experimental control
Think of your augmentation policy as an experimental control: it defines the invariances your model is allowed to assume. When two papers report different numbers on the same dataset, the augmentation difference is the first suspect — before architecture, before hyperparameters. This is why the policy belongs in the methods section with parameters, not in a footnote. And when you review others' work, ask the question this chapter trained you to ask: which invariances did they assume, and did the domain actually grant them?
Chapter 9 — Training CNNs: Learning Rates, Schedules, Regularization, Pitfalls
The optimizer question, settled pragmatically
Two serious options:
- SGD with momentum (0.9): the classic. Often reaches slightly better final generalization for CNNs; needs more learning-rate tuning. The ImageNet-era default and still the choice for squeezing the last 0.5%.
- Adam / AdamW: adaptive per-parameter learning rates; converges faster and is far less sensitive to the initial LR. The default for most research code today. AdamW (decoupled weight decay) is strictly preferable to Adam — regular Adam's weight decay interacts badly with the adaptive scaling.
Recommendation: start with AdamW (lr=1e-3 or 3e-4, weight_decay=0.01–0.05). If you're chasing leaderboard numbers on a well-understood benchmark, try SGD with momentum afterward. Don't agonize — the learning-rate schedule matters more than the optimizer choice.
Learning-rate schedules: the actual secret
A constant learning rate is almost never optimal. The standard pattern: start low (warmup), go high, decay down.
- Warmup (first 5–10% of training): linearly ramp LR from 0 to the target. Prevents early instability, especially with large batches or fresh heads on pretrained backbones.
- Decay: cosine annealing (smooth decay following a cosine curve to near-zero) is the modern default and usually beats step decay. Step decay (÷10 at epochs 30/60/90) is the ResNet-paper classic and still fine.
- OneCycle: ramp up then down in a single cycle; pairs well with AdamW and often trains faster.
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
for epoch in range(100):
train_one_epoch(model, loader, optimizer)
scheduler.step() # call after each epoch (or each batch for OneCycle)
Why decay helps: early training needs large steps to explore; late training needs small steps to settle into the minimum without bouncing out. A schedule is worth 1–3% accuracy on most tasks — more than many architecture tweaks.
Regularization: the full menu
- Weight decay (L2): penalizes large weights. With AdamW, 0.01–0.05; with SGD, 1e-4. It's the baseline regularizer — always on.
- Dropout (0.2–0.5, in FC/head layers): randomly zeroes activations. Less used inside modern conv stacks (BatchNorm already regularizes), still standard in classifier heads.
- Data augmentation (Chapter 8): the strongest regularizer you have. Treat it as regularization first, data multiplication second.
- Label smoothing (ε=0.1): softens one-hot targets to (1−ε) for the true class, ε/(K−1) elsewhere. Prevents overconfident predictions; typically +0.3–0.5%.
- Early stopping: monitor validation loss; stop when it hasn't improved for N epochs (patience 10–20). Save the best checkpoint, not the last.
- Stochastic depth / DropPath: randomly drops entire residual blocks during training. The ResNet-era answer to "how do I regularize a 100-layer network." Available in
timm.
Reading training curves: a diagnostic manual
Plot training loss, validation loss, and validation accuracy every run. The shapes diagnose everything:
- Both losses decrease together, gap small: healthy. Keep going or train longer.
- Training loss ↓, validation loss ↑ (gap widening): overfitting. Add augmentation, weight decay, dropout; get more data; use a smaller model; early-stop.
- Both losses plateau high: underfitting. Model too small, LR too low, or a bug. Increase capacity, raise LR, check preprocessing (wrong normalization is the classic silent killer — see below).
- Training loss oscillates wildly / NaN: LR too high, or unstable numerics. Lower LR, add warmup, check for exploding gradients (gradient clipping at norm 1.0 is cheap insurance).
- Validation accuracy noisy epoch-to-epoch: normal with small validation sets. Don't early-stop on noise — smooth the curve or use a larger patience.
- Loss drops then flatlines early: LR decayed too aggressively, or the optimizer fell into a sharp minimum. Try cosine instead of step decay.
Common pitfalls (the ones that eat weeks)
- Wrong normalization. Using ImageNet mean/std on non-ImageNet-like data, or forgetting normalization entirely, silently degrades everything by several points. Verify: after your transform pipeline, a batch's mean should be ≈ 0.
- Forgetting
model.train()/model.eval(). Dropout and BatchNorm behave differently per mode. Eval withoutmodel.eval()gives garbage; training withoutmodel.train()after eval disables dropout/BN updates. - Data leakage: augmentation or normalization statistics computed on the full dataset (including test); duplicate images across splits; patient-level leakage in medical data (same patient's scans in train and test). Split by patient/study, not by image.
- Untuned baselines: comparing your tuned method against a baseline trained with default hyperparameters. Tune the baseline's LR at minimum, or the comparison is meaningless.
- Batch size too small with BatchNorm: < 16 per GPU makes BN statistics noisy. Accumulate gradients to simulate larger batches, or switch to GroupNorm.
- Class imbalance ignored: with 95/5 class splits, accuracy is a lie (predicting "negative" always gives 95%). Use weighted loss, oversampling, or focal loss — and report F1/AUC, not just accuracy (Chapter 12).
# Class-weighted loss for imbalance:
counts = torch.tensor([9500., 500.]) # class frequencies
weights = counts.sum() / (len(counts) * counts)
criterion = nn.CrossEntropyLoss(weight=weights)
For your research. Log everything: hyperparameters, git commit hash, random seeds, dataset version, hardware. Tools like Weights & Biases or MLflow make this painless, and "we ran each experiment with 3 seeds and report mean ± std" is the difference between a result and an anecdote. Reviewers increasingly demand seed-averaged results — a single lucky seed is not a finding. When a run fails, the training curve plus this chapter's diagnostics usually identify the cause in minutes; build the habit of plotting before tweaking.
Speed and stability: mixed precision, accumulation, the LR finder
Mixed precision — a free speedup. Modern GPUs compute in FP16/BF16 far faster than FP32. PyTorch's automatic mixed precision gives roughly 1.5–2× speedup with a few lines and usually zero accuracy change:
from torch.amp import GradScaler, autocast
scaler = GradScaler()
for x, y in loader:
optimizer.zero_grad()
with autocast(device_type='cuda'):
loss = criterion(model(x), y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
scheduler.step()
The scaler prevents underflow of small gradients in FP16. Use AMP by default on any CUDA GPU from the last five years — there is no reason not to, and "trained with AMP" needs no justification in a paper.
Gradient accumulation — faking bigger batches. Your GPU fits batch 16 but you want the stability of 64: accumulate gradients over 4 steps before updating.
accum_steps = 4
for i, (x, y) in enumerate(loader):
loss = criterion(model(x), y) / accum_steps
loss.backward()
if (i + 1) % accum_steps == 0:
optimizer.step()
optimizer.zero_grad()
Caveat: BatchNorm still sees batches of 16 — statistics don't accumulate. Combine with GroupNorm or accept the noise. This is how small-GPU researchers train big-batch recipes, and it's worth stating in your methods ("effective batch size 64 via 4× accumulation").
The learning-rate range test. Instead of guessing the learning rate, sweep it exponentially in one short run and plot loss versus LR (Smith, 2017): loss falls, then explodes. Pick the LR about one order of magnitude below the explosion point — the steepest-descent region. Implementations exist (torch_lr_finder); it replaces an afternoon of guessing with a ten-minute run. Do this before any serious training, especially from scratch.
NaN and stall debugging checklist.
1. Loss NaN in the first epochs → learning rate too high. Halve it and add warmup.
2. Loss NaN suddenly mid-training → mixed-precision overflow (use BF16 or raise the loss scale) or a corrupt batch (broken JPEGs decode to garbage — check inputs for NaN).
3. Accuracy stuck at 1/N_classes → the network predicts a single class. Check label encoding (0-indexed? correctly shuffled?), then suspect LR far too high.
4. Loss decreases but accuracy doesn't move → label mismatch or metric bug. First, overfit a single batch: a healthy setup reaches ~100% training accuracy on 32 images within a few epochs. If it can't, the bug is in data or loss — not capacity.
5. Everything worked yesterday → you changed the seed, the data path, or the normalization. git diff and the paper ledger (Chapter 12) exist for exactly this moment.
Seed averaging, the honest way. Report mean ± std over at least 3 seeds, and keep a versioned run.sh per experiment so the exact command is reproducible:
python train.py --backbone resnet50 --lr 3e-4 --seed 0 --augment full
python train.py --backbone resnet50 --lr 3e-4 --seed 1 --augment full
python train.py --backbone resnet50 --lr 3e-4 --seed 2 --augment full
A result without seeds is a rumor; a result with three seeds and a script is evidence. Reviewers increasingly treat it that way.
Optimizers compared, and the regularization cookbook
SGD vs AdamW, honestly. The 2020s consensus, stripped of tribalism: AdamW converges faster, is robust to LR choice, and wins on most research timelines — it's the right default. SGD with momentum (0.9), tuned carefully with step decay, still edges out AdamW by 0.2–0.5% on some well-studied CNN benchmarks (ImageNet, CIFAR) — the price is a full LR-schedule search per task. So: use AdamW for exploration, ablations, and deadlines; consider SGD for the final benchmark push if you can afford the tuning. One more consideration: AdamW's adaptivity interacts with weight decay, which is why decoupled AdamW (not Adam + L2) is the correct form — with vanilla Adam, the L2 term gets scaled by the adaptive denominator and under-regularizes exactly the parameters that need it most.
The regularization cookbook — doses and interactions. Regularizers interact: heavy augmentation + high weight decay + dropout can underfit. Treat total regularization as one budget:
| Situation | Prescription |
|---|---|
| Small data (<5K), from scratch | Strong aug + wd 0.05 + label smoothing 0.1 + dropout 0.5 in head; expect to need all of it |
| Small data, fine-tuning | Moderate aug + wd 0.01 + label smoothing; dropout 0.2; freezing is itself regularization |
| Large data (100K+), from scratch | Mild aug + wd 1e-4 (SGD) / 0.05 (AdamW); dropout optional; data is the regularizer |
| Very deep net (100+ layers) | Add stochastic depth / DropPath 0.1–0.2 |
If training accuracy won't rise: you're over-regularized — remove, don't add. If the train/val gap widens: add in this order — augmentation, weight decay, label smoothing, dropout, smaller model. One at a time; that's an ablation, and it goes in the paper.
Distributed training in one paragraph. When one GPU is too slow, DistributedDataParallel splits batches across GPUs/machines with near-linear speedup — but the effective batch size grows, and large batches need LR scaling (linear rule: double batch → double LR, plus longer warmup). BatchNorm statistics are computed per-GPU unless you use SyncBatchNorm. You probably don't need this for thesis-scale work; but if your lab has a cluster, DDP + the linear scaling rule + SyncBatchNorm is the entire recipe, and mixed precision (this chapter) stacks multiplicatively with it.
Troubleshooting flowchart (text edition)
Loss NaN / inf?
├── Yes → in first epochs? → halve LR, add warmup
│ └── mid-training? → check inputs for NaN (corrupt files),
│ try BF16 / raise loss scale
├── No ↓
Accuracy stuck at chance (1/#classes)?
├── Yes → labels correct & 0-indexed? → fix encoding
│ └── labels fine? → LR too high; drop 10× and retry
├── No ↓
Train loss falls, val loss rises?
├── Yes → OVERFITTING → add augmentation → raise wd →
│ label smoothing → dropout → smaller model → early stop
│ (one at a time; it's an ablation)
├── No ↓
Both losses plateau high?
├── Yes → UNDERFITTING → capacity up? LR up? → check normalization
│ → overfit-one-batch test → if that fails, bug in data/loss
├── No ↓
Loss oscillates wildly?
├── Yes → LR too high → lower it; add warmup; gradient clip (norm 1.0)
├── No ↓
Val accuracy noisy?
└── Yes → small val set → smooth curves, larger patience;
don't early-stop on noise
Work the chart top to bottom; don't skip to your favorite hypothesis. Most "mysterious" training failures are row 1 or row 2 — learning rate and labels — not architecture. The chart's real purpose is discipline: it forces you to rule out the common causes before redesigning the network.
Augmentation strength as a curriculum
Ramp it up. One refinement worth knowing: start training with mild augmentation and strengthen it over epochs — a curriculum. Early epochs see near-original images (stable gradients, fast initial learning); later epochs see heavily distorted ones (robustness, regularization when overfitting risk is highest). Implementation: interpolate RandAugment magnitude 4→10, or crop scale (0.8,1.0)→(0.08,1.0), linearly over the first half of training. Gains are modest (+0.2–0.5%) but consistent, and it stabilizes the notoriously twitchy first epochs of from-scratch training. For fine-tuning, the subtler version suffices: keep augmentation mild while the head warms up (stage 1), then strengthen for full fine-tuning (stage 2) — matching regularization to the phase where it's needed.
Augmentation and the validation gap — a diagnostic note. Heavy augmentation inflates the train–validation gap even in a healthy model, because training images are harder versions of validation images. Don't misread this as overfitting. The check: evaluate the training set with validation transforms (no augmentation) — if that accuracy matches validation accuracy, the gap is augmentation difficulty, not memorization. True overfitting shows a gap even without augmentation. Log both numbers; their difference quantifies your augmentation's difficulty contribution.
When augmentation isn't enough: synthetic data
When real variation is unobtainable — rare diseases, dangerous field conditions, classified satellite regions — consider synthetic data: GAN/diffusion-generated images, physics-based simulators, or 3D renders. Rules: (1) never let synthetic images into the test set — evaluation must be on real data; (2) validate the sim-to-real gap explicitly (train synthetic-only, test on real — the gap quantifies what the simulator lacks); (3) mix synthetic with real rather than replacing it. Synthetic data is augmentation taken to its logical extreme, and it inherits augmentation's prime directive: every synthetic image asserts label preservation. A simulator that renders anatomically impossible pathology is worse than no data — have a domain expert review samples, just like the augmentation grid.
Key takeaways - AdamW (lr 3e-4, wd 0.05) is the sane default; the LR schedule (warmup + cosine decay) matters more than the optimizer. - Regularize in layers: weight decay + augmentation + label smoothing + early stopping + dropout in the head. - Training curves are diagnostic instruments: learn the five shapes (healthy, overfitting, underfitting, unstable, noisy-val). - Silent killers: wrong normalization, forgotten train/eval modes, data leakage, untuned baselines, tiny batches with BatchNorm. - Log hyperparameters, seeds, and code versions; report mean ± std over multiple seeds.
Chapter 10 — Visualizing What CNNs Learn
A CNN with 25M parameters is a black box — unless you open it. Visualization serves two purposes: debugging (is the model looking at the right thing?) and communication (figures that make a paper convincing). This chapter covers the three techniques you'll actually use.
1. First-layer filters: the direct look
The first convolutional layer's kernels operate directly on pixels, so you can just... look at them. Reshape each 3×3×3 (or 7×7×3) kernel into an image:
import matplotlib.pyplot as plt
conv1 = model.conv1.weight.data.cpu() # (64, 3, 7, 7) for ResNet
fig, axes = plt.subplots(8, 8, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
f = conv1[i].permute(1, 2, 0) # -> (H, W, C)
f = (f - f.min()) / (f.max() - f.min()) # normalize for display
ax.imshow(f); ax.axis('off')
plt.savefig('conv1_filters.png', dpi=150)
What you'll see in a trained network: oriented edge detectors (Gabor-like stripes), color blobs, and opponent-color patterns. In an untrained network: noise. If your trained first layer still looks like noise, something is wrong — the network isn't learning low-level structure (check LR, normalization, or whether the task is solvable at all). This is a 30-second sanity check worth running on every trained model.
2. Activation/feature maps: what fires where
Pick a layer, feed an image, and visualize its output channels. Early layers show edge/texture maps; deep layers show part/object detectors — some channels will fire precisely on eyes, wheels, or (in your domain) lesions.
activations = {}
def hook_fn(name):
def hook(module, inp, out):
activations[name] = out.detach()
return hook
model.layer3[0].conv2.register_forward_hook(hook_fn('layer3'))
model(torch.randn(1, 3, 224, 224))
feat = activations['layer3'][0] # (C, H, W)
# plot a grid of the first 64 channels...
Forward hooks are the mechanism — learn them; they're also how you extract features for transfer learning analysis and how Grad-CAM (below) gets its data.
3. Grad-CAM: where the network looked
Grad-CAM (Selvaraju et al., 2017) answers: "which image regions most influenced the prediction for class c?" The recipe:
- Forward pass; get the score y^c for the target class (before softmax).
- Backpropagate to the last convolutional layer's feature maps A^k.
- Global-average-pool the gradients per channel: α_k = mean(∂y^c/∂A^k). Each α_k is the "importance" of channel k for class c.
- Weight the feature maps by importance and ReLU: heatmap = ReLU(Σ_k α_k A^k).
- Upsample to image size, overlay on the input.
# Sketch (full implementations exist in the pytorch-grad-cam library):
features, grads = {}, {}
model.layer4.register_forward_hook(lambda m, i, o: features.update(f=o.detach()))
model.layer4.register_full_backward_hook(lambda m, gi, go: grads.update(g=go[0].detach()))
model.zero_grad()
score = model(img)[0, target_class]
score.backward()
weights = grads['g'].mean(dim=(2, 3), keepdim=True) # global average pool of gradients
cam = (weights * features['f']).sum(dim=1, keepdim=True)
cam = torch.relu(cam)
cam = (cam - cam.min()) / (cam.max() - cam.min()) # normalize to [0,1]
# upsample cam to 224x224 and overlay on the original image
Use pip install grad-cam for the battle-tested implementation rather than hand-rolling it in production code — but implement it once yourself, because the exercise teaches you exactly what the heatmap means and doesn't mean.
What Grad-CAM doesn't prove (say this in your paper if you use it): a heatmap over the tumor doesn't prove the model "understands" tumors — it shows spatial correlation between region activation and the class score. Heatmaps can also be misleadingly reassuring: a model can attend to the right region for the wrong reason (e.g., a chest drain tube correlated with disease rather than the disease itself — a famous failure mode). Pair visualizations with quantitative analysis (ablations, subgroup metrics), never as a substitute.
Saliency maps and occlusion (quick tour)
- Vanilla saliency: |∂y^c/∂(input pixels)| — which pixels would change the score fastest. Noisy but simple; guided backprop variants are cleaner.
- Occlusion sensitivity (Zeiler & Fergus): slide a gray patch over the image, record the score drop. Model-agnostic and brutally honest — if covering the background drops the score, your model uses background. Slow (hundreds of forward passes) but assumption-free.
Debugging with visualization: a field guide
- Heatmap on background, not object → the model learned a spurious correlation (hospital token on X-rays, ruler marks on skin lesions — both real published failure modes). Fix with data cleaning, augmentation, or masking.
- All heatmaps identical across classes → the head isn't discriminating; check the classifier layer and label quality.
- First-layer filters noisy + low accuracy → optimization failure, not architecture failure. Revisit Chapter 9 before redesigning the network.
- Strong heatmap + strong metrics on a too-easy dataset → check for leakage (Chapter 9, pitfall 3). Visualization plus skepticism catches what metrics miss.
For your research. One good Grad-CAM figure is worth a page of prose in a paper — put your best two or three examples (correct prediction attending the right region; an informative failure case) in the results. Caption them honestly: "Grad-CAM visualizations suggest the model attends to X; they are illustrative, not proof of causal reasoning." Reviewers respect calibrated claims. And keep the raw code that generated every figure in your repo — "how was this heatmap normalized?" is a question you want to answer from code, not memory.
Feature visualization, saliency sanity checks, and occlusion
Feature visualization (activation maximization). Instead of asking "what fired," ask "what would maximally fire this neuron": start from noise and gradient-ascent the input to maximize a channel's activation — with regularization (jitter, blur, frequency penalization), or the result is adversarial static. Olah et al.'s Distill work (2017) turned this into an art form: deep channels visualize as dog faces, buildings, textures. The lucent library implements it for PyTorch. Use it to discover what a channel "wants" — excellent for paper figures, and for finding surprising detectors. (A crisp "text detector" channel in your medical model is a red flag worth investigating before a reviewer does.)
# Conceptual sketch (use lucent for real work):
img = torch.randn(1, 3, 224, 224, requires_grad=True)
opt = torch.optim.Adam([img], lr=0.05)
for _ in range(256):
opt.zero_grad()
act = captured['layer4'] # via forward hook on the target layer
loss = -act[:, 17].mean() # maximize channel 17
loss.backward(); opt.step()
# apply jitter / blur regularization to img here each step
Saliency sanity checks. Adebayo et al. (2018) showed that some popular saliency methods produce plausible-looking heatmaps even for randomly initialized networks — they're edge detectors, not explanations. Before trusting any visualization in your paper, run two checks: (a) model randomization — randomize the network's weights; if the heatmap barely changes, the method isn't explaining your model; (b) label randomization — train on shuffled labels; a method that still highlights "sensible" regions is fooling you. Grad-CAM passes both; vanilla gradients partially fail. Run the checks and cite them: "we verified our Grad-CAM explanations with the Adebayo et al. randomization sanity checks" is a sentence reviewers like reading.
Occlusion sensitivity, in code. The assumption-free method from this chapter — slow but honest:
import numpy as np
model.eval()
base = model(img)[0, target].item()
H, W, patch, stride = 224, 224, 32, 16
heat = np.zeros((H, W))
with torch.no_grad():
for y in range(0, H - patch + 1, stride):
for x in range(0, W - patch + 1, stride):
occ = img.clone(); occ[:, :, y:y+patch, x:x+patch] = 0
drop = base - model(occ)[0, target].item()
heat[y:y+patch, x:x+patch] += drop
Regions whose occlusion drops the score most are the most important. Now compare against Grad-CAM: agreement between two independent methods is far more convincing than either alone. "Grad-CAM and occlusion sensitivity agree on the lesion region" makes a strong figure panel — two witnesses beat one.
Figure checklist for papers. Every visualization figure needs: (a) the method named and cited, (b) normalization stated (per-image min-max? fixed scale?), (c) the layer identified (Grad-CAM at layer4 looks very different from layer3 — say which), and (d) at least one failure case alongside the successes. Reviewers distrust a page of perfect heatmaps; one honest failure with a one-sentence hypothesis reads as science rather than marketing.
Beyond Grad-CAM: Score-CAM, TCAV, and choosing your explainer
Grad-CAM's siblings. Grad-CAM's gradients can be noisy (saturating ReLUs flatten them). Two descendants fix this differently: Grad-CAM++ (Chattopadhyay et al., 2018) uses second-order gradients for sharper localization, especially with multiple object instances; Score-CAM (Wang et al., 2020) drops gradients entirely — it masks the input with each upsampled feature map and measures the score change, weighting maps by their causal contribution. Score-CAM is slower (one forward pass per channel) but gradient-free, hence immune to saturation artifacts. Rule of thumb: Grad-CAM for routine debugging and paper figures (fast, standard, passes sanity checks); Score-CAM when gradients look suspicious or a reviewer challenges the explanation's faithfulness. The pytorch-grad-cam library implements all three behind one API — no reason to hand-roll.
TCAV: from "where" to "what." Grad-CAM answers where the network looked; TCAV (Kim et al., 2018, "Testing with Concept Activation Vectors") answers what it saw: given a human-defined concept (e.g., "striped texture," illustrated by example images), how sensitive is the class prediction to that concept's direction in activation space? A TCAV score of 0.9 for "microaneurysm texture" on a diabetic-retinopathy model is a far stronger claim than a heatmap — it quantifies reliance on a named, domain-meaningful concept. Cost: you must curate concept example sets (50–100 images each), which needs domain expertise. For medical and scientific papers, one TCAV result ("the model's decisions are sensitive to the clinically relevant concept, not to the scanner-artifact concept") can carry more weight than ten heatmaps.
Choosing your explainer — a decision table.
| Question | Method | Cost |
|---|---|---|
| Which pixels mattered for this prediction? | Saliency / guided backprop | 1 backward pass; noisy |
| Which regions drove this class score? | Grad-CAM | 1 backward pass; standard |
| Same, but gradients are saturated | Score-CAM | C forward passes; faithful |
| Which named concept does the model rely on? | TCAV | Concept sets + statistics |
| Honest model-agnostic check | Occlusion sensitivity | Hundreds of forward passes |
| What does this neuron detect? | Feature visualization | Optimization loop |
For a paper: lead with Grad-CAM (everyone reads it), corroborate with occlusion on key examples, and reach for TCAV when the claim is about what rather than where. Never present a single method's output as proof — explanations corroborate quantitative results; they don't replace them.
Reading two Grad-CAMs like a reviewer
Example A: the good heatmap. Chest X-ray, model predicts pneumonia (0.91). Grad-CAM at layer4 highlights the lower-left lung zone, matching the radiologist's opacity annotation. Occlusion sensitivity agrees (this chapter's two-witness rule). First-layer filters show Gabor structure; per-subgroup AUC holds across two hospitals. Verdict: the visualization corroborates the quantitative results. In the paper: one figure, caption states method/layer/normalization, and the text claims only "the model's salient regions coincide with clinically relevant zones" — not "the model understands pneumonia."
Example B: the heatmap that kills a paper (before submission). Skin-lesion classifier, 94% accuracy, Grad-CAM highlights... the ruler marks beside the lesion, not the lesion. The dataset's malignant cases were photographed with rulers more often — a spurious correlation, the classic published failure mode. The honest response, and what to do: (1) don't ship it — 94% is a lie about deployment; (2) crop/mask rulers or add ruler-augmented benign examples; (3) retrain and re-visualize; (4) report the episode in the paper's limitations ("initial models exploited ruler artifacts; we removed them and accuracy fell to 89%, which we report"). Reviewers reward this enormously — it demonstrates exactly the scientific judgment they want to see. A paper that hides its artifacts and a paper that found and fixed them are different papers; write the second one.
The one-sentence rule. For every visualization in your paper, write one sentence saying what it doesn't prove. If you can't write that sentence, you don't understand the figure well enough to publish it.
Key takeaways - First-layer filters: look at them. Edge/color structure = learning; noise = something's wrong. - Feature maps via forward hooks show the hierarchy: edges → textures → parts. - Grad-CAM = gradient-weighted feature maps: importance α_k = global-averaged ∂y^c/∂A^k, then ReLU(Σ α_k A^k). - Heatmaps show correlation, not understanding. Pair with quantitative analysis; watch for spurious correlations (tubes, tokens, rulers). - Visualize to debug first, to publish second.
Chapter 11 — CNNs for Your Domain: Medical, Agriculture, Satellite
The core recipe — pretrained backbone, adapted head, augmentation, tuned training — is the same everywhere. What changes is the data's structure and the domain's constraints. This chapter adapts the recipe for three research-heavy domains.
Medical imaging
Data realities: small datasets (hundreds to low thousands), grayscale or specialized modalities (X-ray, CT, MRI, histopathology), expert annotations that are expensive and sometimes disagree, strict patient-level splitting, and high stakes for errors.
Recipe adjustments:
- Backbone: start from ImageNet-pretrained ResNet-50/EfficientNet anyway — early filters transfer. For radiology specifically, consider RadImageNet-pretrained weights (domain-specific pretraining beats ImageNet here). For 3D volumes (CT/MRI), use 3D CNNs (nn.Conv3d) or 2.5D approaches (multiple slice orientations) — a 2D ImageNet backbone on slices is a legitimate, common baseline.
- Splitting: split by patient, never by image. Same-patient images in train and test is leakage, and it's the most common fatal flaw in medical-imaging papers.
- Augmentation: conservative. Small rotations (±10°), mild crops, intensity jitter. No horizontal flips for anatomy with left-right asymmetry (chest X-ray: the heart's position matters). No elastic deformations unless validated — they can create anatomically impossible images.
- Metrics: accuracy is nearly useless under class imbalance. Report AUC, sensitivity/recall, specificity, F1, and show the ROC curve. For segmentation (tumors, organs), the U-Net architecture (Ronneberger et al., 2015) — an encoder-decoder with skip connections — is the default; report Dice coefficient and IoU.
- Interpretability: Grad-CAM figures are close to mandatory for clinical credibility (Chapter 10), plus subgroup analysis (does performance hold across age groups, scanners, hospitals?).
# U-Net sketch: encoder-decoder with skip connections (segmentation default)
class UNetBlock(nn.Module):
def __init__(self, in_ch, out_ch):
super().__init__()
self.block = nn.Sequential(
nn.Conv2d(in_ch, out_ch, 3, padding=1), nn.ReLU(inplace=True),
nn.Conv2d(out_ch, out_ch, 3, padding=1), nn.ReLU(inplace=True))
def forward(self, x): return self.block(x)
# Encoder downsamples (maxpool), decoder upsamples (transpose conv),
# skip connections concatenate encoder features into the decoder at each scale.
Agriculture
Data realities: field photographs with messy backgrounds, strong illumination/weather variation, seasonal shifts (a model trained on spring wheat fails on summer wheat), class imbalance (healthy >> diseased), and deployment on phones or drones in low-connectivity areas.
Recipe adjustments:
- Backbone: EfficientNet-B0/B3 or MobileNetV3 — accuracy per FLOP matters because inference happens on edge devices. Leaf-disease classification papers routinely hit 95%+ with these.
- Augmentation: aggressive photometric augmentation (brightness/contrast/hue — sunlight varies enormously), moderate geometric. MixUp/CutMix help with cluttered backgrounds. Season-aware validation: hold out an entire season or field site as the test set — random splits overestimate real performance because same-field images are near-duplicates.
- Deployment thinking: quantize (INT8) and benchmark on the actual phone; "98% accuracy, runs at 40 ms on a mid-range Android" is a complete result. Consider the offline constraint — the model must ship in the app.
- Beyond classification: counting (fruit, plants) uses density-map regression or detection (Faster R-CNN, YOLO variants); weed detection is often framed as segmentation. Match the task formulation to the agronomist's actual question — "is this leaf diseased?" vs. "what fraction of the field is affected?" are different tasks.
Satellite / remote sensing
Data realities: multispectral input (RGB + near-infrared + more; Sentinel-2 has 13 bands), enormous images (tiles of 10,000×10,000 px — you train on patches), no canonical orientation, georeferencing metadata, and labels that are scarce and noisy.
Recipe adjustments:
- Input channels: adapt conv1 for N bands (Chapter 7, Option B). Don't throw away the infrared band — NDVI-like vegetation signals live there, and ImageNet-pretrained filters on RGB still transfer for the RGB subset.
- Patch-based training: tile big scenes into 256×256 or 512×512 patches; at inference, run sliding-window with overlap and stitch (average the overlaps). For dense prediction (land cover), encoder-decoder architectures (U-Net, DeepLabV3+) are standard.
- Augmentation: full 360° rotations and flips are safe (no canonical orientation); scale jitter simulates different ground resolutions. Spatial splitting: split train/test by geographic region, not randomly — nearby patches are autocorrelated, and random splits leak.
- Receptive field matters: land-cover classes (urban vs. forest) need large context — verify your network's RF covers the discriminative scale (Chapter 3), or use dilated convolutions / DeepLab-style ASPP (atrous spatial pyramid pooling) for multi-scale context.
- Pretraining: ImageNet transfer works, but remote-sensing-specific pretraining (e.g., SSL on unlabeled Sentinel-2 tiles, or the SatMAE line of work) increasingly wins. "In-domain pretraining beats ImageNet on our task" is a publishable finding.
The domain-adaptation checklist (all three)
- [ ] Split by the right unit (patient / field-season / geographic region), not by image
- [ ] Augmentation policy reviewed against domain invariances (no lying transforms)
- [ ] Metrics match the domain (AUC/Dice, not raw accuracy)
- [ ] Backbone input adapted (channels, resolution) with initialization stated
- [ ] Visualizations inspected by a domain eye (radiologist, agronomist) before publication
- [ ] Deployment constraints (latency, device, offline) measured, not assumed
For your research. The fastest path to a solid applied paper: take the standard recipe, adapt it carefully to one domain using the checklist above, and report honestly — including where the standard recipe failed and what you changed. "ImageNet pretraining underperformed in-domain SSL by 4 points on our histopathology task" is more interesting than another 99%-accuracy leaf classifier with a random split. Domain reviewers reward domain competence: cite the domain's datasets and prior art, not just the vision papers.
3D volumes, in-domain pretraining, and multi-task heads
3D CNNs for volumetric data. CT and MRI are volumes, not images. The honest approach is 3D convolution — the same operation with one more dimension:
conv3d = nn.Conv3d(in_channels=1, out_channels=32, kernel_size=3, padding=1)
x = torch.randn(1, 1, 64, 128, 128) # (B, C, D, H, W)
print(conv3d(x).shape) # (1, 32, 64, 128, 128)
Costs explode cubically: a 3×3×3 kernel has 27 weights per channel pair (versus 9), and volumes are large. Practical compromises: (a) 2.5D — run axial, coronal, and sagittal slices through a 2D network and fuse the predictions, which keeps ImageNet pretraining usable; (b) anisotropic kernels (3×3×1) when slice spacing differs from in-plane resolution — extremely common in CT, where voxels aren't cubes. Report voxel-spacing handling explicitly: "resampled to 1mm³ isotropic" is a methods sentence medical reviewers look for, and its absence is a common criticism.
In-domain self-supervised pretraining. When your domain differs strongly from ImageNet (histopathology, satellite), pretrain on your own unlabeled images with contrastive learning, then fine-tune on labels. The lightly library makes SimCLR nearly turnkey:
# pip install lightly
from lightly.models import ResNetGenerator
from lightly.loss import NTXentLoss
backbone = ResNetGenerator('resnet-50') # train on unlabeled domain images
criterion = NTXentLoss(temperature=0.5) # then attach your head, fine-tune
The publishable pattern: "in-domain SSL pretraining on 50K unlabeled tiles, then fine-tuning, beats ImageNet transfer by 3.1 points." Unlabeled data is usually abundant; the cost is one extra pretraining run — a fair trade for a clean, honest contribution that directly addresses the domain gap (Chapter 7's negative transfer).
Multi-task heads. Real deployments rarely need one prediction: a field app wants disease class and severity; a radiology tool wants the finding and its location. Share the backbone, attach multiple heads, train with a weighted loss sum. Benefits: shared features regularize each task (auxiliary tasks act as inductive bias), and one model ships instead of three. Start with equal loss weights, tune by per-task validation — or use uncertainty weighting (Kendall et al., 2018) to learn the balance automatically. Report per-task metrics separately; a multi-task paper lives or dies on showing that no task got worse.
Datasheets for datasets. Gebru et al. (2021) proposed documenting datasets like hardware datasheets: collection protocol, annotator expertise and agreement, known biases, intended uses. For domain papers, a half-page dataset description (or appendix datasheet) preempts the reviewer's hardest questions — "who labeled this, and how well did they agree?" Report inter-annotator agreement (Cohen's κ) for your labels: if experts disagree 15% of the time, a 95% model-accuracy claim needs that context, and providing it signals unusual seriousness.
Two end-to-end project plans
Project plan A: leaf-disease classifier (agriculture). Dataset: 12,000 field photos, 8 disease classes + healthy, heavy imbalance. (1) Split by field site, holding out 2 entire sites as test — random splits would leak near-duplicate images. (2) Backbone: EfficientNet-B0 pretrained (ImageNet), input 224, grayscale? No — keep RGB, disease coloration matters. (3) Augmentation: flips, crops (0.5–1.0), strong brightness/contrast jitter, ±30° rotation, CutMix; no hue jitter (disease coloration is signal). (4) Training: weighted sampler, AdamW 3e-4 backbone-staged fine-tuning (Chapter 7's ladder), label smoothing, 3 seeds. (5) Metrics: per-class F1 + confusion matrix + Grad-CAM review by a plant pathologist. (6) Deployment: INT8 quantization, benchmark on a mid-range Android, report ms/image. Timeline: 2 weeks data curation, 1 week training/ablations, 1 week figures + writing. This plan is a workshop paper.
Project plan B: land-cover segmentation (satellite). Dataset: Sentinel-2 tiles, 10 bands, 6 land-cover classes, labels from a national survey (noisy). (1) Preprocess: resample bands to 10m, normalize per-band, tile into 256×256 patches with 32px overlap. (2) Architecture: U-Net with ResNet-34 encoder; adapt conv1 to 10 channels (RGB weights kept, new bands from channel mean — Chapter 7). (3) Split by geographic region (spatial autocorrelation makes random splits leak). (4) Augmentation: 360° rotations, flips, scale jitter; per-band brightness only. (5) Loss: cross-entropy + Dice, class-weighted for rare classes (wetlands). (6) Pretraining comparison: ImageNet transfer vs in-domain SSL on 100K unlabeled tiles — the publishable question. (7) Inference: sliding window with overlap averaging, stitch to full tiles; report mIoU + per-class IoU. (8) Figures: prediction overlays on 3 diverse regions including one failure case (cloud shadows — honest limitation). Timeline: 3 weeks (tiling and SSL pretraining dominate). This plan is a conference paper's core.
The meta-lesson. Both plans are the same skeleton: domain-aware split → adapted backbone → honest augmentation → staged training → domain metrics → expert-reviewed visualization → deployment measurement. Memorize the skeleton; the domain fills in the parameters. When your supervisor asks "what's the plan," this — one page, six steps, with the domain-specific choices filled in — is the answer.
How much labeled data do you need? (Learning curves decide)
Don't guess — measure. Train on 10%, 25%, 50%, 100% of your labeled data (stratified subsets, same protocol) and plot accuracy vs. data size. The curve's shape tells you the strategy:
- Still rising steeply at 100% → labeling more data is your highest-ROI move. Budget for annotation before architecture experiments.
- Flattening → more labels won't help much; invest in augmentation, pretraining, or harder negatives.
- Flat from 10% → the task is either trivial (a linear probe would do) or the labels are noisy — check annotator agreement before anything else.
Annotation budgeting math. If labeling costs $2/image (expert) and the curve suggests you need 3× data for +2%, that's a concrete proposal: "$12K for 6,000 labels → ~2% gain" versus "2 weeks of architecture experiments → uncertain gain." Supervisors and grant reviewers think in these terms. Also consider active learning: label the images the current model is most uncertain about — often 2–3× more sample-efficient than random labeling. And weak supervision: cheap noisy labels (heuristics, reports) for pretraining + a small clean set for fine-tuning — the standard playbook when expert labels cost dollars each.
The 1,000-image rule of thumb. With transfer learning: <300 images → linear probe territory, expect modest results; 300–2,000 → staged fine-tuning shines; 2,000–20,000 → fine-tuning + augmentation, competitive results; >20,000 → from-scratch becomes viable, SSL pretraining worth it. Calibrate with your learning curve, but use the rule to set expectations before the first run.
Augmentation and the validation gap: a diagnostic note
A subtle interaction: heavy augmentation inflates the train–validation gap even in a healthy model, because the training images are harder versions of the validation images. Don't misread this as overfitting. The check: evaluate the training set with validation transforms (no augmentation) — if that accuracy matches validation accuracy, the gap is augmentation difficulty, not memorization, and all is well. True overfitting shows as a gap even without augmentation. This two-evaluation trick (train-with-train-transforms vs train-with-val-transforms) takes one extra evaluation loop and resolves the ambiguity completely. Log both numbers; the difference quantifies your augmentation's difficulty contribution — occasionally an interesting paper footnote in itself.
Key takeaways - Medical: patient-level splits, conservative augmentation (no flips on asymmetric anatomy), AUC/Dice metrics, U-Net for segmentation, Grad-CAM expected. - Agriculture: edge-efficient backbones, aggressive photometric augmentation, season/site-held-out validation, deployment latency as a result. - Satellite: N-channel conv1 adaptation, patch training + stitched inference, rotation-safe augmentation, geographic splits, large receptive fields. - The universal checklist: right split unit, honest augmentation, domain metrics, adapted inputs, domain-expert eyeballs, measured deployment.
Chapter 12 — From CNN Experiments to a Paper
What reviewers actually check
A vision paper's credibility rests on a short list: (1) baselines that were actually tuned, (2) ablations showing each component earns its place, (3) statistics (seeds, variance), (4) a methods section detailed enough to reproduce, and (5) honest limitations. Everything in this chapter serves that list.
Ablations: proving each piece matters
An ablation study removes (or swaps) one component at a time and measures the damage. It's how you answer "was it the architecture, the augmentation, or the pretraining?" Design ablations before running them — decide the question each row answers:
| Variant | Accuracy (%) | Δ vs full |
|---|---|---|
| Full model (EfficientNet-B0, fine-tuned, aug) | 87.3 ± 0.4 | — |
| − pretraining (from scratch) | 81.1 ± 0.7 | −6.2 |
| − augmentation (no aug) | 84.0 ± 0.5 | −3.3 |
| − fine-tuning (frozen backbone) | 85.6 ± 0.3 | −1.7 |
| Backbone → ResNet-50 | 86.9 ± 0.5 | −0.4 |
Each row isolates one decision. The Δ column is the story: pretraining mattered most, augmentation second. Rules: change exactly one thing per row; keep everything else (seeds, schedule, epochs) fixed; run 3+ seeds per row. Yes, ablations multiply compute — budget for them from the start. A paper with no ablations reads as "we tried things until a number went up," which is exactly what reviewers suspect.
Tables: the results section's backbone
Main results table conventions:
- Rows: methods (yours + baselines), grouped logically. Baselines first, yours last (or bolded).
- Columns: metrics appropriate to the task (accuracy, F1, AUC — never accuracy alone under imbalance), plus params and FLOPs when efficiency is claimed.
- Every number: mean ± std over ≥3 seeds. A bare "87.3" with no variance is an anecdote.
- Bold the best per column; underline second-best (common convention).
- Footnotes: mark TTA, ensembling, or any inference-time trickery. "† uses 5-crop TTA."
| Method | Backbone | Params | F1 (%) | AUC |
|---|---|---|---|---|
| Baseline (frozen) | ResNet-50 | 23.5M | 78.2 ± 0.6 | 0.891 ± 0.004 |
| Fine-tuned | ResNet-50 | 23.5M | 83.4 ± 0.4 | 0.923 ± 0.003 |
| Ours | EfficientNet-B3 | 12.2M | **85.1 ± 0.3** | **0.936 ± 0.002** |
Reporting checklist (methods section)
A reproducible methods section states: dataset (source, size, split ratios, split unit — patient/field/region), preprocessing (resize, normalization values), architecture (backbone, weight source + version, head design), training (optimizer, LR, schedule, epochs, batch size, augmentation policy with parameters, regularization, seeds, hardware, training time), evaluation (metrics, TTA if used, statistical treatment). If any item is missing, a reviewer can't reproduce your work — and will say so.
Figures that strengthen the paper
- Architecture diagram: one clean figure of your model (boxes + tensor shapes at each stage). Readers decide whether to keep reading based on this figure.
- Training curves: train/val loss + metric, showing convergence and the absence of overfitting drama.
- Qualitative results: best/worst Grad-CAM or segmentation overlays — including a failure case with a one-sentence hypothesis ("fails on low-contrast lesions; future work: contrast-aware augmentation").
- Ablation/confusion visuals: confusion matrix for classification; it shows which classes confuse the model, which is often the most interesting result.
Limitations and ethics (write them before the camera-ready)
Every paper needs an honest limitations paragraph: dataset biases (single hospital, single season, single geography), failure modes you observed, compute constraints that limited ablations, and the gap between benchmark and deployment. For human-impact domains (medical, agriculture-as-livelihood), add the ethical note: intended use, who could be harmed by errors, and why the system is assistive rather than autonomous. Reviewers — and increasingly, conferences — expect this. It also happens to be good science.
The one-week paper sprint (a realistic plan)
- Days 1–2: final baselines + ablations (seeds running in parallel).
- Day 3: figures (architecture, curves, Grad-CAM, confusion matrix).
- Day 4: results + methods draft (tables first, prose follows the tables).
- Day 5: intro + related work (related work is easier once you know exactly what your contribution is).
- Day 6: limitations, abstract, full read-through; check every number against logs.
- Day 7: co-author review buffer.
The golden rule: tables before prose. Write the results tables first; the story they tell is your paper. Prose written before the numbers exist always needs rewriting.
For your research. Start a "paper ledger" text file on day one of any project: every experiment gets one line (date, config, seed, result, one-line interpretation). When it's time to write, the ledger becomes your ablations and tables with zero archaeology. And version your datasets — a results table is meaningless if you can't say exactly which images were in the test split six months later. These two habits (ledger + dataset versioning) separate researchers whose work reproduces from those whose work doesn't.
Statistics, code release, and surviving peer review
Beyond mean ± std: is the difference real? With three seeds each, "85.1 ± 0.3 vs 84.6 ± 0.4" might be noise. The cheap rigorous options: (a) paired bootstrap — resample the test set 1,000 times, compute the metric difference each time, and report the 95% confidence interval of the difference; if it excludes zero, claim the win; (b) more seeds (5–10) when compute allows — variance shrinks as 1/√n.
import numpy as np
rng = np.random.default_rng(0)
diffs = []
for _ in range(1000):
idx = rng.integers(0, len(y_true), len(y_true))
diffs.append(metric(y_true[idx], pred_a[idx]) - metric(y_true[idx], pred_b[idx]))
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"Δ = {np.mean(diffs):.3f}, 95% CI [{lo:.3f}, {hi:.3f}]")
If the interval includes zero, write "no significant difference" — honesty here is what separates research from marketing, and reviewers can tell which one they're reading.
Code and weight release checklist. "Code will be released" with no link increasingly irritates reviewers. Release at submission (anonymized) or at camera-ready at the latest: the training script, exact configs and hyperparameters, the environment file, pretrained weights, the figure-generation notebook, and a README that reproduces the main table in the stated hardware and time. Scrub hardcoded paths and credentials first. Your future self — reproducing this work for the journal extension — will thank you even more than the reviewers will.
Responding to reviewers. The response letter is a genre with conventions: numbered responses mirroring the reviewers, each opening with what you did ("We added the requested ResNet-101 baseline; see Table 3, row 4"), quoting changed text where relevant, and conceding gracefully where the reviewer is right ("We agree the claim was overstated; we softened it on p. 4"). Never try to argue a reviewer into submission — persuade with new experiments instead. The most common revision requests for CNN papers are stronger baselines, ablations, variance reporting, and dataset details — exactly what this chapter told you to prepare before submission. A paper written to this chapter's checklist tends to get "accept with minor revisions" instead of "reject: insufficient evaluation."
Venue strategy, in one paragraph. Workshops (fast, short papers) are for early results and feedback; conferences (CVPR/ICCV/ECCV, MICCAI for medical, or top domain venues) are for archival impact — note that the rebuttal phase rewards precisely the extra experiments this book teaches you to run; journals (TPAMI, Medical Image Analysis) are for extended, mature work with full ablations. Match the venue to the result's maturity, not to your impatience: a premature conference submission that gets rejected costs more time than a workshop paper that earns useful feedback.
Writing related work, the abstract, and a reproducibility checklist
Related work as positioning, not listing. The failure mode: a chronological dump of every CNN paper since 2012. The fix: organize by the question your paper answers. "Prior work improves accuracy via (a) architectures, (b) pretraining, (c) augmentation. We show that on [domain], (b) dominates and (a) is saturated — contrary to [cite], who found...". Each paragraph ends with the gap your work fills. Cite the originals (this book's references give you the canonical twelve), not the survey that cited them — reviewers check. And cite competitors fairly: the paper you're beating is doing you the favor of existing; misrepresenting it is the fastest way to get a hostile review from its authors.
The abstract formula. Five sentences: (1) the problem and why it matters (one clause of domain motivation); (2) what prior work misses; (3) what you did (method, one sentence); (4) the key result with numbers ("92.1 ± 0.3% F1, +3.4 over the strongest baseline"); (5) the implication or released artifact. Write it last, even though it appears first — it's a summary of the tables, and the tables come first (Chapter 12's golden rule).
Reproducibility checklist (paste into your repo's README). - [ ] Dataset: source, version/hash, exact split files or split code with seed - [ ] Preprocessing: resize, normalization values, channel handling - [ ] Architecture: backbone name, weight source + version, head definition - [ ] Training: optimizer, LR, schedule, epochs, batch size (effective, incl. accumulation), augmentation policy with parameters, regularization, seeds - [ ] Hardware + software: GPU type, CUDA/PyTorch/timm versions, training time per run - [ ] Evaluation: metrics, TTA policy if used, statistical treatment (seeds, CI) - [ ] Checkpoints: final weights + best-validation weights, with config to reload them - [ ] Figures: script/notebook regenerating every figure from logged outputs
If a stranger with your repo and a GPU can't reproduce your main table in the stated time, the checklist found the gap before the reviewers did. This is the unglamorous work that separates papers people build on from papers people cite and ignore.
Reviewer comments you'll get (and the fixes)
| Comment | What they mean | Fix |
|---|---|---|
| "Baselines are weak" | Your method beats an untuned strawman | Tune baseline LR/schedule; add a standard strong baseline (ResNet-50/EfficientNet) |
| "No ablation" | Can't tell what causes the gain | One-variable-per-row table (this chapter); budget compute upfront |
| "Single seed" | Result might be luck | 3+ seeds, mean ± std; bootstrap CI for close calls |
| "Dataset details missing" | Can't judge or reproduce | Split unit, sizes, annotator agreement, preprocessing — full checklist |
| "Incremental" | Contribution unclear | Sharpen the claim: one sentence on what we now know; lead with the ablation that proves it |
| "Why this architecture?" | Choice unjustified | RF/param/FLOP reasoning (Chapters 1–3) or a 2-rung comparison |
| "Overclaims" | Text exceeds evidence | Soften verbs (suggests→demonstrates only with stats); add limitations paragraph |
| "No failure analysis" | Only successes shown | Confusion matrix + Grad-CAM failure case + one-sentence hypothesis each |
| "Code?" | Reproducibility doubt | Release repo + weights + figure scripts at submission |
| "Related work incomplete" | Missed key prior art | Position by idea-lineage (Chapter 4's timeline); cite originals, not surveys |
Keep this table next to the manuscript while drafting. Every row is a rejection reason you can eliminate before submission — which is the entire thesis of this chapter: papers are accepted in the months before writing, through the experiments you chose to run.
Failure-mode gallery (start yours on day one)
Keep a running document of every informative failure with the image, prediction, Grad-CAM, and your hypothesis: (a) low-contrast lesions missed — hypothesis: augmentation never simulated them; fix: contrast jitter; (b) distant-field wheat classified as barley — hypothesis: texture-only decision (receptive field too small for head morphology); fix: larger input resolution; (c) night-time satellite tiles misclassified — hypothesis: no night examples in training; fix: collect or synthesize. Three columns — observation, hypothesis, fix — reviewed weekly. Half of these become ablation rows; one or two become the paper's limitations section; the gallery itself becomes your expertise. Researchers who keep failure galleries debug in days what takes others weeks.
Key takeaways - Reviewers check: tuned baselines, ablations, seed statistics, reproducible methods, honest limitations. - Ablations change one thing per row, fixed everything else, 3+ seeds. Budget compute for them upfront. - Tables: baselines then yours, mean ± std always, params/FLOPs when relevant, footnotes for TTA/ensembles. - Methods must state: data, splits, preprocessing, architecture + weights, training recipe, evaluation, hardware. - Tables before prose. Keep a paper ledger from day one. Write limitations honestly — it's good science and good strategy.
Learning Dashboard
Architecture comparison
| Architecture | Year | Params | FLOPs (224px) | ImageNet top-1* | Best use-case |
|---|---|---|---|---|---|
| LeNet-5 | 1998 | 60K | ~0.4M | n/a | Teaching; tiny grayscale tasks |
| AlexNet | 2012 | 60M | 0.7G | 63.3% | Historical baseline; teaching |
| VGG-16 | 2014 | 138M | 15.5G | 71.6% | Feature extractor; style transfer |
| ResNet-18 | 2015 | 11.7M | 1.8G | 69.8% | Fast baseline; small data |
| ResNet-50 | 2015 | 25.6M | 4.1G | 76.1–80.2% | Default research backbone |
| ResNet-152 | 2015 | 60M | 11.6G | 77.3–81.3% | Max-accuracy ResNet |
| MobileNetV3-Small | 2019 | 2.5M | 0.06G | 67.4% | Phones; real-time edge |
| MobileNetV3-Large | 2019 | 5.4M | 0.22G | 75.2% | Edge with headroom |
| EfficientNet-B0 | 2019 | 5.3M | 0.39G | 77.1% | Strong tiny baseline |
| EfficientNet-B4 | 2019 | 19M | 4.2G | 82.9% | 1-GPU accuracy work |
| EfficientNet-B7 | 2019 | 66M | 37G | 84.3% | Max accuracy, big compute |
*Top-1 varies by weight version; torchvision v2 weights are higher than v1. Always report the weight version you used.
Augmentation decision table
| Data property | Safe augmentations | Avoid | Why |
|---|---|---|---|
| Natural photos, symmetric-ish classes | H-flip, crops, color jitter, ±15° rot | V-flip | Vertical flips unnatural for most scenes |
| Text / digits | Tiny shifts, mild blur | Any flip/rotation | Flips change the label ("3" vs "Ɛ") |
| Chest X-ray / asymmetric anatomy | Small crops, intensity jitter | H-flip, elastic | Heart position is diagnostic |
| Histopathology | H/V flip, 90° rotations, stain jitter | Heavy color jitter | Stain color carries signal |
| Satellite tiles | 360° rotation, flips, scale jitter | Perspective warp | No canonical orientation; warp unphysical |
| Small objects / detection | Constrained crops (≥0.5 scale) | Aggressive RandomResizedCrop | Object can be cropped out entirely |
| Color-critical (ripeness, staining) | Geometric only | Hue jitter | Hue is the label signal |
| Rare classes, imbalanced | Oversample + mild aug | MixUp/CutMix (careful) | Blending drowns rare-class signal |
Hyperparameter cheat sheet
| Setting | From scratch | Fine-tuning (backbone) | New head |
|---|---|---|---|
| Optimizer | AdamW | AdamW or SGD+mom | AdamW |
| Learning rate | 1e-3 – 3e-3 | 1e-5 – 1e-4 | 1e-3 |
| Weight decay | 0.05 | 1e-4 (SGD) / 0.01 (AdamW) | 0.01 |
| Schedule | Cosine, T_max=epochs | Cosine or OneCycle | Cosine |
| Warmup | 5–10% of steps | Always (esp. fresh head) | 5% of steps |
| Batch size | As large as fits | ≥16 for BatchNorm | As large as fits |
| Epochs | 100–300 | 20–50 | 5–10 (head-only stage) |
| Label smoothing | 0.1 | 0.1 | 0.1 |
| Dropout (head) | 0.2–0.5 | 0.2–0.5 | 0.5 |
| Early stopping patience | 15–20 epochs | 10 epochs | 5 epochs |
One-page training recipe (tear out and keep)
Setup. Backbone: ResNet-50 (default) or EfficientNet-B0 (limited compute); weights: torchvision IMAGENET1K_V2 / timm pretrained — record exact version. Input: resize per backbone (224 / 380), normalize with ImageNet mean/std when using ImageNet weights. Split by the right unit (patient / site / region), stratified, seed fixed.
Augmentation. Train: RandomResizedCrop + horizontal flip (p=0.5, unless domain forbids) + mild color/intensity jitter + one advanced method (RandAugment or CutMix). Validate: deterministic resize + center crop. Inspect 64 augmented samples before training.
Optimization. AdamW, lr 3e-4 (from scratch) / 1e-4 backbone + 1e-3 head (fine-tune); weight decay 0.05 / 0.01; cosine schedule with 5–10% warmup; mixed precision on; label smoothing 0.1; early stopping patience 10–20 on validation loss.
Protocol. Stage 1: train head only (5–10 epochs). Stage 2: unfreeze, discriminative LRs, 20–50 epochs. 3+ seeds, mean ± std. Ablate: pretraining, augmentation, backbone — one row per decision.
Evaluation. Metrics for the domain (F1/AUC/Dice, not just accuracy); confusion matrix; Grad-CAM on 20+ samples incl. failures; bootstrap CI for close comparisons; report params, FLOPs, and measured latency.
Before submitting. Paper ledger complete; dataset versioned; code + weights + figure scripts released; limitations paragraph written; every number in the paper traced to a log line.
Appendix A — Complete Training Loop Template
The following is a minimal but complete training script incorporating this book's recommendations. Adapt it; don't rewrite from memory each time.
import torch, torch.nn as nn
from torch.utils.data import DataLoader
from torch.amp import GradScaler, autocast
from torchvision import models, transforms
# ---------- config (log all of these) ----------
NUM_CLASSES, EPOCHS, BATCH, SEED = 10, 50, 32, 0
torch.manual_seed(SEED)
train_tf = transforms.Compose([
transforms.RandomResizedCrop(224, scale=(0.5, 1.0)),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(0.2, 0.2, 0.2, 0.05),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
val_tf = transforms.Compose([
transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
# train_ds / val_ds built with the right split unit (patient/site/region)
# ---------- model ----------
model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)
model.cuda()
# Stage 1: head only
for p in model.parameters(): p.requires_grad = False
for p in model.fc.parameters(): p.requires_grad = True
opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3, weight_decay=0.01)
def run_epoch(loader, train):
model.train(train)
tot_loss, correct, n = 0.0, 0, 0
for x, y in loader:
x, y = x.cuda(), y.cuda()
with autocast(device_type='cuda'), torch.set_grad_enabled(train):
out = model(x)
loss = nn.CrossEntropyLoss(label_smoothing=0.1)(out, y)
if train:
opt.zero_grad(); scaler.scale(loss).backward()
scaler.step(opt); scaler.update()
tot_loss += loss.item() * len(x); correct += (out.argmax(1) == y).sum().item(); n += len(x)
return tot_loss / n, correct / n
scaler = GradScaler()
train_loader = DataLoader(train_ds, batch_size=BATCH, shuffle=True, num_workers=4)
val_loader = DataLoader(val_ds, batch_size=BATCH, num_workers=4)
for ep in range(5):
tr_loss, tr_acc = run_epoch(train_loader, True)
va_loss, va_acc = run_epoch(val_loader, False)
print(f"[head] ep{ep} train {tr_loss:.3f}/{tr_acc:.3f} val {va_loss:.3f}/{va_acc:.3f}")
# Stage 2: fine-tune all with discriminative LRs
for p in model.parameters(): p.requires_grad = True
opt = torch.optim.AdamW([
{'params': model.layer4.parameters(), 'lr': 1e-4},
{'params': model.fc.parameters(), 'lr': 1e-3},
], lr=1e-5, weight_decay=0.01)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=EPOCHS)
best, patience, wait = float('inf'), 10, 0
for ep in range(EPOCHS):
tr_loss, tr_acc = run_epoch(train_loader, True)
va_loss, va_acc = run_epoch(val_loader, False)
sched.step()
print(f"ep{ep} train {tr_loss:.3f}/{tr_acc:.3f} val {va_loss:.3f}/{va_acc:.3f}")
if va_loss < best:
best, wait = va_loss, 0
torch.save({'model': model.state_dict(), 'epoch': ep}, 'best.pt')
else:
wait += 1
if wait >= patience:
print("early stopping"); break
Deliberate choices encoded above: staged fine-tuning, discriminative LRs, AMP, label smoothing, cosine schedule, early stopping on validation loss with best-checkpoint saving. The main things it omits (add as needed): weighted sampling for imbalance, gradient accumulation for small GPUs, LR warmup (add a linear warmup scheduler for the first epochs), and multi-seed looping (wrap in a for seed in [0,1,2] shell loop).
Appendix B — Notation and Shape Conventions
- Tensors: NCHW — (batch, channels, height, width). A "224px RGB batch of 32" is (32, 3, 224, 224).
- Conv2d weight shape: (C_out, C_in, K_h, K_w). The i-th output channel's kernel is
weight[i]. - "3×3 conv, 64→128" means kernel 3×3, 64 input channels, 128 output channels: params = 128 × (9×64 + 1).
- RF: receptive field; s: stride; P: padding. Output size: floor((W − K + 2P)/S) + 1.
- lr: learning rate; wd: weight decay; TTA: test-time augmentation; GAP: global average pooling.
- "Frozen" =
requires_grad=False; "eval mode" =model.eval()(affects BN/dropout, independent of freezing). - Seeds: report data seed and init seed separately when diagnosing variance.
Further Reading
- I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016 — Chapters 9 (CNNs) and 7–8 (regularization/optimization). The free online version remains the best single reference.
- C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, "Understanding deep learning requires rethinking generalization," in Proc. ICLR, 2017 — the memorization paper cited in Chapter 1; short and mind-changing.
- S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in Proc. ICML, 2015 — the other half of why deep nets train.
- J. Johnson, "CNN lecture notes," Stanford CS231n — the clearest free course notes on CNN mechanics; work its assignments if you want implementation fluency.
- O. Russakovsky et al., "ImageNet large scale visual recognition challenge," Int. J. Comput. Vis., vol. 115, pp. 211–252, 2015 — the dataset paper behind everything in Chapters 4–6; read it to understand what ImageNet pretraining actually transfers.
- M. Gebru et al., "Datasheets for datasets," Commun. ACM, vol. 64, no. 12, pp. 86–92, 2021 — the dataset-documentation proposal from Chapter 11.
- K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification," in Proc. ICCV, 2015 — He initialization, cited in Chapter 1's expansion.
References
[1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Adv. Neural Inf. Process. Syst. 25, Lake Tahoe, NV, USA, 2012, pp. 1097–1105.
[3] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, 2015.
[4] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, 2016, pp. 770–778.
[5] K. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks," in Proc. Eur. Conf. Comput. Vis. (ECCV), Amsterdam, Netherlands, 2016, pp. 630–645.
[6] A. G. Howard et al., "MobileNets: Efficient convolutional neural networks for mobile vision applications," arXiv:1704.04861, 2017.
[7] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "MobileNetV2: Inverted residuals and linear bottlenecks," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Salt Lake City, UT, USA, 2018, pp. 4510–4520.
[8] M. Tan and Q. V. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," in Proc. Int. Conf. Mach. Learn. (ICML), Long Beach, CA, USA, 2019, pp. 6105–6114.
[9] M. Lin, Q. Chen, and S. Yan, "Network in network," in Proc. Int. Conf. Learn. Represent. (ICLR), Banff, Canada, 2014.
[10] R. R. Selvaraju et al., "Grad-CAM: Visual explanations from deep networks via gradient-based localization," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Venice, Italy, 2017, pp. 618–626.
[11] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in Proc. Med. Image Comput. Comput.-Assist. Interv. (MICCAI), Munich, Germany, 2015, pp. 234–241.
[12] M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in Proc. Eur. Conf. Comput. Vis. (ECCV), Zurich, Switzerland, 2014, pp. 818–833.
Glossary
- Activation map / feature map: The 2-D output grid produced by applying one kernel across the input; a layer's output volume stacks one map per kernel.
- Batch normalization: Layer that normalizes each channel to zero mean/unit variance per mini-batch (with learned scale/shift); stabilizes and accelerates training.
- Bottleneck block: Residual block using 1×1 → 3×3 → 1×1 convolutions to cut computation in deep ResNets.
- Channel: One 2-D slice of an image or feature volume (e.g., R/G/B); convolutions mix information across channels.
- Compound scaling: EfficientNet's rule for jointly scaling network depth, width, and input resolution.
- Convolution (deep learning): Cross-correlation of a learned kernel over the input — sliding element-wise multiply-and-sum; the name "convolution" is conventional.
- Cross-correlation: The unflipped sliding-dot-product operation that deep learning libraries compute and call convolution.
- Data augmentation: Random transformations of training images (flips, crops, color shifts) that synthesize dataset variation.
- Depthwise separable convolution: Factorized convolution — a per-channel spatial filter followed by a 1×1 channel mixer; ~8–9× cheaper than standard convolution.
- Discriminative learning rates: Using smaller learning rates for early (pretrained) layers and larger ones for late layers/head during fine-tuning.
- Dropout: Regularization that randomly zeroes activations during training to prevent co-adaptation.
- Effective receptive field: The input region that actually influences a neuron — smaller and Gaussian-shaped compared to the theoretical receptive field.
- Feature extraction (transfer): Freezing a pretrained backbone and training only a new classifier head.
- Fine-tuning: Training some or all of a pretrained network's weights on a new task, usually with small learning rates.
- FLOPs: Floating-point operations; a hardware-independent measure of a model's computational cost.
- Global average pooling (GAP): Averaging each feature map to a single value; the standard parameter-free bridge to the classifier.
- Grad-CAM: Visualization weighting a layer's feature maps by their gradient-derived importance for a class, showing influential image regions.
- Inductive bias: Assumptions built into an architecture (e.g., locality and weight sharing in CNNs) that guide learning.
- Kernel / filter: The small learned weight matrix slid across the input in a convolution.
- Label smoothing: Regularization that softens one-hot targets, discouraging overconfident predictions.
- Negative transfer: When pretraining on a source domain hurts (rather than helps) performance on the target domain.
- Padding: Border pixels added around an input so convolution preserves spatial size ("same" padding) or controls shrinkage.
- Pooling: Fixed downsampling operation (max/average) over windows; gives approximate translation invariance, no learned parameters.
- Receptive field: The input-image region that can affect a given neuron's value; grows with depth and stride.
- Residual / skip connection: An identity shortcut adding a block's input to its output (y = F(x) + x); enables very deep networks by preserving gradient flow.
- Stride: The step size of the sliding kernel; stride > 1 downsamples the output.
- Test-time augmentation (TTA): Averaging predictions over augmented versions of each test image for free accuracy gains.
- Translation equivariance: A shift of the input shifts the feature map equally (property of convolution).
- Translation invariance: A shift of the input leaves the output (approximately) unchanged (property of pooling).
- Warmup: Gradually ramping the learning rate from 0 at the start of training to avoid early instability.
- Weight decay: L2 penalty on weights; the baseline regularizer, decoupled from gradients in AdamW.
- Atrous/dilated convolution: Convolution with gaps inserted in the kernel, enlarging the receptive field without extra parameters or resolution loss.
- Backbone: The pretrained convolutional feature extractor (e.g., ResNet-50) to which a task-specific head is attached.
- Catastrophic forgetting: Loss of pretrained features during aggressive fine-tuning on a small dataset.
- Checkpoint: Saved model weights (and optimizer state) during/after training; best-validation checkpointing is standard.
- Concept activation vector (TCAV): Direction in activation space representing a human-defined concept, used to quantify a model's reliance on it.
- Confusion matrix: Table of predicted vs. true classes showing exactly which classes a classifier confuses.
- CutMix: Augmentation pasting a patch from one image onto another, mixing labels by area proportion.
- Effective batch size: Batch size × gradient-accumulation steps; the batch size the optimizer effectively sees.
- Head: Task-specific layers (usually classifier) attached to a pretrained backbone.
- Learning-rate finder: Single-run exponential LR sweep used to pick a training LR (Smith, 2017).
- Linear probing: Training only the classifier head on frozen backbone features; first stage of LP-FT.
- MixUp: Augmentation blending two images and their labels with a random mixing coefficient.
- Quantization: Reducing weight/activation precision (e.g., FP32→INT8) to shrink models and speed inference.
- SyncBatchNorm: Batch normalization with statistics synchronized across GPUs in distributed training.
- Weight tying: Sharing the same parameters across positions or layers (convolution's core mechanism).
Practice Exercises
- Hand convolution. Using the 5×5 input and 3×3 vertical-edge kernel from Chapter 2, compute the output at position (1, 1) by hand, then verify with the PyTorch snippet. Then recompute with stride 2 and padding 1, predicting the output shape first with the sizing formula.
- Parameter audit. For a network with: conv1 (3→64, 7×7, stride 2), maxpool, conv2 (64→128, 3×3), conv3 (128→256, 3×3), GAP, FC (256→10) — compute total parameters by hand using the per-layer formula, then verify in PyTorch. Which layer dominates the parameter count?
- Receptive field mapping. Compute the theoretical receptive field at each stage of the network in Exercise 2 (input 224×224). Is the final RF large enough to cover a 100-pixel object? What would you change if it weren't?
- Build a residual block from scratch. Implement
BasicBlock(Chapter 5) without looking, then stack 8 of them into a tiny ResNet for CIFAR-10. Train it for 20 epochs. Then remove the skip connections and retrain — compare training loss curves and explain the difference. - Research: ablation design. Pick a published CNN paper in your domain. List every design decision the authors made (backbone, augmentation, pretraining, loss). Design a 5-row ablation table that isolates the paper's central claim. Which row would falsify the claim if it showed no drop?
- Transfer learning shootout. Take a small public dataset in your domain (or CIFAR-10 as proxy). Compare three protocols with identical budgets: (a) from-scratch training, (b) frozen-backbone feature extraction, (c) staged fine-tuning with discriminative LRs. Report mean ± std over 3 seeds. Which wins, and by how much per protocol?
- Augmentation forensics. Build the training augmentation pipeline from Chapter 8, save a 64-image grid of augmented samples from your dataset, and inspect every image. Find one augmentation that violates label preservation for your data and justify its removal in two sentences (methods-section style).
- Research: spurious correlation hunt. Train a classifier, then run Grad-CAM on 50 test images. Find at least one case where the heatmap focuses outside the object of interest. Hypothesize the spurious cue, propose a data or augmentation fix, and re-run. Did the heatmap move?
- Efficiency under constraint. You have one GPU and 12 hours. Compare EfficientNet-B0, MobileNetV3-Large, and ResNet-18 (all pretrained, fine-tuned) on your task: report accuracy, params, FLOPs, and measured inference latency on your target device. Write the one-paragraph "model selection" justification for your paper.
- Research: from ledger to paper. Run a 4-experiment mini-project (baseline + 3 ablations, 3 seeds each) on any image task, keeping a paper ledger (date, config, seed, result, interpretation — Chapter 12). Produce: one results table with mean ± std, one training-curve figure, one Grad-CAM figure with an honest caption, and a limitations paragraph. This is a complete paper skeleton — keep the habit.
End of Book 12 — Convolutional Neural Networks for Images · AstolixGen Learning Series