Breaking the Depth Barrier: Understanding Deep Residual Learning (ResNet)
Breaking the Depth Barrier: Understanding Deep Residual Learning (ResNet)
In the early days of Deep Learning, the prevailing wisdom was simple: deeper is better. The logic was intuitive—more layers allow a network to learn more complex, hierarchical features. However, researchers soon hit a wall. As networks grew deeper, they didn't just become harder to train; they actually performed worse than shallower versions.
This is known as the Degradation Problem, and it was solved by Kaiming He and his team through a groundbreaking paper: "Deep Residual Learning for Image Recognition."
In this post, we will dive deep into the intuition behind ResNets, the mathematics of residual mapping, and a production-ready PyTorch implementation.
The Paradox: Why Depth Isn't Always Better
Before ResNet, adding more layers to a neural network often led to a surprising result: higher training error.
Crucially, this wasn't caused by overfitting (where training error is low but test error is high). Instead, the model simply failed to converge. This suggested that the optimization process becomes exponentially more difficult as the network depth increases, leading to the "degradation problem."
The Intuition: Learning the Difference
The authors proposed a shift in perspective. Instead of forcing a stack of layers to learn a completely new underlying mapping $H(x)$, why not let them learn the residual (the difference) between the input and the output?
If the ideal mapping is $H(x)$, we define the residual as: $$F(x) = H(x) - x$$
The original mapping then becomes: $$H(x) = F(x) + x$$
Why does this help? If the optimal mapping is close to an identity function (i.e., the layer doesn't need to change the input much), it is much easier for the network to push the weights of $F(x)$ toward zero than to learn an identity mapping from scratch using multiple non-linear layers.
Architecture Deep Dive
The core of ResNet is the Residual Block. Unlike traditional "plain" networks, ResNet introduces Identity Shortcut Connections that skip one or more layers.
The Mathematical Framework
Depending on the dimensions of the input, ResNet uses two types of connections:
- Identity Shortcut: Used when input and output dimensions match. $$y = F(x, {W_i}) + x$$
- Projection Shortcut: Used when dimensions change (e.g., during downsampling). A linear projection $W_s$ is applied to match the dimensions. $$y = F(x, {W_i}) + W_s x$$
Visualizing the Flow
The following diagram illustrates the macro-architecture of a ResNet and the micro-logic inside a single BasicBlock.
Implementation in PyTorch
Below is a professional implementation of ResNet-18. I have adapted the initial layer for CIFAR-10 (32x32 images) to prevent excessive downsampling at the start.
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from sklearn.metrics import accuracy_score
class BasicBlock(nn.Module):
"""
The fundamental building block of ResNet.
Implements the F(x) + x logic.
"""
def __init__(self, in_planes, planes, stride=1):
super(BasicBlock, self).__init__()
# First convolution: handles downsampling via stride
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
# Second convolution: maintains dimensions
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
# Shortcut connection: matches dimensions if stride > 1 or channel count changes
self.shortcut = nn.Sequential()
if stride != 1 or in_planes != planes:
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, planes, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(planes)
)
def forward(self, x):
residual = x # Save input for identity mapping
out = F.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
# The core ResNet operation: F(x) + x
out += self.shortcut(residual)
out = F.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, num_blocks, num_classes=10):
super(ResNet, self).__init__()
self.in_planes = 64
# Initial layer: Optimized for CIFAR-10 (32x32)
self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(64)
# 4 Stages of residual blocks
self.layer1 = self._make_layer(block, num_blocks[0], 64, stride=1)
self.layer2 = self._make_layer(block, num_blocks[1], 128, stride=2)
self.layer3 = self._make_layer(block, num_blocks[2], 256, stride=2)
self.layer4 = self._make_layer(block, num_blocks[3], 512, stride=2)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.linear = nn.Linear(512, num_classes)
def _make_layer(self, block, num_blocks, planes, stride):
strides = [stride] + [1] * (num_blocks - 1)
layers = []
for s in strides:
layers.append(block(self.in_planes, planes, s))
self.in_planes = planes
return nn.Sequential(*layers)
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = self.layer1(out)
out = self.layer2(out)
out = self.layer3(out)
out = self.layer4(out)
out = self.avgpool(out)
out = torch.flatten(out, 1)
return self.linear(out)
def get_resnet18():
return ResNet(BasicBlock, [2, 2, 2, 2])Key Takeaways & Impact
The introduction of ResNet changed the trajectory of Computer Vision. By solving the degradation problem, it enabled the training of networks with hundreds or even thousands of layers.
Summary of Contributions:
- Overcame the Degradation Problem: Proved that deeper networks could be optimized if they learned residuals.
- Efficiency: ResNet-152 is significantly deeper than VGG-16 but has lower complexity (fewer parameters) due to the use of global average pooling instead of massive fully connected layers.
- Versatility: The residual block is now a standard component in almost every modern architecture, including Transformers (via residual connections around attention layers).
Performance Checklist for Implementation:
| Feature | Implementation Detail | Purpose |
|---|---|---|
| Batch Norm | After every Conv layer | Stabilizes training and allows higher learning rates |
| 1x1 Conv | In the shortcut path | Matches dimensions when stride $\neq 1$ |
| Global Avg Pool | Before the final Linear layer | Reduces parameter count and prevents overfitting |
| ReLU | After addition | Introduces non-linearity to the combined signal |
ResNet didn't just win the ILSVRC 2015 competition; it provided a fundamental blueprint for how to build deep, stable, and scalable neural networks.