Scaling Smarter: Understanding EfficientNet and Compound Scaling
Scaling Smarter: Understanding EfficientNet and Compound Scaling
In the quest for higher accuracy in Computer Vision, the traditional approach has been "bigger is better." Whether it was adding more layers (ResNet), increasing channel width, or feeding in higher-resolution images, the industry focused on scaling a single dimension to push the state-of-the-art.
However, scaling one dimension in isolation quickly leads to diminishing returns. If you increase the resolution of an image but keep the network shallow, the model lacks the receptive field to capture the new, finer details.
Enter EfficientNet. Instead of arbitrary scaling, EfficientNet introduces Compound Scaling—a principled method to scale depth, width, and resolution uniformly.
The Core Intuition: The Synergy of Dimensions
The fundamental thesis of EfficientNet is that network depth, width, and resolution are not independent; they are synergistic.
- Depth ($d$): More layers capture more complex features but are harder to train.
- Width ($w$): More channels capture more fine-grained patterns but can lead to overfitting.
- Resolution ($r$): Higher resolution provides more detail but increases computational cost quadratically.
EfficientNet argues that if the input resolution is increased, the network needs more layers to expand its receptive field and more channels to capture the increased detail.
The Mathematical Framework
Rather than tuning these three hyperparameters manually, EfficientNet uses a single compound coefficient $\phi$ to scale them proportionally:
$$\text{Depth: } d = \alpha^\phi$$ $$\text{Width: } w = \beta^\phi$$ $$\text{Resolution: } r = \gamma^\phi$$
The Constraint: To ensure the computational cost (FLOPs) increases predictably, the constants are constrained such that: $$\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$$ This means that for any $\phi$, the total FLOPs of the model increase by approximately $2^\phi$.
Architectural Blueprint
EfficientNet isn't just about scaling; it starts with a highly efficient baseline (B0) discovered via Neural Architecture Search (NAS). The primary building block is the MBConv (Mobile Inverted Bottleneck Convolution).
The MBConv Block
The MBConv block reduces computation by using Depthwise Separable Convolutions, consisting of three phases:
- Expansion: A $1 \times 1$ convolution expands the channel dimension to create a higher-dimensional feature space.
- Depthwise Convolution: A $3 \times 3$ convolution filters the image spatially.
- Projection: A $1 \times 1$ convolution projects the features back to a lower-dimensional space.
System Workflow
The following diagram illustrates how the compound coefficient $\phi$ flows through the data pipeline and the model architecture.
Implementation in PyTorch
Below is a production-ready implementation of the Compound Scaling logic and the MBConv architecture.
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import math
class MBConv(nn.Module):
"""
Mobile Inverted Bottleneck Convolution (MBConv)
The building block of EfficientNet.
"""
def __init__(self, in_channels, out_channels, expand_ratio, stride):
super(MBConv, self).__init__()
hidden_dim = in_channels * expand_ratio
# 1. Expansion phase (Pointwise Conv)
self.expand = nn.Sequential(
nn.Conv2d(in_channels, hidden_dim, 1, bias=False),
nn.BatchNorm2d(hidden_dim),
nn.SiLU()
) if expand_ratio != 1 else nn.Identity()
# 2. Depthwise phase
self.depthwise = nn.Sequential(
nn.Conv2d(hidden_dim, hidden_dim, 3, stride=stride, padding=1, groups=hidden_dim, bias=False),
nn.BatchNorm2d(hidden_dim),
nn.SiLU()
)
# 3. Projection phase (Pointwise Conv)
self.project = nn.Sequential(
nn.Conv2d(hidden_dim, out_channels, 1, bias=False),
nn.BatchNorm2d(out_channels)
)
def forward(self, x):
res = x
x = self.expand(x)
x = self.depthwise(x)
x = self.project(x)
if res.shape == x.shape:
x = x + res
return nn.functional.silu(x)
class EfficientNetBaseline(nn.Module):
"""
A baseline architecture that can be scaled via depth and width coefficients.
"""
def __init__(self, depth_scale=1.0, width_scale=1.0, input_res=32):
super(EfficientNetBaseline, self).__init__()
def scale_w(w): return int(math.ceil(w * width_scale))
def scale_d(d): return int(math.ceil(d * depth_scale))
# Baseline config: (expand_ratio, channels, stride, repeats)
self.config = [
(1, scale_w(16), 1, 1),
(6, scale_w(24), 2, 2),
(6, scale_w(40), 2, 2),
(6, scale_w(80), 2, 3),
]
layers = []
in_channels = 3
# Initial Stem
layers.append(nn.Sequential(
nn.Conv2d(3, scale_w(32), 3, stride=2, padding=1, bias=False),
nn.BatchNorm2d(scale_w(32)),
nn.SiLU()
))
in_channels = scale_w(32)
for expand_ratio, out_channels, stride, repeats in self.config:
layers.append(MBConv(in_channels, out_channels, expand_ratio, stride))
in_channels = out_channels
for _ in range(scale_d(repeats) - 1):
layers.append(MBConv(in_channels, out_channels, expand_ratio, 1))
self.features = nn.Sequential(*layers)
self.classifier = nn.Sequential(
nn.AdaptiveAvgPool2d(1),
nn.Flatten(),
nn.Linear(in_channels, 10)
)
def forward(self, x):
return self.classifier(self.features(x))
class CompoundScaler:
"""Implements the Compound Scaling logic."""
def __init__(self, phi=0, alpha=1.2, beta=1.1, gamma=1.15):
self.phi = phi
self.alpha, self.beta, self.gamma = alpha, beta, gamma
def get_scales(self):
return self.alpha**self.phi, self.beta**self.phi, self.gamma**self.phi
# --- Execution Pipeline ---
if __name__ == '__main__':
PHI = 1.0 # Increase this to scale the model up (B1, B2, etc.)
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
scaler = CompoundScaler(phi=PHI)
d_scale, w_scale, r_scale = scaler.get_scales()
scaled_res = int(32 * r_scale)
print(f"Scaling (phi={PHI}): Depth={d_scale:.2f}, Width={w_scale:.2f}, Res={scaled_res}")
transform = transforms.Compose([
transforms.Resize((scaled_res, scaled_res)),
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])
train_set = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_set, batch_size=64, shuffle=True)
model = EfficientNetBaseline(depth_scale=d_scale, width_scale=w_scale).to(DEVICE)
print(f"Total Parameters: {sum(p.numel() for p in model.parameters()):,}")Key Takeaways for Practitioners
- Stop Scaling Randomly: If you increase your input image resolution, don't forget to increase your model's depth and width.
- Start Small: Use a baseline like EfficientNet-B0. Find your scaling constants ($\alpha, \beta, \gamma$) on a small proxy dataset before scaling up to the full model.
- Efficiency First: By using MBConv and Compound Scaling, you can achieve state-of-the-art accuracy with significantly fewer parameters and FLOPs than traditional CNNs.
Summary Table: The Compound Scaling Effect
| Dimension | Scaling Factor | Impact on Model |
|---|---|---|
| Depth | $\alpha^\phi$ | Increases receptive field and complexity. |
| Width | $\beta^\phi$ | Increases capacity to capture fine-grained features. |
| Resolution | $\gamma^\phi$ | Provides more pixels for the model to analyze. |
| Total FLOPs | $\approx 2^\phi$ | Predictable growth in computational cost. |