Scaling Vision Transformers: A Deep Dive into Swin Transformer
Scaling Vision Transformers: A Deep Dive into Swin Transformer
In the evolution of Computer Vision, the Vision Transformer (ViT) marked a paradigm shift by proving that the Transformer architecture—originally designed for NLP—could outperform Convolutional Neural Networks (CNNs) on large-scale datasets. However, ViT came with two significant "bottlenecks": a lack of multi-scale representations and a computational cost that exploded quadratically with image resolution.
Enter the Swin Transformer. By introducing a hierarchical structure and a clever "shifted window" attention mechanism, Swin bridges the gap between the global modeling power of Transformers and the efficiency of CNNs.
The Core Intuition: Why "Swin"?
The "Swin" in Swin Transformer stands for Shifted Window. To understand why this is necessary, we have to look at the math of attention.
The Complexity Problem
In a standard ViT, every pixel (patch) attends to every other pixel. If an image has $N$ patches, the complexity is: $$\text{Complexity of Global Self-Attention: } \mathcal{O}(N^2) \rightarrow \mathcal{O}(H^2 W^2)$$
For a high-resolution image, this becomes computationally impossible. Swin solves this by computing attention within local, non-overlapping windows. This reduces the complexity to: $$\text{Complexity of Swin Window-based Self-Attention: } \mathcal{O}(N) \rightarrow \mathcal{O}(HW)$$
But there is a catch: if windows never communicate, the model cannot learn global context. Swin solves this by shifting the window partitions in consecutive layers, allowing information to "leak" across window boundaries.
Architectural Blueprint
The Swin Transformer is designed as a hierarchy of stages, mimicking the pooling layers of a ResNet.
1. Hierarchical Feature Maps
Unlike ViT, which maintains a constant resolution, Swin uses Patch Merging. This process concatenates $2 \times 2$ neighboring patches and applies a linear layer to reduce spatial resolution by half while doubling the channel depth. This allows the model to detect small objects in early stages and large, complex objects in later stages.
2. The Swin Block Logic
Each stage consists of multiple Swin Transformer blocks. These blocks alternate between two types of attention:
- W-MSA (Window Multi-head Self-Attention): Attention is computed within fixed windows.
- SW-MSA (Shifted Window Multi-head Self-Attention): The window grid is shifted, creating cross-window connections.
High-Level Architecture Flow
Implementation in PyTorch
Below is a production-ready implementation of the Swin Transformer core components.
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple
class WindowAttention(nn.Module):
"""Window-based Multi-head Self-Attention (W-MSA)."""
def __init__(self, dim, window_size, num_heads):
super().__init__()
self.dim = dim
self.window_size = window_size
self.num_heads = num_heads
head_dim = dim // num_heads
self.scale = head_dim ** -0.5
self.qkv = nn.Linear(dim, dim * 3)
self.proj = nn.Linear(dim, dim)
def forward(self, x):
B_win, N, C = x.shape
# Generate Q, K, V tensors
qkv = self.qkv(x).reshape(B_win, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
attn = (q @ k.transpose(-2, -1)) * self.scale
attn = attn.softmax(dim=-1)
x = (attn @ v).transpose(1, 2).reshape(B_win, N, C)
return self.proj(x)
class SwinBlock(nn.Module):
"""A single Swin Transformer Block with Shifted Window support."""
def __init__(self, dim, input_resolution, num_heads, window_size=7, shift_size=0):
super().__init__()
self.dim = dim
self.shift_size = shift_size
self.window_size = window_size
self.norm1 = nn.LayerNorm(dim)
self.attn = WindowAttention(dim, window_size, num_heads)
self.norm2 = nn.LayerNorm(dim)
self.mlp = nn.Sequential(
nn.Linear(dim, 4 * dim),
nn.GELU(),
nn.Linear(4 * dim, dim)
)
def forward(self, x):
B, H, W, C = x.shape
X = x
# 1. Shift Window (SW-MSA)
if self.shift_size > 0:
X = torch.roll(X, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
# 2. Partition into windows: (B, H, W, C) -> (B*num_win, ws*ws, C)
X = X.view(B, H // self.window_size, self.window_size, W // self.window_size, self.window_size, C)
X = X.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, self.window_size**2, C)
# 3. Window Attention + MLP
shortcut = X
X = self.norm1(X)
X = self.attn(X)
X = shortcut + X
X = X + self.mlp(self.norm2(X))
# 4. Reverse Partition & Shift
X = X.view(B, H // self.window_size, W // self.window_size, self.window_size, self.window_size, C)
X = X.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, C)
if self.shift_size > 0:
X = torch.roll(X, shifts=(self.shift_size, self.shift_size), dims=(1, 2))
return X
class PatchMerging(nn.Module):
"""Downsamples spatial resolution by 2x and increases channel depth by 2x."""
def __init__(self, input_dim, output_dim):
super().__init__()
self.reduction = nn.Linear(4 * input_dim, output_dim, bias=False)
def forward(self, x):
B, H, W, C = x.shape
# Space-to-Depth: Partition into 2x2 patches
x0 = x[:, 0::2, 0::2, :]
x1 = x[:, 1::2, 0::2, :]
x2 = x[:, 0::2, 1::2, :]
x3 = x[:, 1::2, 1::2, :]
x = torch.cat([x0, x1, x2, x3], dim=-1)
return self.reduction(x)
class SwinTransformer(nn.Module):
"""Swin Transformer Backbone."""
def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, depths=[2, 2, 6, 2], num_heads=[3, 6, 12, 24], window_size=7, num_classes=10):
super().__init__()
self.patch_embed = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
self.layers = nn.ModuleList()
for i in range(len(depths)):
# Stage Blocks
stage = nn.Sequential(*[
SwinBlock(
dim=embed_dim * (2**i),
input_resolution=(img_size // (patch_size * (2**i))),
num_heads=num_heads[i],
window_size=window_size,
shift_size=(0 if j % 2 == 0 else window_size // 2)
) for j in range(depths[i])
])
self.layers.append(stage)
# Patch Merging (except last stage)
if i < len(depths) - 1:
self.layers.append(PatchMerging(embed_dim * (2**i), embed_dim * (2**(i+1))))
self.head = nn.Linear(embed_dim * (2**(len(depths)-1)), num_classes)
def forward(self, x):
x = self.patch_embed(x)
x = x.permute(0, 2, 3, 1) # (B, H, W, C)
for layer in self.layers:
if isinstance(layer, nn.Sequential):
for block in layer: x = block(x)
else:
x = layer(x)
x = x.mean(dim=(1, 2)) # Global Average Pooling
return self.head(x)Key Takeaways for Practitioners
When to use Swin Transformer?
- Dense Prediction Tasks: Because of its hierarchical nature, Swin is significantly better than ViT for Object Detection and Semantic Segmentation (it works naturally with FPNs).
- High-Resolution Images: If your input images are larger than $224 \times 224$, the linear complexity of Swin makes it the only viable Transformer choice.
Summary Table: ViT vs. Swin
| Feature | Vision Transformer (ViT) | Swin Transformer |
|---|---|---|
| Attention | Global (All-to-All) | Local Window $\rightarrow$ Shifted Window |
| Complexity | Quadratic $\mathcal{O}(N^2)$ | Linear $\mathcal{O}(N)$ |
| Resolution | Single Scale | Hierarchical (Multi-scale) |
| Best Use Case | Image Classification | Detection, Segmentation, Classification |
By combining the local inductive bias of CNNs with the long-range dependency modeling of Transformers, the Swin Transformer provides a scalable, efficient, and powerful backbone for the next generation of computer vision models.