Beyond NeRF: Real-Time Radiance Fields with 3D Gaussian Splatting
Beyond NeRF: Real-Time Radiance Fields with 3D Gaussian Splatting
For years, Neural Radiance Fields (NeRFs) have captivated the computer vision community by enabling photorealistic 3D reconstructions from a handful of 2D images. However, NeRFs come with a heavy price: rendering speed. Because NeRFs represent scenes as continuous functions stored in a neural network, rendering a single frame requires querying that network millions of times via expensive ray-marching.
Enter 3D Gaussian Splatting (3DGS).
By shifting from an implicit neural representation to an explicit, point-based representation, 3DGS achieves what was previously thought impossible: photorealistic quality with real-time frame rates.
The Core Intuition: From "Querying" to "Painting"
If NeRF is like asking a neural network, "What color is the air at this exact coordinate?" for every single pixel, 3D Gaussian Splatting is like painting a 3D scene with millions of fuzzy, stretchable ellipsoids.
Instead of a dense volume, the scene is composed of a collection of 3D Gaussians. Each Gaussian is a "splat" that stores:
- Position ($\mu$): Where it is in 3D space.
- Covariance ($\Sigma$): Its shape, size, and orientation (how it's stretched).
- Opacity ($\alpha$): How transparent it is.
- Color: Represented via Spherical Harmonics to allow the color to change depending on the angle you view it from.
To render an image, these 3D ellipsoids are projected (splatted) onto a 2D plane and blended together. This leverages the power of GPU rasterizationāthe same technology that makes modern video games fastārather than the slow process of volumetric sampling.
The Technical Architecture
1. The Mathematical Foundation
To ensure the "splats" look smooth and are differentiable (so we can train them), the model uses a 3D Gaussian distribution. The shape of each Gaussian is defined by its covariance matrix $\Sigma$. To ensure $\Sigma$ remains positive semi-definite during optimization, it is decomposed into a rotation matrix $R$ and a scaling matrix $S$:
$$\Sigma = R S S^T R^T$$
When it comes to rendering, the final color of a pixel is computed using $\alpha$-blending, summing the contributions of Gaussians sorted by depth:
$$\text{Color} = \sum_{i \in N} c_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j)$$
2. The Pipeline Workflow
The transition from a set of photos to a real-time 3D scene follows this pipeline:
Implementation: A Simplified PyTorch Example
While the production version of 3DGS uses highly optimized CUDA kernels for tile-based rasterization, we can implement the core logic in PyTorch to understand the differentiable nature of the process.
import torch
import torch.nn as nn
import torch.nn.functional as F
class GaussianModel(nn.Module):
def __init__(self, num_points: int):
super().__init__()
# Learnable parameters for each Gaussian
self.means = nn.Parameter(torch.randn(num_points, 3))
self.scales = nn.Parameter(torch.randn(num_points, 3))
self.rotations = nn.Parameter(torch.randn(num_points, 4)) # Quaternions
self.opacities = nn.Parameter(torch.randn(num_points, 1))
self.colors = nn.Parameter(torch.randn(num_points, 3))
def get_covariance(self) -> torch.Tensor:
"""Computes the 3D covariance matrix Ī£ = R S S^T R^T"""
q = F.normalize(self.rotations, p=2, dim=-1)
# Simplified Quaternion to Rotation Matrix conversion
w, x, y, z = q[:, 0], q[:, 1], q[:, 2], q[:, 3]
R = torch.stack([
1 - 2*y**2 - 2*z**2, 2*x*y - 2*w*z, 2*x*z + 2*w*y,
2*x*y + 2*w*z, 1 - 2*x**2 - 2*z**2, 2*y*z - 2*w*x,
2*x*z - 2*w*y, 2*y*z + 2*w*x, 1 - 2*x**2 - 2*y**2
], dim=-1).reshape(-1, 3, 3)
S = torch.diag_embed(torch.exp(self.scales))
return R @ S @ S.transpose(-1, -2) @ R.transpose(-1, -2)
class GaussianRasterizer(nn.Module):
def __init__(self, width: int, height: int):
super().__init__()
self.width, self.height = width, height
def forward(self, model: GaussianModel, view_matrix: torch.Tensor) -> torch.Tensor:
# 1. Project 3D means to 2D screen space
means_homo = torch.cat([model.means, torch.ones(model.means.shape[0], 1).to(model.means.device)], dim=-1)
projected = means_homo @ view_matrix.T
z = projected[:, 2:3]
uv = projected[:, 0:2] / (z + 1e-6)
px = (uv[:, 0] + 1) * 0.5 * self.width
py = (uv[:, 1] + 1) * 0.5 * self.height
# 2. Sort by depth for alpha blending
indices = torch.argsort(z.squeeze(), descending=False)
image = torch.zeros((self.height, self.width, 3), device=model.means.device)
transmittance = torch.ones((self.height, self.width, 1), device=model.means.device)
opacity = torch.sigmoid(model.opacities)
colors = torch.sigmoid(model.colors)
# 3. Differentiable Splatting (Simplified loop)
for idx in indices[:500]:
p_x, p_y = int(px[idx]), int(py[idx])
if 0 <= p_x < self.width and 0 <= p_y < self.height:
alpha = opacity[idx] * transmittance[p_y, p_x]
image[p_y, p_x] += alpha * colors[idx]
transmittance[p_y, p_x] *= (1 - opacity[idx])
return imageAdaptive Density Control: The Secret Sauce
A static number of Gaussians isn't enough. Some areas of a scene (like a flat wall) need very few, while others (like a leafy tree) need millions. 3DGS solves this with Adaptive Density Control:
- Cloning: If a Gaussian has a large gradient (meaning it's struggling to cover an area), the system clones it to increase detail.
- Splitting: If a Gaussian is too large, it is split into smaller ones to refine the geometry.
- Pruning: Gaussians with opacity $\alpha$ below a certain threshold are deleted to keep the model lean and fast.
Summary: Why This Matters
3D Gaussian Splatting represents a paradigm shift in neural rendering. By combining the differentiability of NeRFs with the efficiency of rasterization, it unlocks:
- Real-time VR/AR: Explore photorealistic 3D scenes at 100+ FPS.
- Faster Training: No more waiting days for a scene to converge.
- Explicit Control: Unlike a "black box" neural network, you can manually move, delete, or edit individual Gaussians in the scene.
The era of "waiting for the ray-marcher" is over. The era of the splat has arrived.