NeRF: Representing Scenes as Neural Radiance Fields
TL;DR: NeRF encodes a 3D scene as a continuous function: feed in a 3D coordinate and a viewing direction, get back a color and a density. Render any viewpoint by marching rays through this function and compositing the results via classical volume rendering. The entire scene fits in 5MB of MLP weights — no voxel grids, no meshes, no explicit 3D structure. Two key tricks make it work: positional encoding (so the MLP can represent high-frequency detail) and hierarchical sampling (so compute is concentrated where geometry actually exists).
These paper reviews are written more for me and less for others. LLMs have been used in formatting
This post draws from NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (Mildenhall et al., 2020).
The Idea
Given a set of photographs of a scene taken from known camera positions, synthesise new photographs from viewpoints that were never captured. This is the novel view synthesis problem.
Prior approaches represented scenes explicitly — as voxel grids, meshes, or multiplane images. NeRF’s contribution is representing a scene as a continuous 5D function parameterised by an MLP:
\[F_\Theta : (\mathbf{x}, \mathbf{d}) \rightarrow (\mathbf{c}, \sigma)\]- $\mathbf{x} = (x, y, z)$: 3D location
- $\mathbf{d}$: viewing direction (a 3D unit vector)
- $\mathbf{c} = (r, g, b)$: emitted color
- $\sigma$: volume density — the differential probability of a ray terminating at that point
The network learns to map any point in 3D space to “what’s there” (density) and “what it looks like from this angle” (color).
The Multiview Consistency Trick
A critical architectural constraint: density $\sigma$ depends only on position $\mathbf{x}$, while color $\mathbf{c}$ depends on both position and direction.
Why: geometry shouldn’t change depending on where you look from. A chair leg is in the same place regardless of viewpoint. If density could depend on viewing direction, the network could cheat — hallucinating different geometry per view to fit each training image independently, rather than learning a coherent 3D scene.
Color is allowed to vary with direction because physically it does — specular highlights, reflections, and glossy surfaces all look different from different angles. This is what makes NeRF handle non-Lambertian (shiny, reflective) surfaces, unlike methods that assume matte appearance.
This is a good general lesson: encode invariances into architecture where you can, rather than hoping the loss enforces them.
In the MLP, this is enforced structurally: the direction $\mathbf{d}$ only enters at the very end, after $\sigma$ has already been produced. The network processes position through 8 layers before direction enters for a single additional layer to produce color.
Volume Rendering: From Function to Pixels
Rendering one pixel means casting a ray from the camera through that pixel and integrating along it. The continuous rendering integral:
\[C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\, \sigma(\mathbf{r}(t))\, \mathbf{c}(\mathbf{r}(t), \mathbf{d})\, dt, \qquad T(t) = \exp\!\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s))\, ds\right)\]$T(t)$ is the accumulated transmittance — the probability the ray travels from the near bound $t_n$ to point $t$ without hitting anything. The integrand reads naturally: “the color at $t$, weighted by how dense it is there, weighted by how likely the ray was to survive getting there.”
This is not something NeRF invented — it’s the classical volume rendering equation from Kajiya & Von Herzen (1984). NeRF’s contribution is putting a learnable MLP inside this well-established integral.
In practice, the integral is approximated by sampling points along the ray and summing:
\[\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i \big(1 - \exp(-\sigma_i \delta_i)\big) \mathbf{c}_i\]where $\delta_i$ is the distance between adjacent samples. Setting $\alpha_i = 1 - \exp(-\sigma_i \delta_i)$ reduces this to standard alpha compositing — the same formula that later appears in 3D Gaussian Splatting.
Why Positional Encoding Is Needed
An established result (Rahaman et al., 2018): deep networks have a spectral bias — they preferentially learn low-frequency functions and struggle with high-frequency detail. An MLP fed raw $(x, y, z)$ coordinates produces oversmoothed geometry and blurry textures.
The fix: map each coordinate to a higher-dimensional space using sinusoids at exponentially increasing frequencies before feeding it to the network:
\[\gamma(p) = \Big(\sin(2^0 \pi p),\, \cos(2^0 \pi p),\, \ldots,\, \sin(2^{L-1} \pi p),\, \cos(2^{L-1} \pi p)\Big)\]Applied separately to each coordinate: $L = 10$ for position (→ 60 dimensions) and $L = 4$ for direction (→ 24 dimensions). The lower $L$ for direction makes sense — view-dependent appearance is inherently smoother than spatial geometry.
A useful heuristic from the ablations: the benefit of increasing $L$ is capped once $2^L$ exceeds the maximum frequency present in the input images. For their data, $2^{10} = 1024$ matches the image resolution, and $L = 15$ gives no further gain.
Note on naming: this shares the functional form with transformer positional encodings but serves the opposite purpose. Transformers inject discrete token order into an order-agnostic architecture. NeRF maps continuous coordinates into a space where the MLP can represent high-frequency functions.
Hierarchical Sampling: Don’t Waste Compute on Empty Space
Densely evaluating the MLP at $N$ points along every ray is wasteful — free space and occluded regions contribute nothing to the final color but get sampled anyway.
Solution: train two networks simultaneously, “coarse” and “fine.”
- Sample $N_c = 64$ locations along the ray using stratified sampling (one random sample per evenly spaced bin — this keeps the representation continuous across training iterations, unlike a fixed grid)
- Evaluate the coarse network. Its per-sample weights form a probability distribution along the ray — peaked wherever the coarse network thinks there’s geometry
- Sample $N_f = 128$ additional locations from that distribution, concentrating samples where they matter
- Evaluate the fine network at the union of both sample sets ($N_c + N_f = 192$), compute the final color
The loss is simple — squared error between rendered and ground-truth pixel colors, applied to both coarse and fine renderings:
\[\mathcal{L} = \sum_{\mathbf{r} \in \mathcal{R}} \left[\big\lVert \hat{C}_c(\mathbf{r}) - C(\mathbf{r})\big\rVert_2^2 + \big\lVert \hat{C}_f(\mathbf{r}) - C(\mathbf{r})\big\rVert_2^2\right]\]The coarse loss is retained even though only the fine render is the final output — without it, the coarse weight distribution degrades and stops usefully guiding the fine samples.
Training and Cost
Each scene is trained independently — one network per scene, no generalisation across scenes. Training on 100-300K iterations takes 1-2 days on a single V100. A batch is 4096 rays (not images — individual pixel rays).
The result is 5MB of network weights — the entire scene stored in a compact MLP. For comparison, LLFF (the best prior method on forward-facing scenes) stores a voxel grid per input image, totalling >15GB for one scene. That’s a 3000x compression, and NeRF’s weights are smaller than the input images themselves.
The tradeoff: rendering is slow. Each pixel requires ~256 network queries (64 coarse + 192 fine), and a 640K-pixel image needs ~150-200 million queries — about 30 seconds per frame on a V100. The entire follow-up line (InstantNGP, Plenoxels, 3DGS) is the field answering this efficiency problem.
Results
NeRF was evaluated on three datasets: simple synthetic objects (Diffuse Synthetic 360°), complex synthetic objects with non-Lambertian materials (Realistic Synthetic 360° — the canonical “Blender dataset”), and real handheld cellphone captures (LLFF’s dataset).
NeRF won on PSNR and SSIM across the board. The one exception: LPIPS on real forward-facing scenes, where LLFF — a method specifically designed for forward-facing captures — edged it out. A fair loss on LLFF’s home turf.
The ablations rank component importance clearly: positional encoding and view dependence matter most (removing either drops PSNR by 2-3 dB). Hierarchical sampling matters least for quality but is critical for compute efficiency.
A strong sample-efficiency result: with only 25 input images, NeRF still beats all baselines given 100 images.
Limitations
The paper is explicit about what NeRF doesn’t do:
- Slow: both training (hours per scene) and rendering (30s per frame). The authors themselves flag this as the primary direction for future work.
- Per-scene: no generalisation — every new scene trains from scratch with no shared knowledge.
- Static scenes only: no moving objects, no temporal dynamics.
- Known camera poses required: needs COLMAP or equivalent as a preprocessing step.
- Interpretability: a voxel grid or mesh lets you reason about expected quality and failure modes. An MLP’s weights offer no such intuition.
Key Takeaways
- NeRF represents a scene as a continuous 5D function (position + direction → color + density), parameterised by an MLP and rendered via classical volume rendering.
- The core architectural insight: density depends only on position (enforcing multiview-consistent geometry), while color depends on both position and direction (allowing view-dependent appearance).
- Positional encoding overcomes the spectral bias of MLPs, enabling high-frequency detail. The bandwidth should match the input image resolution.
- Hierarchical sampling (coarse network guides fine sampling) concentrates compute where geometry exists.
- 5MB of weights encodes an entire scene — 3000x more compact than voxel-based alternatives.
- The efficiency limitations (hours to train, 30s to render) defined the research agenda for the next several years.