Neural and Learned Rendering

This last chapter is about a change of approach. Instead of modeling a scene and simulating light through it, learned methods choose a flexible representation (blobs, a grid of values, a neural network), render it, and optimize its numbers until the rendering matches real photographs or reference images. The same toolbox, gradients and neural networks, has also crept into ordinary renderers, as denoisers and upscalers. We'll cover the core idea, gradient descent; neural networks and neural fields; the two big ways to capture real scenes, radiance fields and Gaussian splatting; and the learned helpers inside today's real-time renderers.

Fitting starts with a loss: one number that says how wrong the current result is. For images, the usual loss is the average squared difference between rendered and target pixels. The loss depends on every parameter (every blob's position, size and color), so it's a landscape over a space with thousands of dimensions, and we want its lowest point.

The gradient is the direction in which the loss rises fastest: for each parameter, how much the loss changes when that parameter changes a little (its derivative, from Chapter 27). Gradient descent takes a small step the other way, and repeats. The step size, the learning rate, is the main knob: too small and progress is slow, too big and each step overshoots the valley and bounces from side to side.

That last case matters: a blob only "feels" the pixels it overlaps. Real systems fix it by adding blobs where the error stays high and removing ones that fade away. And plain gradient descent is rarely used as is: Adam (Diederik Kingma and Jimmy Ba, 2014), which the hero uses, keeps a running average of each parameter's gradient and of its size, and scales each parameter's step so that all of them move at a sensible pace, whatever their units.

Computing the gradient is the other half. For the hero's blobs, the derivatives were written by hand: how a pixel changes when a blob moves, grows or turns. Bigger systems use automatic differentiation: record every operation of the forward computation, then apply the chain rule backward through them (backpropagation), getting every parameter's gradient for about the cost of one more forward pass. A renderer built so that this works is a differentiable renderer.

A neural network, in its simplest form (a multilayer perceptron), is a chain of layers. Each layer multiplies its input vector by a matrix of weights, adds a vector of biases, and passes every result through a simple bend, usually max(0, x) (a ReLU). With enough layers and weights, the chain can approximate almost any function, and gradient descent finds weights that make it approximate the one you want, from examples.

A neural field uses a network as a function of position: give it coordinates (x, y) and it returns a color; or (x, y, z) and it returns a density, or a signed distance (Chapter 15). The network is the image, the shape or the scene, stored in its weights, and it can be sampled anywhere, at any resolution.

There's a catch: fed raw coordinates, networks learn smooth, slowly varying functions easily and sharp detail very slowly (spectral bias). The fix, from Matthew Tancik and colleagues (2020), is to feed the network Fourier features instead: sines and cosines of the coordinates at several frequencies, so fast-varying inputs are available from the start.

Big networks are slow to evaluate: millions of multiply-adds per sample, per pixel. Instant neural graphics primitives (Thomas Müller and colleagues, 2022) moved most of the memory into a multiresolution hash grid: learned feature vectors stored at the corners of grids of several sizes, looked up by hashing the cell's coordinates, interpolated, and passed to a tiny network. Training dropped from hours to seconds. The lesson recurs across the field: grids and lookups are fast, networks are flexible, and the best systems combine them.

Take fifty photographs of an object from all around. Work out where each camera was (structure from motion, a computer vision technique, finds the camera positions by matching features between photos). Now fit a neural field whose input is a 3D position and a viewing direction, and whose output is a density and a color: a radiance field. Render it the way Chapter 35 rendered fog, marching each pixel's ray and adding up color weighted by how much light survives to the camera. Compare with the photo's pixel, and backpropagate.

That's NeRF (neural radiance fields, Ben Mildenhall and colleagues, 2020). Because the volume rendering is differentiable, the error at a pixel tells the network to raise the density where a surface should be and lower it in empty air. The viewing direction lets colors change with angle, so shiny highlights and reflections come out right. New views, from cameras that never existed, are just more rays.

In code, one ray's pixel is a short loop over its samples, near to far, spaced dt apart. Each sample blocks a share alpha of the light, from its density; it adds its color times that share, times T, the light that got past everything in front of it:

var T = 1.0;                               // light still getting through
var C = vec3f(0.0);
for (var i = 0u; i < SAMPLES; i++) {
  let alpha = 1.0 - exp(-density[i] * dt); // how much this step blocks
  C += T * alpha * color[i];
  T *= 1.0 - alpha;
}
C += T * background;                       // what gets through it all

Training runs this loop, compares C with the photo's pixel, and then walks the samples a second time to work out each one's share of the error. A sample's color matters in proportion to its weight T · alpha. Its density matters in two ways: more density adds more of its own color, but also dims everything behind it, so the second walk keeps a running total of the color still to come. Those slopes then flow back through the network into the hash grid.

The trainer above is a small version of the hash grid approach from Part 2, run entirely in compute shaders. Each point along a ray looks up 12 levels of a hash grid (2 features each, blended from the 8 corners of its cell), and a density network turns those 24 numbers into a density and 16 features. A second network takes the features plus the viewing direction, as 9 spherical harmonics, and gives a color: that's why the teal sphere's highlight can come out right, since the field's color is allowed to depend on where you look from. Each training step:

1. pick 512 random pixels from the photos, and the ray through each;
2. evaluate the field at 64 points along every ray, keeping each layer's inputs;
3. volume render each ray, compare with its pixel, and work out how the pixel would change with each sample's density and color;
4. backpropagate through both networks, and scatter the slope into the hash grid's corners with integer atomics;
5. add up each weight's gradient over all 32,768 samples, which is a matrix product;
6. and take an Adam step for the network and for the table.

The fog it starts as fills the box the field covers; outside it there's nothing to learn, and rays get the background. Within a hundred steps the fog has cleared to blurry shapes in the right places, and the detail, the checkerboard's edges and the box's corners, sharpens over the next few thousand.

The slice shows what the field really is. There are no objects in it, only a density at every point. The spheres' cross-sections are densest at their rims, where the photos saw surfaces; their insides hold whatever training happened to leave, since no photo ever saw in there. Let the cut sweep down to the ground and the patches under the objects come out empty: no camera could see the ground beneath them, so nothing ever made it solid. Compare the splats' swirl in Part 4: a splat scene is a list of blobs you can pick up and move; a radiance field is a function you can only ask about one point at a time.

Radiance fields capture real places with startling fidelity, but the original NeRF, one big network with no grid, took a day or two to train and about half a minute to render each frame. Hash grids cut that to minutes and a frame rate, and baking a trained field into textures and meshes makes it render like ordinary geometry. The approach that made captured scenes fast enough for real time came next, and dropped the network altogether.

3D Gaussian splatting (Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, 2023) represents a scene as millions of 3D Gaussians: fuzzy ellipsoids, each with a position, a shape (three scales and a rotation), an opacity, and a color that varies with viewing direction, stored as spherical harmonics (Chapter 24). To render: project each Gaussian to a 2D ellipse on screen (a splat), sort the splats by depth, and alpha-blend them front to back, like the transparent objects of Chapter 21.

The projection is the one piece of real math. A Gaussian's shape is its covariance, a 3 × 3 matrix built from its rotation R and scales: stretch a ball by the scales, then turn it. Seen through a perspective camera, a Gaussian isn't exactly a Gaussian on screen anymore, but close enough if the projection is treated as a straight-line map near the splat's center. That map's slope, two rows t0 and t1 (how a small 3D move shifts the screen point across and down), turns the 3D covariance into a 2D one:

// the 3D covariance: Sigma = R S S^T R^T
let M = R * mat3x3f(s.x, 0.0, 0.0, 0.0, s.y, 0.0, 0.0, 0.0, s.z);
let sigma = M * transpose(M);

// pc: the center in camera space; cam.r0, r1, r2: the camera's axes;
// f: the focal length in pixels
let iz = 1.0 / pc.z;
let t0 = (f * iz) * cam.r0 - (f * pc.x * iz * iz) * cam.r2;
let t1 = (f * iz) * cam.r1 - (f * pc.y * iz * iz) * cam.r2;

// the 2D covariance, widened so no splat is thinner than a pixel
let a = dot(t0, sigma * t0) + 0.3;
let b = dot(t1, sigma * t0);
let c = dot(t1, sigma * t1) + 0.3;
// its inverse, the "conic", is what every pixel needs
let conic = vec3f(c, -b, a) / (a * c - b * b);

Blending is then the same loop as the radiance field's, with alpha coming from how far the pixel is from each splat's center, measured in the splat's own stretched units:

// for each splat in this pixel's tile, sorted near to far
let d = pixel - center;
let e = 0.5 * (conic.x * d.x * d.x + conic.z * d.y * d.y) + conic.y * d.x * d.y;
let alpha = min(0.99, opacity * exp(-e));
C += T * alpha * color;
T *= 1.0 - alpha;

Training is the hero demo in 3D. Start from the sparse points that structure from motion found; render; compare with the photos; backpropagate into every Gaussian's parameters with Adam. Every few hundred steps, split or clone Gaussians where the error stays high and remove nearly transparent ones. A few minutes of training gives a scene that renders at hundreds of frames per second, because rendering is just rasterizing blobs: no network, no ray marching.

The trainer does the full backward pass described above: the error at each pixel goes back through the blend to each splat's 2D center, ellipse and opacity, and from there, link by link through the projection, to its 3D position, its three scales and its rotation. Two more pieces make the result sharp. Every 250 steps, splats that have faded to nothing are moved onto visible ones (picked in proportion to their opacity), and each pair splits the old opacity between them, so the picture doesn't change but there are now two splats to refine where detail is needed. And after the first 800 steps, each splat's color starts to depend on the viewing direction, through three spherical harmonic weights per color channel, which is what lets the teal sphere's highlight move as the view turns. Real trainers go further: they split and clone splats by their gradients, grow to millions of them, and use three bands of harmonics instead of one.

Compare the two figures on the same scene. Both learn it in seconds, but the splats are re-rendered every frame, while the radiance field is re-rendered only every few frames while it trains: evaluating two networks at up to 128 points per pixel costs far more than blending a few splats. That difference, at the size of a real capture, is why splatting took over for real-time viewing.

On the GPU, a splat renderer is a compute pipeline you could now write: project the Gaussians and compute each one's ellipse; sort by depth (a radix or bitonic sort); then either draw each splat as an instanced quad with blending, or bin the splats into screen tiles and blend each tile's list in a compute shader. The hard parts are memory (a captured room can be a gigabyte of floats, so compression matters), sorting fast enough, and the view-dependent color.

For the whole story in depth, see the companion book How Splats Learn: recovering where each photo was taken, the Gaussians and their covariances, tile-based rendering, the loss, backpropagation, Adam, and growing and pruning the blobs, every step from photographs to a trained scene.

To train splats on your own footage, try the video splat trainer. It takes a video or a set of photos, works out where each frame was taken, and trains a scene in your browser with WebGPU.

A path tracer at one sample per pixel makes a picture that's mostly noise (Chapter 15), and real-time path tracing depends on cleaning it up. Chapter 36's à-trous filter does it with hand-written rules: average each pixel with its neighbors, but trust a neighbor less if it faces a different way, sits at a different depth, or looks much brighter. Each rule has a constant someone picked by eye, such as "a normal power of 12", and the right values change from scene to scene and with the amount of noise. A neural denoiser keeps the same idea and the same inputs, and lets training choose the rules.

What it sees

The noisy picture alone isn't enough: a speck of noise and a real small detail look the same. So the renderer also hands over guide images it gets almost for free, because they come from where each pixel's ray first hits a surface, before any random bounces: the surface's color (its albedo), its normal, and its depth. They're clean, and they show where the edges are. As in Chapter 36, the color is divided by the albedo before filtering and multiplied back afterward, so the network only has to smooth the lighting, and textures stay sharp. A real-time denoiser also gets motion vectors and its own output from the previous frame (Chapter 31), so it can build on what it already worked out.

Predicting weights, not pixels

The network could simply output the clean color for each pixel. Many denoisers do, but there's a sturdier design: have it output, for every pixel, the weights of a small filter, and apply that filter to the noisy picture. This is a kernel-predicting denoiser (Steve Bako, Thijs Vogels and colleagues, 2017). Every output pixel is then a weighted average of real colors from nearby, so the network can't invent colors that aren't there, and it trains faster and more reliably. Passing the scores through a softmax (raise e to each score, then divide by the total) keeps the weights positive and adding up to 1, so the filter never makes the picture brighter or darker overall.

Training

Training needs pairs: the same frame at one sample per pixel and at thousands, rendered from many scenes, so that what's learned carries over to scenes it has never seen. The loss is usually a relative error, each pixel's error divided by its brightness, so that a dark corner counts as much as a bright window. Then it's gradient descent (Part 1), as everywhere else in this chapter.

The plain blur it starts as smears every edge: the balls bleed into the walls behind them. Within a few hundred steps, the network learns to stop at edges, giving only a few percent of a pixel's weight to neighbors on a different surface. Then it improves on the hand-tuned filter in a less obvious place. Click a pixel in the middle of the back wall and compare the two grids: the hand-tuned penalties, set cautiously so that edges stay sharp, trust about 35 of the 81 neighbors, while the trained network has learned it can trust more than 50 of them there, and average away more noise. That's where its lower error comes from, on a view it never trained on. It didn't find a new rule; it found better constants, and a better shape for each rule, than the ones picked by eye.

Here is the denoiser, for one pixel. Each neighbor is described by five numbers, all 0 when it matches the pixel in the middle, and the network turns them into a score:

// The denoised color of pixel p. noisy: each pixel's noisy light (its
// color divided by its albedo).
function denoisePixel(p) {
  const near = neighborsOf(p);                     // the 9 x 9 around p
  const scores = near.map((q) => net(differences(p, q)));
  const weights = softmax(scores);                 // positive, adding up to 1
  let light = [0, 0, 0];
  near.forEach((q, k) => {
    light = add(light, scale(noisy[q.index], weights[k]));
  });
  // put the texture back
  return light.map((v, c) => v * p.albedo[c]);
}

// The network: 5 numbers in, 12 hidden (ReLU), 1 score out. W1, b1, W2
// and b2 are its 85 weights, the numbers that training changes.
function net(f) {
  let score = b2;
  for (let h = 0; h < 12; h++) {
    let a = b1[h];
    for (let c = 0; c < 5; c++) {
      a += W1[h][c] * f[c];
    }
    score += W2[h] * Math.max(a, 0);
  }
  return score;
}

function softmax(s) {
  const top = Math.max(...s);
  const e = s.map((v) => Math.exp(v - top));
  const total = e.reduce((sum, v) => sum + v, 0);
  return e.map((v) => v / total);
}

And the five differences. Each pixel is a record of its guides: position, normal, depth, albedo, and lum, the logarithm of its brightness after a light 3 × 3 blur, because the raw noisy brightness of a single pixel says almost nothing:

// How neighbor q differs from pixel p: five numbers, each 0 if they match.
function differences(p, q) {
  const albedoGap = sub(p.albedo, q.albedo).reduce((s, v) => s + Math.abs(v), 0);
  return [
    ((p.x - q.x) ** 2 + (p.y - q.y) ** 2) / 16,   // how far apart
    1 - dot(p.normal, q.normal),                  // facing different ways
    Math.min(4, 20 * Math.abs(p.depth - q.depth) / p.depth),
    albedoGap,                                    // different surface colors
    Math.min(4, Math.abs(p.lum - q.lum)),         // different brightness
  ];
}

// The 9 x 9 pixels around p, repeating the edge pixels at the borders.
function neighborsOf(p) {
  const near = [];
  for (let j = -4; j <= 4; j++) {
    for (let i = -4; i <= 4; i++) {
      const x = Math.min(W - 1, Math.max(0, p.x + i));
      const y = Math.min(H - 1, Math.max(0, p.y + j));
      near.push(pixels[y * W + x]);
    }
  }
  return near;
}

The hand-tuned filter is the same code with net replaced by fixed penalties: -(1.5 f[0] + 10 f[1] + 3 f[2] + 5 f[3] + 3 f[4]). Training needs one more piece: how the loss changes with each neighbor's score. Through the weighted average and the softmax, it comes out short:

// dLoss: how the loss changes with each color channel of the output.
// Raising neighbor k's score pulls the output toward that neighbor's light.
const dScore = weights[k] * dot(dLoss, sub(noisy[q.index], light));

From there, the chain rule carries dScore back through net to its 85 weights, exactly as in the neural field at the end of the chapter, and Adam takes a step.

Real denoisers

Production denoisers are bigger in every direction, but built the same way. Instead of a fixed set of five differences, the network looks at the guide images itself, usually through a U-Net (Olaf Ronneberger and colleagues, 2015): it shrinks the image step by step to see wide areas at once, then grows it back, passing detail across at each size, so that each pixel's filter can depend on everything around it. Some predict a filter for each of several sizes and blend the results, like several à-trous passes chosen by the network.

For real time, the network is recurrent (Chakravarty R. Alla Chaitanya and colleagues, 2017): it takes last frame's output, moved along the motion vectors, as one more input, so it accumulates over time like TAA and doesn't flicker. They run on a GPU's matrix units, fast enough for every frame, often merged with upscaling (Chapter 31), so the one network denoises and adds resolution at once. Offline renderers use the same idea to stop at a few dozen samples per pixel instead of thousands. Because a trained network only sees the guides and the noise, one network serves every scene, with no constants to tune.

Denoising isn't the only place where learning has entered ordinary renderers. It helps wherever hand-written rules struggle:

• Upscalers. Render at a lower resolution, with TAA's jitter (Chapter 31), and let a network reconstruct the full-resolution frame from the current and previous samples, deciding pixel by pixel when to trust the history: more pixels for a fraction of the cost.
• Frame generation. From two rendered frames and their motion vectors, predict a frame in between, doubling the displayed frame rate (though not responsiveness).
• Neural materials and textures. Replace a complicated layered material with a small network evaluated per pixel, or compress a material's many textures into a few learned feature maps decoded by a tiny network.

These run as compute work, mostly matrix multiplications, which modern GPUs accelerate with special matrix units. On the web, WebGPU compute shaders run them today (faster with the shader-f16 and subgroups features), and the emerging WebNN API exposes the operating system's machine-learning accelerators.

Everything in this chapter so far, losses, gradients, Adam, networks, Fourier features, fits in about a hundred and forty lines of plain JavaScript. The playground at the end of this section trains a tiny network to be an image: given a pixel's (x, y), it returns the brightness there. There's no library and no GPU, just loops over arrays of numbers. Here are its pieces, in the order the code runs them.

The training data

The target is a 48 × 48 picture, a letter in a ring, drawn with the ordinary canvas API. The code turns every pixel into one example: an input, the pixel's position scaled to run from 0 to 1 across the image, and the answer the network should give, the pixel's brightness from 0 to 1. That's 2,304 examples. The network never sees the picture as a picture, only as a pile of "at this position, this brightness" facts, and it has to find one function that agrees with all of them.

samples.push({ p: [x / (S - 1), y / (S - 1)], value: targetPixels[(y * S + x) * 4] / 255 });
    

Fourier features

Before a position goes into the network, features expands it into sines and cosines of x and y at several frequencies, each double the last. With FREQS = 5, the slowest wave rises and falls once across the whole image, and the fastest repeats 8 times, a full cycle every 6 pixels. Those fast waves are what let the network make a sharp edge: instead of building a cliff out of slow, gentle inputs, it can mix waves that already change quickly. Five frequencies give 4 × 5 = 20 inputs.

for (let k = 0; k < FREQS; k++) {
  const f = Math.pow(2, k) * Math.PI;
  out.push(Math.sin(f * p[0]), Math.cos(f * p[0]), Math.sin(f * p[1]), Math.cos(f * p[1]));
}
    

The network

Three layers: 20 inputs to 32 neurons, 32 to 32, and 32 to the single output, brightness. Each layer is a flat array of weights, one per input per output, plus one bias per output. The hidden layers pass their sums through ReLU; the last doesn't, so the output can be any number. In all that's 1,761 numbers, fewer than the 2,304 pixels it learns to reproduce, so the network can't just memorize pixels: it has to find a compact description of the picture.

The weights start random, but not just any random: each is drawn between ±√(6 ÷ inputs). With more inputs, each weight starts smaller, so a layer's outputs come out about the same size as its inputs. Start them too big and the numbers grow layer after layer; too small and they fade to nothing, and so do the gradients.

Learning from one pixel's error

forward runs a position through the layers and keeps every layer's output along the way, because the backward pass needs them. The loss for one pixel is the squared error, err * err. Its gradient with respect to the output is 2 * err; the code passes err and lets the learning rate absorb the 2. backward then walks the layers from last to first, applying the chain rule.

Batches and Adam

Computing the exact gradient would mean running all 2,304 pixels before every step. Instead, each step uses a batch of 128 pixels picked at random: backward adds their gradients up (+=), and adam divides by 128 to average them. A batch's average points roughly downhill, and taking many cheap, roughly right steps beats taking few exact ones. This is stochastic gradient descent.

adam keeps two running averages for every weight: m, of its gradient, which smooths out the batch-to-batch jitter, and v, of its squared gradient, which measures how big that weight's gradients usually are. The step is LEARNING_RATE · m / √v, so every weight moves about the same distance per step, whether its gradients are huge or tiny. Both averages start at zero, which would make the first steps too small; dividing by 1 − 0.9step and 1 − 0.999step corrects that.

Each animation frame runs 6 training steps, then evaluates the network at all 2,304 pixels, forward passes only, to draw the right-hand image. Every 300 steps it logs the average squared error to the console.

Run it and watch the right side: within a second or two the ring and the letter appear as soft smudges, and over the next ten seconds the edges sharpen. The console's error falls from around 0.02 after 300 steps to about 0.003 after 3,600.

• Learned rendering optimizes a representation until its renderings match target images: a loss measures the error, the gradient says how to reduce it, gradient descent (or Adam) takes small steps.
• Differentiable renderers provide gradients, by hand or by automatic differentiation (backpropagation).
• Neural networks are chains of weighted sums and bends; neural fields map coordinates to values. Fourier features and hash grids make detail fast to learn.
• NeRF fits a radiance field to photographs through differentiable volume rendering.
• Gaussian splatting fits millions of 3D Gaussians, rendered by projecting, sorting and blending splats: fast enough for real time.
• A neural denoiser learns how much to trust each neighbor, from the noisy picture and clean guide images (albedo, normal, depth). Kernel prediction keeps each output pixel a weighted average of real nearby colors.
• Networks also upscale, generate frames, and encode materials inside conventional renderers.

That's the last technique in the book. Chapter 39 closes it: a look back, the few ideas that keep returning, and where to go next.

1. Memorized, or learned? Let the denoiser finish training, then press new noise a few times. Each press renders fresh one-sample images. How do the hand-tuned and learned errors compare each time? What would it mean if the learned error jumped up?

   Hint

   Both errors move a little with each new noise, but the learned filter stays about a fifth lower every time. If its error jumped, it would have memorized the particular noise it trained on (overfitting) instead of learning rules. With only 85 weights, and inputs that describe how two pixels differ rather than where they are, it has little room to memorize.

2. Raw coordinates. In the playground, set FREQS = 0. How does the result differ after the same number of steps? Then try FREQS = 8.

   Hint

   With raw coordinates, the letter stays a soft blur for a long time. With too many frequencies, the network can fit noise in between the training pixels: the result gets speckled artifacts, because frequencies above the image's resolution are free to do anything between samples.

3. Learning rate. In the gradient descent figure, start the blob in the valley's narrow direction and raise the learning rate until it bounces. Why does it bounce in one direction but not the other?

   Hint

   The loss curves sharply as the width changes but gently as the position changes, so the same step overshoots in one direction and is cautious in the other. Adam's per-parameter step scaling is a fix for exactly this.

4. Blobs as a format. The hero stores the picture as 2,000 blobs of 8 numbers each. How many bytes is that, in 32-bit floats, compared with the 160 × 100 picture as 8-bit RGB? What can the blobs do that the pixels can't?

   Hint

   2,000 × 8 × 4 = 64,000 bytes; the picture is 160 × 100 × 3 = 48,000 bytes, so it's no compression at this size. But the blobs can be drawn at any resolution, smoothly, and (in 3D) from any viewpoint, which is the point of splatting.