Real-Time Global Illumination

Chapter 12 lit each surface from the light sources alone, and added a small constant "ambient" term so shadows weren't pitch black. Chapter 15 showed what that constant stands in for: light that reaches a surface after bouncing off other surfaces, called indirect light, or, together with the direct light, global illumination (GI). It's why rooms are bright in their corners, why a red carpet tints a white ceiling pink, and why the shady side of a building isn't black. Getting it right is most of what makes a picture look real, and real-time renderers approach it in layers, each handling part of the problem.

Start with the simplest observation. Indirect light comes from all around, more or less evenly, like light from an overcast sky. A point on an open floor receives it from the whole hemisphere above it. A point deep in a crack, or in the corner where a wall meets the floor, sees much less of its surroundings, because nearby surfaces block most directions. So it receives less indirect light.

Ambient occlusion (AO) measures exactly that: for each point, the share of directions above the surface that are not blocked within some short distance. Multiply the ambient light by it, and corners, cracks and contact points darken naturally. It isn't physically exact (blocked directions aren't really dark, they're lit by whatever blocks them), but it captures the most noticeable effect of indirect light, and it's cheap enough for everyone to use.

Screen-space ambient occlusion

Tracing rays against the scene is the honest way to measure AO, and baking it into textures ahead of time was common for years. In 2007, Martin Mittring described a way to compute it every frame from nothing but the depth buffer: screen-space ambient occlusion (SSAO). For each pixel, scatter a handful of test points in a small ball around its surface point. Project each test point onto the screen and compare its depth with the depth buffer there. If the depth buffer holds something nearer the camera, the test point is probably inside some object, so that direction counts as blocked.

The depth buffer only knows the front surface of everything. What lies behind is unknown, and SSAO assumes it's solid. That causes its characteristic artifacts: dark halos around thin objects, which are assumed to be thick, and occlusion that changes as the camera moves, because what the depth buffer sees changes. Later methods (horizon-based AO by Louis Bavoil and colleagues in 2008, and ground-truth AO by Jorge Jimenez and colleagues in 2016) search along a few directions in the depth buffer for the highest "horizon" that blocks the sky, which is more accurate for the same cost.

Here's SSAO running on the GPU. The first pass writes each pixel's position and normal (in the camera's frame) into a G-buffer (Chapter 21); the second runs SSAO into a texture of its own; the third shows it. Start with SHOW = 0, the AO alone, and try different SAMPLES and RADIUS. Then set BLUR = false to see the raw result: a fine, regular speckle, from turning the test points a different way at every pixel. The 16 turns repeat every 4 × 4 pixels, each appearing once per block, so the blur, which averages each pixel's 4 × 4 block, sees every turn exactly once and the speckle cancels out. Real renderers blur in the same spirit, or let temporal antialiasing (Chapter 19) average the noise over frames.

Kernel SSAO is the first and simplest of a family. Chapter 33 picks the story up again with the methods that replaced it, horizon-based AO (HBAO, HBAO+ and GTAO), which search the depth buffer for the horizon around each pixel instead of testing points in a ball, and compares all of them with ray-traced AO in one scene.

If the lights and the walls never move, the bounced light never changes either, so why compute it every frame? A lightmap is a texture that stores the light arriving at every point of the static scenery, computed ahead of time ("baked") with a slow, accurate method, typically a path tracer like Chapter 15's running for minutes or hours. At run time, the renderer just multiplies each surface's color by its lightmap texel: perfect soft bounced light for the price of a texture lookup.

The trick that makes it affordable is that light changes slowly across a surface. A floor's color texture needs hundreds of texels per meter to show its tiles and scratches, but the light falling on it, even the edge of a soft shadow, changes over tens of centimeters. So the lightmap can be coarse, a few texels per meter, and the detail still comes from the color texture: the two are sampled separately and multiplied.

Lightmap coordinates

A surface's ordinary texture coordinates (Chapter 11) usually repeat: a floor tile texture is laid down many times over, and both sides of a symmetrical model may share the same part of their texture. A lightmap can't share. Every point of every surface gets different light, so it needs its own texel. Each object therefore gets a second set of texture coordinates, just for the lightmap, laid out so that no two points share a texel:

• Unwrapping. Each object's surface is cut into pieces that lie flat without stretching, called charts. A box unfolds into six squares; a curved surface is cut along its sharper edges.
• Even texel density. Every chart is scaled to the same number of texels per meter, so a big wall and a small box get equally sharp lighting.
• Packing. The charts from many objects are packed together into one texture, the lightmap atlas, with a gap of a texel or two around each. The gaps matter: a texture filter blends neighboring texels, and without a gap, one chart's light would bleed onto its neighbor.

Modeling tools and engines do the unwrapping and packing automatically, but the result is worth seeing.

Baking

Baking visits every texel of the atlas. From the chart it belongs to, it works out the point on the surface at the texel's center and the surface's normal there, then gathers the light arriving at that point: from each light, with a shadow ray to see if it's blocked, and from every other direction, by tracing rays into the scene and finding what they hit, bounce after bounce, as the path tracer of Chapter 15 does. The figure bakes in a fraction of a second because it's tiny and allows only one bounce; a real level, at full quality, takes minutes to hours.

A lightmap stores the light arriving at each point, its irradiance, not the color you see. That's what lets the color texture stay sharp and separate: the renderer multiplies the two at each pixel. It also means the same baked light works for any surface color that's later swapped in, as long as the swap doesn't change what light bounces where.

A few problems come up again and again:

• Seams. Two charts that meet in the room, such as the floor and a wall, sit apart in the atlas, and their texels are baked separately. Where they meet, their light doesn't quite match, and a line shows. Bakers sample right up to the chart edges and blend across known seams to hide it.
• Gaps and bleeding. Bilinear filtering near a chart's edge reads texels from the gap around it. Bakers fill the gap by copying each chart's edge texels outward (dilation), which is what "fill gaps" does in the figure.
• Texels inside walls. A texel near a corner can have its center just behind the neighboring wall, where the bake sees only darkness, or where it sees light from the far side of the wall. Bakers detect such texels and move their sample point back into the open.
• Resolution. The lightmap's texel density limits how sharp its shadows can be. A higher density sharpens them, at the cost of memory and baking time, both of which grow with the square of the density.

Lightmaps are still everywhere, in architecture visualization and in games with fixed scenery. Modern ones store more than one color per texel: directional lightmaps also record the main direction light arrives from, or a few spherical harmonics numbers (Part 3), so that bumps in a normal map still catch the light correctly. They're stored as HDR textures (Chapter 25), compressed with GPU texture formats. But they have a hard limit: change anything, open a door, move the sun, and the baked light is wrong. And they only cover the static scenery; moving objects need something else.

That something else is the light probe: a record of the light arriving at one point in space, from every direction. Scatter probes through the scene, usually on a regular 3D grid, and any surface, static or moving, can look up the light arriving near it by blending its nearest probes. A probe is like a small, very blurry spherical photograph of its surroundings.

Storing a sphere of light in a few numbers

A probe needs to store how much light, of what color, arrives from each direction. Storing a full picture per probe would cost far too much memory for thousands of probes. But diffuse surfaces (Chapter 12) gather light from a whole hemisphere at once, which smooths away all fine detail: a diffuse surface can't tell a small bright window from a big dim one. So a very blurry summary of the light is enough, and blurry things can be described with very few numbers.

In code, storing light this way is a running sum. For each direction d you look in, with light L arriving from it, add L times each wave's value in that direction. The first wave is the same everywhere; the next three are just d.x, d.y and d.z, each times a fixed size. This is what each of the hero's probes does with its 128 rays:

// for every ray direction d, with light L arriving along it
sh[0] += L * 0.282095;
sh[1] += L * 0.488603 * d.x;
sh[2] += L * 0.488603 * d.y;
sh[3] += L * 0.488603 * d.z;
// after all the rays: the average, times the sphere's area
for (var k = 0; k < 4; k++) {
  sh[k] *= 4.0 * PI / f32(RAYS);
}

Reading the light back for a surface facing n undoes the sum, with one twist. A diffuse surface doesn't want the light from direction n; it wants all the light over its hemisphere, each direction weighted by how squarely it hits (Chapter 12). That blurring has a neat effect on the waves: it keeps the average, keeps two-thirds of the x, y, z waves, and only a quarter of the next band. So the blur costs nothing: it's folded into the constants.

// the light gathered by a surface facing n
// 0.886227 = pi * 0.282095, 1.023328 = 2pi/3 x 0.488603
fn irradiance(sh: array<vec3f, 4>, n: vec3f) -> vec3f {
  let E = 0.886227 * sh[0] + 1.023328 * (sh[1] * n.x + sh[2] * n.y + sh[3] * n.z);
  return max(E, vec3f(0.0));
}

The playground does the full nine-number version on the CPU. It measures an environment (a blue sky, a brown ground and a bright window) from 2,000 directions, and draws three balls: a mirror ball showing the environment itself, a white diffuse ball lit by adding up all 2,000 directions (the true answer), and the same ball lit from the SH numbers. Set BANDS to 1, 2 or 3 and compare the last two; then edit env.

Filling the probes

Probes can be baked like lightmaps, which is fine when the lighting is fixed. To follow moving lights, the probes must be updated as the game runs. In the hero image at the top, each of 144 probes traces 128 rays every frame (Chapter 15), works out the light arriving along each, and folds the result into its four SH numbers per color, blending it with what it measured before so the noise averages out over frames.

Rays are random, so each frame's measurement is a little noisy, and noisy probes make the whole room flicker. Three standard remedies are at work in the hero: spread each frame's rays evenly over the sphere (a spiral of directions, turned by a random rotation each frame) rather than purely at random; keep most of the previous value (97% here), so the noise of dozens of frames averages out; and, so that the probes still react quickly when the lamp moves, keep less of the old value wherever the measured light changed a lot. This trade-off, smooth but slow versus responsive but noisy, comes up in every technique that accumulates over time (Chapter 31).

How does a probe ray know the light at the surface it hits? Direct light is easy: test a shadow ray to the lamp, as in Chapter 13. For indirect light at the hit point, the trick is to ask the probes themselves, as they were last frame. One bounce comes from this frame's rays, and the bounce before that from last frame's probes, and so on: frame by frame, the light accumulates any number of bounces. Choose "probes: many bounces" in the hero and watch the room fill with light. This is the heart of dynamic diffuse global illumination (DDGI), published by Zander Majercik and colleagues in 2019.

Leaks

Probes sit on a grid that knows nothing about walls. A surface on one side of a thin wall may blend probes from the other side, and pick up their light: a bright stripe along the bottom of a wall in a dark room, or dark blotches near a probe that ended up inside a solid object. Fighting these leaks is most of the work in a probe system. Renderers switch off probes that land inside geometry (the hero does that: turn on "show probes" and the ones inside the tall box are red), weight probes by whether they sit in front of the surface, and, in DDGI, test whether each probe can see the point being lit.

Here is how the hero blends the eight probes at the corners of the grid cell around a point p: the usual trilinear weights (Chapter 11, in 3D), with probes inside boxes skipped and probes behind the surface turned down:

let g = (p - GRID_MIN) / GRID_STEP;    // position in grid cells
let base = vec3i(floor(g));
let f = g - vec3f(base);               // how far across the cell, 0 to 1
var sum = vec3f(0.0);
var wsum = 0.0;
for (var k = 0; k < 8; k++) {
  // one of the cell's 8 corners
  let o = vec3i(k & 1, (k >> 1) & 1, (k >> 2) & 1);
  let pp = probePos(base + o);
  if (insideBox(pp)) {
    continue;
  }
  let tri = mix(1.0 - f, f, vec3f(o));  // trilinear weight, per axis
  let facing = (dot(normalize(pp - p), n) + 1.0) * 0.5;
  let w = tri.x * tri.y * tri.z * (facing * facing + 0.05);
  sum += irradiance(probe(base + o), n) * w;
  wsum += w;
}
let E = sum / max(wsum, 1e-4);

The grid, the probes' stored light and the solid boxes are the scene's; here is the hero's version (irradiance is the function from Storing a sphere of light in a few numbers, above):

const GRID_MIN = vec3f(-2.5, 0.35, -2.5);   // the corner probe's position
const GRID_STEP = vec3f(1.0, 0.9333, 1.0);  // spacing between probes
const GRID_N = vec3i(6, 4, 6);              // probes along x, y and z

// each probe's four SH numbers per color, probe after probe
@group(0) @binding(1) var<storage, read> probeSH: array<array<vec3f, 4>>;

// the probe at grid cell c: its place in the list, position and light
fn probeIndex(c: vec3i) -> i32 {
  return (c.z * GRID_N.y + c.y) * GRID_N.x + c.x;
}
fn probePos(c: vec3i) -> vec3f {
  return GRID_MIN + vec3f(c) * GRID_STEP;
}
fn probe(c: vec3i) -> array<vec3f, 4> {
  return probeSH[probeIndex(c)];
}

// the two solid boxes in the room; a probe inside one sees only darkness
const BOX_MIN = array<vec3f, 2>(vec3f(-1.9, 0.0, -1.9), vec3f(0.4, 0.0, -0.4));
const BOX_MAX = array<vec3f, 2>(vec3f(-0.7, 2.2, -0.7), vec3f(1.6, 1.1, 0.8));
fn insideBox(p: vec3f) -> bool {
  for (var k = 0; k < 2; k++) {
    if (all(p > BOX_MIN[k]) && all(p < BOX_MAX[k])) {
      return true;
    }
  }
  return false;
}

Dividing by the total weight matters: when probes drop out, the remaining ones share their weight, so the light doesn't simply dim. DDGI's visibility test, below, is one more factor in w.

Tracing a ray from every shaded pixel to each of its eight probes would be far too slow. DDGI's trick is to let each probe remember how far its rays traveled in every direction: alongside the light, each probe keeps a small map of the average distance to the nearest surface, and the average squared distance, blurred a little across nearby directions. To shade a point, look up the map in the direction of the point. If the point is closer than the average distance, the probe can see it. If it's farther, a wall is probably in the way, and Chebyshev's inequality, the same one variance shadow maps use (Chapter 20), turns the mean and variance into a smooth estimate of the chance that the probe sees it. Its weight is scaled down by that chance (cubed, to sharpen it).

It's one more factor in the blend loop above. distanceMoments looks up the probe's stored average distance and average squared distance in the direction from the probe toward the point (a small octahedral texture per probe in DDGI; a ring of directions in the figure below):

let toPoint = p - pp;
let r = length(toPoint);
let m = distanceMoments(base + o, toPoint / r);  // (mean, mean of squares)
var chance = 1.0;
if (r > m.x) {
  // farther than this probe's rays usually reach: maybe behind a wall
  let variance = max(m.y - m.x * m.x, 0.01);
  chance = variance / (variance + (r - m.x) * (r - m.x));
}
w *= chance * chance * chance;

Each probe's distance map is a small square tile in one big texture. The octahedral mapping flattens the sphere of directions onto that square: fold it into a diamond, then fold the bottom half's corners out:

// every probe's map, tile after tile; each texel holds (mean, mean of squares)
@group(0) @binding(2) var momentAtlas: texture_2d<f32>;
const TILE = 16;           // each probe's map is 16 x 16 texels
const TILES_PER_ROW = 24;  // probes side by side in one row of the atlas

fn distanceMoments(c: vec3i, dir: vec3f) -> vec2f {
  let i = probeIndex(c);
  let corner = vec2i(i % TILES_PER_ROW, i / TILES_PER_ROW) * TILE;
  let uv = octahedral(dir) * 0.5 + 0.5;   // -1..1 becomes 0..1
  let texel = corner + vec2i(uv * f32(TILE - 1) + 0.5);  // nearest texel
  return textureLoad(momentAtlas, texel, 0).rg;
}

// a direction becomes a point in the square -1..1
fn octahedral(d: vec3f) -> vec2f {
  let p = d.xy / (abs(d.x) + abs(d.y) + abs(d.z));  // onto the diamond
  if (d.z >= 0.0) {
    return p;
  }
  // the back half: fold its triangles out into the square's corners
  let s = select(vec2f(-1.0), vec2f(1.0), p >= vec2f(0.0));
  return (1.0 - abs(p.yx)) * s;
}

This is the same inequality as the variance shadow map's (Chapter 20), with the probe in the role of the light: there, a point farther than the stored depth was probably in shadow; here, it's probably out of the probe's sight.

Probes with a few SH numbers are fine for diffuse surfaces, but a shiny floor reflects a sharp picture of its surroundings, and a few waves can't hold that. Glossy indirect light needs more detail, and it has its own set of techniques:

• Reflection probes: a full picture of the surroundings, taken from a point, stored as a cube map (six square images), placed in each room. Shiny surfaces nearby look up their reflection in it. Chapter 24 shows how to blur them for rough surfaces.
• Planar reflections: for one flat mirror or a lake, render the scene a second time mirrored across its plane (Chapter 26).
• Screen-space reflections (SSR): for every shiny pixel, march along the reflected ray in small steps, projecting each step onto the screen and comparing its depth with the depth buffer, as SSAO does. Where the ray first passes behind the depth buffer's surface, it has hit something, and the reflection is simply the color already drawn at that pixel.

SSR is cheap and catches reflections of moving objects, which baked probes can't. But it can only reflect what's on screen: look down at a puddle and the sky's reflection vanishes, since the sky isn't in the frame. Renderers blend SSR with reflection probes at the edges of what it can find.

Probes and SSR in a real scene

A reflection probe is a picture taken from one point, but shiny surfaces sit everywhere else. Looking up the probe along the reflected ray only works for things infinitely far away, like the sky: for a wall two meters off, the direction from the floor to the wall isn't the direction from the probe to the wall, and the reflection slides around and comes out the wrong size. Box projection (also called parallax correction) patches this with one assumption: the room is a box. Each reflected ray is traced to where it leaves that box, and the probe is looked up in the direction of that point. For walls that really are the box, the reflection lands in the right place. For anything inside the room, like a pillar or a ball, it still doesn't.

Notice where SSR fails. The floor near the camera reflects what is above it, the tops of the walls and the sky, and those rays climb out of the top of the screen. The chrome sphere reflects in every direction, including back toward the camera, so most of its reflection is missing. And behind each box and ball is an orange shadow of misses, where rays pass behind an object the screen holds only the front of. The probe fills those holes with something plausible, and a careful renderer fades between the two instead of switching, so the seam doesn't show.

Each technique so far covers part of the problem with some trade-off. Research keeps producing more general methods, and the list is a fair map of current practice:

• Voxel cone tracing (Cyril Crassin and colleagues, 2011) stores the lit scene in a 3D grid of small cubes (voxels), with blurrier versions at coarser grids, like mipmaps in 3D. A wide cone of directions is then traced through ever-coarser grids, gathering a whole hemisphere of light in a few lookups.
• Screen-space GI extends SSR's idea to diffuse light: trace short rays through the depth buffer and pick up the colors they hit. Cheap, but limited to what's on screen.
• Radiance cascades (Alexander Sannikov, 2023) start from an observation about probes: nearby light needs many probes but only a few directions, since it changes quickly with position but its directions are few and wide; faraway light is the opposite, needing many directions but few probes. So measure light in cascades: a dense grid of probes with a few short rays, then a grid with half as many probes across, four times as many rays and a range four times as far, and so on. Every cascade costs about the same, the total is small, and merging the cascades from far to near gives every pixel light from all distances, with no noise.
• Surfel and radiance-caching methods scatter small light-measuring disks (surfels) over the visible surfaces, or cache light in a hierarchy of probes near the camera, and update them with a budget of ray-traced samples per frame.
• Hardware ray tracing (Chapter 14) plus heavy reuse: trace just one or two rays per pixel per frame, then share good light paths between neighboring pixels and frames (ReSTIR, Benedikt Bitterli and colleagues, 2020) and remove the remaining noise with a denoiser, a smart blur that respects edges and uses the previous frames. Some recent games run a real path tracer this way.

All of them share the shape of this chapter's hero: measure light sparsely (in probes, voxels, surfels or a few rays), reuse the measurements across space and time, and spend the rays where they matter.

• Indirect light, light that has bounced off other surfaces, is a large part of what we see. Global illumination is direct plus indirect.
• Ambient occlusion darkens ambient light by how hemmed in each point is. SSAO estimates it from the depth buffer, and assumes everything behind the front surfaces is solid.
• Lightmaps bake bounced light into textures: perfect, but static.
• Light probes record the light arriving at points in space, compressed with spherical harmonics. Updated with rays each frame, and reading last frame's probes at the hits, they accumulate many bounces. Probes leak through walls unless the renderer tests visibility.
• Glossy reflections need sharper data: reflection probes, planar reflections, and screen-space reflections, which can only reflect what's on screen.
• Voxels, surfels, and ray tracing with heavy reuse and denoising are pushing real time toward the path tracer.

We've been treating every surface as a perfectly matte one. Chapter 23 gives them real materials: metal, plastic, wood, and everything in between.

1. Keep the edges sharp. The SSAO playground's blur averages each pixel with its whole 4 × 4 block, even neighbors on a different surface, so along silhouettes (a pillar against the floor behind it) the AO of one surface smears onto the other. Make the blur skip neighbors whose depth is too different.

   Hint

   In aoAt, read each neighbor's depth from posTex as well, and add its AO only if abs(neighbor.z - here.z) is small (say, under 0.1 plus a few percent of the distance), counting how many you added and dividing by that count instead of 16. That's a depth-aware or bilateral blur, the kind real renderers use. The noise doesn't cancel quite as perfectly at edges, but it's far less visible than a smeared halo.

2. Bent normals. While computing AO, also average the directions of the unblocked test points. What does that average direction tell you, and how could lighting use it?

   Hint

   It points toward where the open sky is: the "bent normal". Looking up the probe or sky light in that direction instead of along the surface normal makes ambient light come from where it can actually arrive.

3. A small bright window. In the SH playground, make the window six times smaller and six times brighter: change 0.97 to 0.995 and 5 to 30. The mirror ball changes a lot. What happens to the two diffuse balls, and why is that good news for probes?

   Hint

   They barely change. The window's patch of sky shrinks from about 0.19 to 0.03 of the sphere's 4π, so six times the brightness brings in the same total light, and a diffuse surface only feels the total. That's the "can't tell a small bright window from a big dim one" from the text, and it's why nine numbers are enough for diffuse light, but not for a mirror.

4. Count the bounces. In the hero, switch from "direct only" to "probes: many bounces" and watch the ceiling. Why does it take a moment to brighten fully?

   Hint

   Each frame adds one more bounce, and each probe blends its new measurement with 97% of its old one (less when the light changes a lot), so light builds up over a few dozen frames. The same blending causes a slight lag when the lamp moves.

5. SSR's blind spot. In the reflection figure, find a floor point whose reflection would hit the pole if the depth buffer knew the pole's thickness. What would you store to fix it?

   Hint

   A second depth buffer holding the back faces (rendered with front-face culling) tells you where each object ends, so a ray between front and back depth is a real hit, and a ray behind the back depth has passed it.