Taos Engine

Gaussian Splats

Splat from Video — In-Browser Structure-from-Motion

Run the sample · splat_video.ts · src/splats/sfm/

The Splat Trainer cheats: Taos Engine renders their training views, so every frame arrives with its exact camera. This sample removes the cheat. A drone video (or a folder of overlapping photos) goes in, and the camera poses have to be recovered before a single splat can be trained — the same problem every real-world 3D Gaussian Splatting pipeline solves with an offline COLMAP stage. Here that stage runs inside the tab:

video / photos          structure from motion                      train / view
┌──────────────────┐   ┌─────────────────────────────────────┐   ┌────────────────┐
│ pick sharp       │   │ SIFT (worker pool) → GPU matching   │   │ undistort →    │
│ frames (Laplace  │──►│ → tracks → focal search → two-view  │──►│ seed from the  │
│ variance) →      │   │   init → incremental PnP + interim  │   │ sparse cloud → │
│ decode two res   │   │   BA → sparse Schur BA (f, k1, k2)  │   │ GsTrainer      │
└──────────────────┘   └─────────────────────────────────────┘   └────────────────┘

The solver is a port of the open-source splat.js front end (MIT), whose poses match COLMAP's to 0.00% of path length on the Tanks & Temples Truck scene. The trainer behind it is the same sorted-tile GPU trainer the other splat samples use (gs_trainer.ts).

Frames from a video#

A video is not a photo set. Consecutive frames are nearly identical (no parallax, no information), a drone's gimbal pans produce motion-blurred frames that would teach the model blur, and an edited clip is several unrelated scenes — only frames of one shot can be solved together. video_frames.ts handles that in stages, and the sample puts a range editor between them:

  1. Scan — play the clip muted at 3× with requestVideoFrameCallback and, every ~0.1 s of video time, draw a 320-px grayscale and keep three numbers: sharpness (Laplacian variance — blur kills second derivatives), the mean frame difference, and the mean difference of 16×9 block means. The last one is the shot-cut signal: a few pixels of motion barely moves a 20-px block average; a new scene moves most of them. (Plain pixel differences are useless for cuts on aerial footage — dense texture makes 0.1 s of motion change as many pixels as a new scene.) The scan is the slow part of opening a long clip (~2 min for 6 min of 1080p), so it is cached in localStorage per file.
  2. Shots — detectShots turns the block-difference signal into shot boundaries. A cut must be prominent against the ±2 s local baseline, isolated (≥ 2× its immediate neighbors — a fast pan raises the whole neighborhood, a cut is one spike), and among the clip's largest differences overall; a weaker spike is still accepted when the sharpness level changes abruptly across it. Measured on the Santa Fe clip: the six real cuts stand 4–180× above their neighbors, motion jitter at most 1.5×.
  3. Trim — before any scan, the raw video is shown on the timeline: scrub or play it, drag the in/out handles (or press I / O at the playhead) around the section you want, and press Scan range. Only that section is scanned — for a long clip where you already know the interesting stretch, this is the difference between a 13-second scan and a two-minute one. (The whole video is the default selection, so you can scan everything if you want the shot detector to split an edited clip for you.)
  4. Range editor — the timeline now zooms to the scanned section and shows the sharpness curve, the shot bands and cut markers; drag the in/out handles, click a shot chip, scrub or play the preview, set the frame count and the sampling policy. The default selection is the longest shot.
  5. Select — selectFrameTimes picks N frames inside the range: even spacing, sharpest per slot (equal time slots, the sharpest scan sample of each — even baselines for SfM, blur avoided) or sharpest overall with a minimum gap (the original splat.js policy; it tends to bunch frames in the sharpest few seconds).
  6. Capture — seek to the chosen times in order (forward-only decode is fast) and encode full-resolution JPEGs.

Frame spacing is the number that decides whether SfM succeeds. Consecutive frames must overlap enough to share tracks. On the Santa Fe flyover — a fast, low pass — 100 frames spread over the whole 3-minute first shot (1.9 s apart) registered 2 cameras; 100 frames over 40 s (0.4 s apart) registered 100/100. The editor shows the resulting spacing, warns when it gets wide, and the fit 0.4 s spacing button shrinks the range from its start so the chosen frame count lands at that spacing. To cover a longer flight, raise the frame count rather than the spacing.

Those JPEGs then go through the same decoder as photos (frames.ts), which produces two resolutions of every frame: a ≤960-px grayscale for feature detection (feature localization is the pose-precision ceiling, and 960 measurably beats 640), and an RGBA8 training target at the chosen train resolution. The frame's sharpness rides along: the blurriest 45%-below-median cameras keep their poses but are excluded from the loss.

Structure from motion#

sfm.ts is incremental SfM in the COLMAP tradition, with the pieces that matter for quality spelled out:

Everything runs in normalized image coordinates (OpenCV convention: x right, y down, z forward, p_c = R p + t), which is exactly what the trainer's GsCamera expects — no axis flip on the way in. Only the focal length is rescaled from feature scale to training scale (recon_to_trainer.ts).

From reconstruction to splats#

The trainer models a pinhole; the solver found a lens. So the training images are undistorted by the solved k1, k2 (bilinear resample, out-of-frame samples get alpha 0 and leave the loss), the sparse cloud seeds the initial Gaussians (initGaussiansFromPoints, with jittered clones up to ~60k), and the trainer starts with two options the synthetic samples leave off:

The rest is the familiar loop: adaptive batching, MCMC-style refinement, holdout PSNR on a sharp mid-sequence frame, plateau auto-stop, View 3D through GaussianSplatPass (the SfM world is y-down, so the pass's 180° X flip is on and the free camera is parked at a training camera's pose), and a standard .ply export.

Trying it#

The unit gates live in tests/splats/sfm/: geometry (two-view, PnP, triangulation with outliers), bundle adjustment on a drifted 25-camera chain with a distorted lens, global rotation averaging on a noisy orbit, and SIFT on a procedural texture.