A side by side stereo pair of the same dark arena view, each eye showing a barrel distorted image with contact brackets and a white hot thermal figure, framed by amber warning bands at the top and bottom

Personal systems projectPart 1 of 3Drawing 15.1 of 37

Video pipeline

How video gets from the cameras to the display. It covers the exposure solve, the stitch, late reprojection, and the forty five milliseconds all of it has to fit inside.

StatusIn development, 2026
WrittenAbout 129,000 lines of hand-written C++17 across 15 subsystems
Desktop tests105 run, 105 pass, on a laptop with no sensor attached
Not run on hardwareZero lines. Every performance figure below is a budget
OwnedThe helmet display. Everything else is still unbought
TargetJetson Orin Nano 8GB worn on the body, Raspberry Pi 5 on the robot

Sheet 02 of 04/Pipeline

The video pipeline

Every frame goes through capture, exposure balancing, the stitch, thermal fusion and the HUD.

If a camera dies mid run, the stitch marks its sector as dead and doesn't try to cover it up.

The same panoramic arena render with a tall cross hatched amber wedge covering the sector one camera used to see, and a small status readout reading CAM 5 of 6 — *PL 03*Camera two, killed mid run. The stitch marks the dead sector with a hatched amber wedge instead of interpolating over it. The status readout drops from CAM 6/6 to CAM 5/6.

Sheet 03 of 04/Latency

Latency target

The target is motion to photon under 45 milliseconds at the 99th percentile. I use the 99th percentile instead of the average because it's the rare bad frame that makes you sick.

Sheet 04 of 04/Bandwidth

Memory bandwidth

On the Orin, memory bandwidth runs out before the advertised TOPS do, because almost every stage in this pipeline moves a lot of bytes for very little math.