Runtime and safety

Underneath all of the above there is a layer that is mostly unglamorous and is the part I would keep if I threw everything else away.

The rig runs nine subsystem processes plus a supervisor that restarts them, not threads. Separate address spaces, because a thread boundary is not a fault boundary and this helmet is opaque. A segfault in the ultra-wideband solver, or a runaway loop in the chest panel, must not take the render loop with it and blind somebody who is running indoors. The render process also needs real time scheduling on isolated cores, which a monolith cannot give to one subsystem without giving a busy panel redraw the ability to preempt the renderer. The alternative, one process with shared pointers, is simpler and faster on paper, and it loses on fault isolation, which outranks everything else here.

The cost of that choice is inter-process communication, and the design exists to make it near free. Shared memory only, no sockets between local processes, no serialisation, and frame pixels never cross a process boundary, only descriptors do. A single copy of a 2560 by 1200 canvas at 60 Hz would be 1.5 GB/s of pure waste against a budget that already has none to spare.

There are exactly two primitives, and the rule for choosing is one sentence:

If a reader that missed an item would be wrong, use a queue. If it would merely be out of date, use a latest-value slot.

A renderer that misses a head pose update is out of date for 16 ms and then catches up. A robot controller that misses an emergency stop is wrong. That is the entire distinction, and having it written down means the choice stops being a matter of taste.

Both primitives have exactly one writer per segment, statically assigned, because neither can detect a second one. Two writers corrupt a latest-value slot's sequence counter in a way no reader can distinguish from a torn read; two producers on a queue interleave payloads. Adding a runtime guard would cost an atomic operation on the hot path, so ownership is a table instead. The queues drop the new item when full rather than overwriting the oldest, because they carry commands, and discarding something already queued is worse than refusing something new. Drops are counted, never silent, because a full command queue means a consumer is wedged.

Two more rules that are load bearing:

• Time is monotonic nanoseconds, everywhere, with no wall clock in the runtime at all, not even for logging. A clock step backwards makes a time delta negative, which makes a covariance update produce a non-number, which makes the HUD draw contacts at infinity. Silently, once per match, at random.
• Watchdogs are split in two, and the split is the right mental model. An in-loop deadline check asks "did my own last iteration take too long", with no system calls, because it is the only thing that can react inside one frame period. A cross-process heartbeat table asks "is that other process still alive", which catches the case the first one structurally cannot: a process that is not slow but wedged, and is therefore not running any code that could notice. Both have hysteresis, because a loop sitting exactly at its budget would otherwise flip the display between fused and raw passthrough several times a second, which is worse than either state and is a genuine nausea trigger.

Everything else in this project trades against latency, cost or effort. This does not.

The reason is specific: the system puts an opaque, fully enclosing helmet on a person who then runs around a building at speed. Vision is entirely mediated by software. That single fact is what most of the architecture bends around, and it converts a set of engineering preferences into requirements.

A physical bypass comes before anything else. A switch has to route the display to a direct camera feed with the compute module out of the circuit entirely, or the visor has to flip up. Not a software mode, not a fallback in the renderer: a circuit that works when the board is powered off, kernel panicked, or thermally stalled. A safety review on this project caught that the single most important requirement in the safety document had no part number, and that every software component simply assumed it existed. The parts to fix that come to $180 to $290, roughly 5% of a rig, and nothing else is allowed to be ordered first.

Never freeze a frame. A frozen HUD is more dangerous than a black one, because it looks alive and the wearer keeps acting on a world that stopped existing several seconds ago. If the render loop misses frames it falls through to raw passthrough with no stitching, fusion or overlay. If capture itself has stalled it goes black with a hard warning. Black is honest. Stale is a lie.

That produces a degradation ladder with a rung for each failure. Lose thermal and the thermal HUD elements are removed rather than left showing the last known warm bodies. Lose one camera of six and the remaining five are stitched with the dead sector marked, which is the hatched wedge in the render further up this page. Lose the sensing tiers and every contact is removed from the display rather than left decaying on screen. Lose the position mesh and the map stops being written at all. The ladder is written for the full sensor set rather than for what is currently fitted, which is why it still has a rung for losing eye tracking on a rig that has no eye cameras in it.

Each rung has to be independently testable and the test suite deliberately faults each one, because a safety fallback that has never been exercised is not a safety fallback.

The warning band itself is instructive about how this is built. It is drawn straight into the framebuffer after the lens warp, deliberately bypassing the HUD's text renderer, palette, geometry and state, so there is no shared code path that could take it down along with the thing it is reporting. It pulses at 1.5 Hz, which is a rationed alarm rate and is well clear of the 3 to 30 Hz photosensitive band.

The rest is unglamorous and mostly physical. Lithium polymer batteries on a person in a contact sport need a hard case and a fuse. The compute module throttles when hot and a helmet in summer is hot, so the degradation path has to be tested at temperature and not at a desk. The wearer must still be able to hear the real world, so both ears are never occluded. Emergency stops on the robot are physical, latching, and require both a software clear and a physical release, so software alone can never un-stop a stopped machine, and the voice command for it is the one intent that bypasses the confirm-before-execute gate, because a stop command that requires confirmation is worse than no stop command.

And the most likely cause of an actual injury in this entire project is not electrical. It is that the wearer cannot see their own feet, and there are stairs.

Twenty four calibration procedures are specified, in dependency order. Zero have been performed, because there are no cameras to perform them on.

The ordering rule is one sentence: anything that changes the pixel grid comes first, anything measured through the display comes last. That is not tidiness. A downstream fit will happily absorb an upstream error into its own parameters and then report a low residual. Camera extrinsics will quietly soak up a bad focal length. The gun boresight will soak up a bad distortion model in the tracking camera. Both then break the moment anything moves, and the residual never warned you.

The part of that document I actually use is the last column of every table, which is what it looks like when it is wrong. Every one of these, done badly, produces "things are drawn in the wrong place", which is a symptom so generic it is useless. So each one has a signature specific enough to tell it from its neighbours:

• Thermal blobs slide off the people they belong to. Which depths are wrong is the diagnosis: wrong at every depth means the intrinsics or extrinsics, correct at 15 m but wrong under 5 m means the near field parallax term.
• Everything in the display is rotated by a constant three degrees at every range and every bearing. A constant offset is the canvas-to-head transform; a range dependent one is the thermal registration.
• Vertical lines lean during head turns. If they lean the other way, the rolling shutter readout time is too large, and nothing else produces reversed lean.
• All ultra-wideband ranges are biased by a constant and the whole constellation sits uniformly inside or outside the true geometry. That is antenna delay, which is per unit rather than per model. Non-line-of-sight error looks different: it is one sided, always long, and specific to one link.
• The robot's reported joint angles disagree with reality, so a policy trained in simulation against true angles receives a distorted observation and produces a limping gait. A sign error makes one joint drive the wrong way under closed loop, which is a hardware damage risk, so it is the first thing to check.

Some of these have real numbers behind them rather than taste. Microphone positions have to be right to about 5 mm because that is 1.9 degrees of bearing error. Stereo range uncertainty grows as the square of distance. The gun boresight error stack is enumerated term by term to explain where its one to two degrees comes from.

There is also an ordering trap that took writing down: the viewing lens calibration has no dependencies at all, but it has to be done before anyone judges display alignment by eye, because until then every geometric assessment is contaminated by the lens itself. Until it is done, judge the canvas from headless image dumps instead. Which is, conveniently, the only thing I can do right now anyway.