Testing an operating system with no test runner

There is no cargo test here. This is a bare-metal binary with no host runtime, so a test that runs on the development machine is testing something other than the system. Verification is four instruments.

1. Self-tests at boot

Twenty sections run during boot, printing a claim and whether the machine met it. Thirteen of them are registered as named suites that can be re-run on demand. Heap, timer, clock, the Merkle addressing, fifteen sets of published cipher vectors, fault handling, constrained decoding, the agent loop, the linear probe, the situation planner, the initiative policy, the self-modification gate, corpus bundles, adapters, the backward kernels and the trainer's arithmetic.

Registration is deliberately awkward: adding a suite requires a slot in a results table and a compile-time assertion that the two lengths match, so a suite added without a slot fails the build instead of silently never recording a verdict.

Reading that output is the test suite, and skipping it has cost real time twice. A signature verification break sat visible in the log for a full debugging cycle while the output was being filtered to the section under work.

2. A scripted emulator harness

A driver script boots the emulator and drives the shell over a serial socket, so a change can be booted and exercised without a human at a keyboard. It stages the release binary, resets firmware variables to a known state, and attaches serial as a socket, because the emulator's console channel on this host reads console handles and ignores redirected files.

Three things are worth knowing before a session goes sideways. The resident mind wakes fifteen seconds in and holds the model for a whole episode, so the first two commands should always stand it down. The hypervisor accelerator is about 160 times faster than pure emulation on this workload, and everything previously treated as too slow to test here was an untested assumption about the emulator. And the release artefact is what gets staged, so a debug build alone leaves the change under test unbooted.

3. A numeric oracle

A host-side reference implementation reads the same converted checkpoint and prints logits for a token sequence, so a conversion bug shows up there too and only a kernel bug shows as a mismatch. Comparing generated text against a known fact is the cheap end of the same instrument, and it is the one that caught the rotary embedding.

The gradient checks work the same way, and they are directional rather than per-entry for a reason of resolution. Differencing one parameter asks floating point to resolve a quantity below the loss's own rounding, and it reported a value that was a float quantum wearing the costume of a derivative. Stepping every parameter along the gradient asks it to resolve something far larger, and checks the whole vector: 1.148 analytic against 1.135 numeric over 57,600 entries.

4. A differential harness with a canary

The newest instrument runs one program two ways and requires the results to agree on value, step count and error text, bit for bit with no tolerance. Any difference at all is a bug and a tolerance would hide the one worth finding.

The part that makes it a harness rather than a ritual is the canary. A suite that has never reported a difference is indistinguishable from one that compares nothing, so it runs two programs that differ by a single unused declaration, giving identical answers and step counts one apart, and it fails if that is not caught. That is the difference a comparison looking only at answers would wave through, and it earned its place immediately by catching a genuine miscount the first time a code generator was compared against the interpreter.

The corpus runs sixty-four times, which is a number arrived at the hard way elsewhere in this tree: a one-shot check of the multiprocessing split passed while the implementation still deadlocked.

The habit underneath

Measure rather than argue, and publish the measurements that overturned the argument. A renderer survey ranked several changes and the real win was none of them. A design panel predicted a benefit of five to seven and the system reported two. A plan called it near certain that a tree walk was the cost, and the tree walk turned out to be eight percent of it.

Negative results stay in the tree for the same reason. Training the classifier head hurts at this data scale. The product-of-experts ensemble does not improve accuracy. Both are kept, because the reason to know them is the reason they were worth measuring.