The router that never runs the model
The most common decision this system makes is which applet to run for a given instruction. There are two implementations. One of them uses the language model. The other reads a single hidden state and hands it to a statistical method from 1960 — and it is more accurate on held-out data.
That result is the subject of this piece, along with the number that turned out to matter more than accuracy did.
Two ways to decide
The first path decodes the applet name token by token under a grammar, so only names that exist are spellable. It is the obvious design and it works. It also costs a transformer forward pass per token of the name.
The second path never generates anything. It takes the hidden state the model has already computed for the instruction — one vector — and runs a linear classifier over it.
| Method | ridge regression |
|---|---|
| Solved by | Cholesky, in kernel |
| Parameters | 12,672 |
| Time | ~1.6 ms |
| Forward passes | 0 |
A 1960 algorithm
The classifier is Widrow-Hoff: least squares with a ridge penalty, which has a closed-form solution. There is no gradient descent, no learning rate, no epochs and no early stopping. You build the normal equations, factor them with Cholesky, and substitute. The answer is exact, it is the same answer every time, and on this problem it takes about 1.6 ms.
Twelve thousand parameters against a checkpoint of several hundred million. It is worth being clear about why that is not as absurd as it sounds: the hard work has already happened. The transformer has read the instruction and produced a representation in which the distinctions the router needs are close to linearly separable. The router is not doing the understanding; it is reading a decision that is already present.
Which also explains the economics that make everything else in this system affordable. The base model is frozen, so a hidden state is a constant, so it can be cached once per example and reused for any number of candidate routers. Training becomes cheap, judging becomes nearly free, and a verdict reached last month can be re-checked today for the price of a dot product. Anything that proposes to fine-tune the attention path is proposing to give that up, which is a real trade and worth naming out loud before it is made.
Three cores, and the real signal
Three independent methods answer the same question: the ridge probe, a hashed n-gram Bayes classifier, and a lexical matcher. Each has a different failure mode, which is the only reason having three is worth anything.
The probe alone is right about 54.7% of the time. That is the number people quote, and it is the least interesting one on this page.
| All three agree | 90.3% |
|---|---|
| They split | ~50% |
| Probe alone | 54.7% |
The gap is what the system acts on, not the accuracy.
When all three land on the same applet, the answer is right 90.3% of the time. When they split, it is a coin toss. That gap is far more useful than the headline accuracy, because it is available before committing to the answer, and it costs three cheap classifiers rather than a forward pass.
So the system does not use the vote to pick. It uses the agreement to decide whether it is confident enough to act at all. Unanimous, and it runs the applet. Split, and it falls through to the expensive path that actually generates. Most decisions are unanimous, so most decisions cost microseconds.
The physics metaphor that a related research effort was built around survives here in its honest form: agreement as gravity, the core deciding. It turned out to be a gate, not a field.
Two results that did not work
Both stay in the tree, and both are on this page for the same reason.
Training the adapter head hurts. At this data scale it makes routing worse, not better. The obvious move — more capacity on the decision layer — is the wrong one here, and knowing that is worth more than the experiment cost.
The product-of-experts council does not improve accuracy. Combining the three cores' distributions properly, rather than counting agreement, was the theoretically better thing to do. It is not better. The crude count is what ships.
A negative result is only worth keeping if the reason it was worth measuring still holds afterwards, and it does in both cases: somebody will propose both again.
Why the numbers are trustworthy
None of the above means anything without saying how it was measured, and this project got that wrong three separate times before settling it.
A grid sweep was once scored on the test set. Cross-validation was once folded by template family in a way that leaked. And the test set itself used to move, because it was defined positionally and the corpus was being appended to — so every addition silently redefined what "held out" meant.
What fixed it:
- Three splits, not two. Validation is spent freely; the test slice is read once.
- Whole template families are held out, never sampled instances. Instances within a family differ only by slot values, so an instance split measures memorisation while looking like generalisation.
- One place asks for the boundaries. Reading the constants directly is the bug the arrangement exists to prevent.
- The test slice carries a budget. Past three reads, a figure prints as stale and marked unquotable — because a system that improves itself nightly reads the held-out set nightly, and each read makes the number more optimistic.
Sample sizes are stated wherever a figure appears. Numbers from runs too small to support a claim do not become one.