A model can be wrong without being broken

2026-09-01Aperture InstituteGLaDOS v1.2.28

Writing a transformer forward pass from scratch produces a particular kind of bug, and it is not the kind you are braced for. Nothing crashes. No value goes non-finite. No assertion fires. The model loads, runs at the expected speed, and writes fluent, grammatical English.

It is simply wrong, and it looks exactly like a small model being small.

A class of bug with no symptom

This is the trap. A 135-million-parameter model is supposed to be mediocre. When it answers a factual question with something confident and false, the honest first hypothesis is that it is a small model. Every one of the bugs below hid behind that hypothesis, some of them for weeks.

Four are worth writing down, because each is a different way to be plausibly wrong.

RoPE, paired wrongly

Rotary position embedding rotates pairs of dimensions by an angle that depends on position. The question is which dimensions pair up. There are two conventions: interleaved, pairing 2i with 2i+1, and rotate-half, pairing i with i + head_dim/2. Anything trained through the mainstream implementations uses the second.

This kernel used the first, for a long time, and nothing looked broken. Here is why: both are norm-preserving rotations by exactly the same set of angles. No value grows, nothing goes to infinity, no numerical drift accumulates. The model remains perfectly well behaved and attends according to a scrambled notion of distance.

Note that the wrong output is not gibberish. It is a well-formed English sentence that simply stops. That is indistinguishable from a small model losing the thread, which is what it was assumed to be.

Two things Qwen3 states

Qwen3 differs from Llama in two ways, and neither fails loudly.

Head width is stated, not derived. The natural assumption is head_dim = hidden / heads. For this model that would give 64. It is actually 128, declared in the config, so the query projection is [2048, 1024] and the attention path is wider than the residual stream. Derive it instead of reading it and every matrix still multiplies, every shape still lines up, and the attention is computed over the wrong subspace.

Queries and keys are normalised. Qwen3 applies RMSNorm to each head's query and key before RoPE. Skip it and, again, nothing errors.

Both are now recorded per checkpoint in the file header rather than inferred, and older files keep loading because their defaults are exactly the Llama ones. The general lesson is narrow and useful: a quantity you can derive and also read should be read, because the derivation encodes an assumption that some architecture will eventually break, silently.

Twelve per cent of the tokens

Before a tokenizer merges anything, it splits text with a regular expression. SmolLM2 uses the GPT-2 pattern. Qwen3 uses the cl100k one, in which a word may be led by any non-alphanumeric character, digits arrive one at a time, and punctuation swallows the newlines that follow it.

Using the wrong pattern moved about 12% of tokens on the training corpus. Not 12% of characters mangled — 12% of tokens different, silently, with no error anywhere. The model was being fed sequences of a shape it had never seen during training, and it responded the way models do: by producing something fluent and worse.

The tokenizer now carries which pattern its checkpoint trained with, and the host-side converter has a --verify mode that reimplements the kernel's algorithm and diffs it against the reference library. That reimplementation is deliberate. A verifier that shares code with the thing it verifies agrees with it by construction.

Two paths, one position

The last one is not about the architecture, and it is the one most likely to recur in any codebase.

Processing a prompt and generating a token are separate code paths — one batched, one incremental. For a long time the batched path ignored adapters entirely. So an adapted model would process its prompt through the frozen weights and then generate through the adapted ones: the same position computed two different things depending on which path reached it.

Nothing faulted. No logit went non-finite. And it was unreachable until the first adapter anybody wanted to keep got attached, so it sat there correct-looking for as long as nobody used the feature.

A related one, worth a sentence because it generalises: a debug_assert in this tree checked that a dimension matched, and the check itself was wrong — it compared against the wrong length. It never fired, because the whole system is driven in release builds where debug_assert compiles to nothing. A wrong assertion that never runs is indistinguishable from a right one.

The only two things that settle it

None of these has a symptom, so no amount of staring at the output resolves them. Two things do.

A numeric oracle. A reference implementation of the same forward pass, in a different language, on the same checkpoint, compared logit by logit. Ours reads the converted file rather than the original, which is the detail that makes it useful: a bug in the conversion shows up there too, so only a genuine kernel bug appears as a mismatch between the two.

Output containing a fact you can check. The cheap end of the same test. An instruction-tuned model whose attention path is wired correctly will complete “the capital of France is” correctly. One with scrambled positions will write a fluent sentence that does not. It takes seconds and it caught the RoPE bug in the end.

What does not work, and was tried: reading the code carefully. Every one of these bugs was read past repeatedly by people who knew what the code was supposed to do. They are invisible precisely because the wrong version is a reasonable thing to have written.

Back to research →