Worth asking?

What was actually tested

The test asked, for each judgement, what did the code do before, what does Jev do, and what does a frontier model do, on the same items, scored against the same answer key.

Four arms:

Each set was then audited by a separate reviewer whose standing assumption was that the benchmark had been rigged, and whose first job was to compute what a trivial baseline scores. On the table-rule set, answering “clean” to everything scores 90.8% accuracy, so the figures reported below are recall and precision, not accuracy.

Qwen 3.8, read as a System 1 sampler

The claim tested: a general model read the way Jev is read — one pass over the prompt, the answer taken from the probability of the next token — does the same job. The arm scripts that measured Jev and Laya ran unchanged against a local shim that answers TypeSafe's API from qwen/qwen3.8-flash: a yes/no question is P(yes) over P(yes) + P(no); a choice is read two ways, as one letter (A, B, C…) and as one yes/no per option, because a single letter carries position bias. Same items, same answer keys, same scorers.

Jev wins routing; Qwen 3.8 wins the art library; the other two are close. On the table rules, every stored run counted per judgement the way the table below counts, Jev catches 92.8% of violations at 96.3% precision over 16 runs and the readout 91.8% at 94.9% over 7: within a point on each. Both move from run to run, Jev between 80.8% and 100% recall and the readout between 80.8% and 100%. Contradictions differ by one statement. Routing is Jev's clearly: the readout's best reading routes 61 of 90 to Jev's 79. On asset kinds the readout, asked one yes/no per kind, scores 87.6% against Jev's 82.5%, and 69.2% against 55.8% on the hard rows.

Read as one letter, the readout loses to itself. The same model and the same questions score 51.1% on routing and 73.5% on assets when the answer is a letter, against 67.8% and 87.6% asked one option at a time. That costs one call per option: a routing question becomes five calls and an asset question six, 7.4 s median per question.

The whole Qwen 3.8 readout, every set and every repeat, 4,402 questions, was billed $0.30 by OpenRouter. No question went unanswered.

Prompted for text with reasoning off, Qwen 3.8 scores 60.0% F1 against Jev's 98.0% on the same scoring, with 16 false alarms to Jev's none. With reasoning on it did not finish the set: at reasoning_effort: "low" each batch of sixteen took 150 to 290 s, two of six passed the client's 300 s limit, and one more returned malformed JSON. Left on its default it, and GLM, spent all 8,000 tokens reasoning and returned no answer, the same failure the skill-selection study found.

Table rules: 19% to 88% of violations caught

94 paragraphs — 68 clean ones that shipped to a real table, 26 written to break exactly one of the three table rules, 51 of the 94 marked hard by the labeller. The rules: never narrate a player's character, never tell the DM when to roll, never end a sentence explaining what a look means.

Arm C calls a frontier model batched, thresholded and asked for a number. Its recall is 96.2% against Jev's 88.0%, with 19 false alarms in 256, thirteen of them on the hardest rule, against Jev's 9 in 2,048.

Against the batched, thresholded frontier model, Jev's recall is about eight points lower and its precision roughly forty points higher.

Laya cannot read the paragraph under the shipped call. It reads 512 tokens per question, head included, which leaves 384 for the state. The shipped state opens with the three rule texts, 416 tokens before the first paragraph, so no paragraph is ever inside its window, at sixteen per call or one per call. Its 282 scores are the same either way, identical across three repeat runs, and in three shuffled orders every one of the 846 judgements equals the shipped-order score of whichever paragraph held that index. All of them fall between 0.30 and 0.59, none reaches 0.70, so it flags nothing: recall 0, no false alarms, 251 s of CPU for the 94 paragraphs. That row is the shipped call, run unchanged. The rows beneath it ask the same shipped question in a shape the window holds: the paragraph first, then the one rule it is judged against, one question per call. Jev on that shape keeps its recall and gives up seven points of precision, three false alarms in 256 against none. Laya on it, with the whole state inside its window on 269 of the 282 calls, flags 9 of the 26 violations, 2 of them under the rule that was broken, and raises 22 false alarms in 256, at 605 ms a call. Laya's 1024-token checkpoint holds the whole shipped one-paragraph state, 434 to 757 tokens, in every call, and flags all 94 paragraphs: recall 96.2%, precision 26.9%, 135 false alarms in 256, at 1.5 s a call. On the shipped batches of sixteen it reads 893 of each call's 1,024 to 2,427 tokens, about the first five paragraphs, and flags 78 of the 94: recall 38.5%, precision 12.8%. On the fitted shape it answers yes to 253 of the 256 negative judgements, median score 0.95: every paragraph flagged under every rule.

Routing: the incumbent puts half of everything in one bucket

90 messages typed at the assistant, labelled with the lane a reasonable co-DM should run.

The incumbent has 45 false positives on a single lane. When no keyword matches, everything lands in “talk”. And it has only four lanes, so a message that is not about the game at all — a billing question, an attempt to talk the assistant out of its own instructions — has no lane of its own. On the four-lane subset, which leaves those messages out, it scores 41.8% against Jev's 86.1%.

Contradictions: nothing was checking before

60 statements written against the real canon of eight factions, each scored against the eight facts already on file for that faction. Here there was no incumbent check: the database returned the adjacent facts and a human read them. The benchmark built a lexical baseline instead: token overlap plus a polarity cue, with its threshold tuned on this same set.

The 51.7% is the base rate of never ruling, which is the current behaviour. The colliding fact comes back first 86.2% of the time and in the top three 96.6%, against an expected 4.3 rows read in arbitrary order.

What it costs

Jev's published price is $42 per billion input tokens, $0.042 per million, with no charge for the answers it returns (typesafe.ai). co-dm's meter records Jev calls in tokens rather than dollars, so the dollar figures below price the LLM turns from the meter and price Jev's own calls from that published rate.

For routing, the incumbent is a local pure function at microsecond latency and zero marginal cost. The table below prices the turn that runs afterwards in the wrong lane.

In the cost model, Jev's misroutes cost $0.023 less per message routed than the incumbent regex's, against roughly $0.0005 per message to ask Jev at the published rate — about 46 times its own cost.

Speed: the whole guide in under a second

Every frontier model wrote its reasoning out before answering: 20 to 33 seconds per batch of sixteen for the fastest of them, 191 seconds for a single batch of the cheapest. Jev writes a number, and a batch of sixteen comes back in about three quarters of a second. Laya writes a number too and spends the time in its forward pass instead: 42 seconds per batch of sixteen on the CPU.

deepseek-v4.1-flash costs a twelfth of what gpt-5.6-sol costs and took more than twice as long for a third of the work.

The shipped code runs both: the regexes run first, cost 3.5 ms for all 94 paragraphs and catch five real violations, and the judge is asked only about what is left.

Where it loses

• It is not deterministic

  Eight runs over the same 94 paragraphs, shuffled, moved recall between 80.8% and 92.3% and precision between 92.3% and 100%. A single run can land on the best draw in that range.

• The frontier ceiling was never actually measured

  The arm meant to establish it — claude-opus-5 reading each item and reasoning — scored perfectly on both judgements and turned out to have seen the answer key's own notes and each item's provenance before judging. It has been discarded, not quoted. So the ceiling for a careful frontier model is unknown here; what is measured is how one behaves as a deployable classifier, which is 56.8% precision against Jev's 95.4%. Establishing the real ceiling needs an uncontaminated run that has not been done.

• The incumbent is partly being charged for a rule nobody wrote

  9 of the 26 violations test rule 1, which has no incumbent instrument at all, so those are automatic misses and are counted in the 19.2% headline. Restricted to the 85 items whose rules the regexes actually implement, the incumbent reaches 29.4% recall. The gap narrows and does not close.

• The answer key shares an operationalisation with the thing it grades

  The labeller wrote the key against the literal wording of each rule — which is also close to the wording Jev is given in the question. Where that reading of a rule is wrong, both are wrong together and the benchmark scores it correct. Fifteen negatives are marked contested for this reason, and two of the three paragraphs that ever produced a false positive are among them. The routing set has the same shape: its labels were written from the same lane descriptions Jev reads, so 87.8% is agreement with those labels. The DM who owns these rules has reviewed neither key.

• Two no-content shortcuts beat the incumbent

  On the table-rule set, “flag anything under 160 characters” scores 81.9% and “flag anything that was authored rather than taken from the guide” scores 87.2% — both above the incumbent's 77.7%, both reading nothing. The authored violations are shorter than the real paragraphs. On a length-matched slice Jev scores 97.9% and the trivial baseline scores 44.7%.

• The cheap models are slow, not broken

  A single-paragraph probe had deepseek-v4.1-flash spending 1,500 output tokens and emitting nothing. Re-measured over 32 items it answered all 32 cleanly. It took 284 seconds for 32 paragraphs, about 8.9 seconds each, which puts one session guide at a quarter of an hour. gemini-3.8-flash took 184 seconds and 28,845 output tokens, and two of its six replies were malformed JSON, so 32 of the 94 paragraphs got no answer.

  A model that answers in prose has to be parsed, and parsing can fail. Jev's answer is typed, so there is nothing to parse. Across every run here it returned 94 of 94.

• The answer key shares a blind spot with the thing it grades

  The labeller flagged it: 56 of the 68 clean paragraphs were written by this same assistant under these same three rules. Any rule it systematically misreads is baked into the key as correct. Fifteen labels were marked contested for exactly this reason, and the incumbent, Jev and the frontier arm all score 100% on that slice, because the slice cannot separate them.

• On the art library, the evidence currently argues against using it

  The 287-of-300 figure below is a sample measurement, not a scored benchmark, and its ground truth comes from file names — which the incumbent path rule never sees, so that comparison is not neutral and is not claimed as one. A separate 177-row set was scored, and what it found is unflattering: a three-line rule that reads only the file extension beats the shipped path rule, and once the set is reweighted to the library's real composition — about 83% props — simply answering “other” to everything scores 0.83 against Jev's 0.77. On that judgement, at that prevalence, the model does not beat the baseline. It stays in proposals-only. On the same 177 rows, unweighted, the shipped path rule scores 57.6%, Jev 82.5%, the frontier model 96.6%, and always answering “other” 45.8%. Laya, which reads each 116-token state whole, scores 37.3%: it calls 95 of the 177 rows a token, where 24 are, at 589 ms a row. Its 1024-token checkpoint scores 17.0% and calls 162 of the 177 a token.

• Fact triage is not benchmarked at all

  A set exists; the run is unfinished. No result is reported.

A sorting bug the classifier exposed

The art-library judgement (row four below) was sampled before it was trusted. The library holds 181,805 files sorted by the folder they sit in, 151,132 of them filed as tokens, the cut-out figures that stand on a battlemap. Three hundred were put to the model. Eight more, pulled by name from the same pile:

287 of the 300 were not tokens, at 0.91 average confidence. The paths behind the answers showed why: the library's root folder is named Assets and Tokens, and the sorting rule handed that word down to every bracket and cushion inside it. Maps had a guard against this; tokens did not.

The fix was one line in the sorting rule and corrected 59,345 files. The pattern appeared in the classifier's sample of 300.

What is actually switched on

Without the API key, all five switch off and the assistant runs as it did before they were added. In every judgement the incumbent code still runs first, and the model only sees what it passes on.