Skill selection, measured

What is actually being measured

Before the scoreboard, the thing under test. One call carries the posting once as state and sixty independent questions about it; the answers come back typed, so there is nothing to parse and nothing to retry.

{
  "state": { "job_posting": "…" },   // 6.3 KB, sent once
  "model": "jev-latest",
  "questions": {
    "q0": {
      "type": "noul",
      "instructions": "Does this posting call for
        \"RMM\", either by name or by describing
        work that requires it?",
      "criteria": {
        "true":  "The posting asks for it…",
        "false": "The posting does not need it"
      }
    },
    "q1": { … 59 more … }
  }
}

{
  "answers": {
    "q0": { "noul": 0.99 },
    "q1": { "noul": 0.07 },
    …
  }
}

// No "return JSON only, no prose, no
// code fence". No regex over a reply.
// No retry when the fence comes back // wrong.
//
// The other inference scripts in this // repo do all three.
// This work moved onto Jev to
// avoid all three.

Per posting, averaged over the twelve in the benchmark below — 103.9s and 941,194 tokens for all of them

Every alternative below has to either produce the same sixty numbers from a text reply, or find them some other way. The alternatives: GLM-5.3-flash and Haiku 4.5 answering the same sixty questions as text, two embedding arms, a substring matcher, and ConvAI's Laya, an open-weight typed-decision model (base checkpoint, 421M parameters, Apache 2.0) whose model card compares it with Jev on figures it states were "third-party published, never measured here". Laya ran on a 12-thread CPU with no GPU, with the same noul question and criteria as the Jev arm, the same batches of 60 and the same posting as the only state; its package truncates the state to the checkpoint's 512 tokens, which is how it ships.

Two arms run Qwen 3.8 (qwen/qwen3.8-flash, $0.15 in and $0.47 out per million tokens) through OpenRouter. The first prompts it exactly as GLM and Haiku are prompted, with reasoning_effort: "none". The second reads it as a System 1 sampler: one call per skill with the posting first and the noul question last, max_tokens: 1, and the score is P(yes) / (P(yes) + P(no)) from the next-token logprobs. Nothing is generated or parsed; the answer is the prefill. It is the same mechanism as Jev's without a trained decision head, on the Qwen 3.8 weights, and it answers one question per call where Jev answers sixty. Sixteen calls ran in flight; every other arm ran sequentially.

The rerun, on equal terms

The first run, further down, gave the arms unequal treatment in four ways that could move a result. The rerun removes each one and runs every arm again.

Random AUC is ranking quality on the 480 random pairs, which no arm chose: the unbiased figure. Hard AUC is the same on the 240 hardest pairs. Decision is balanced accuracy at a yes/no cutoff tuned on six postings' random pairs and scored on the other six, then swapped, the identical procedure for every arm. Domain AUC is the first run's domain-label metric. vs Jev is a paired bootstrap of the AUC difference; grey intervals include zero. $ per 1,000 is OpenRouter's billed figure per call, and TypeSafe's published $42 per billion input tokens for Jev.

What the rerun shows

One skill per call helps the chat models; it does not help Jev. Asked one skill at a time, Haiku 4.5 rises from 0.885 to 0.948 on the random pairs and Qwen 3.8 from 0.901 to 0.960. Jev scores 0.995 and 0.994 either way, at 18× the cost per decision for one per call. Batching sixty questions costs the chat models accuracy and costs Jev nothing.

The random pairs are easy and the hard pairs are not. 62 of the 480 random pairs are yeses, and the best arms sit near the ceiling there; the hard pairs are where arms separate. On them, neither the Qwen 3.8 readout nor GLM asked one skill per call separates from Jev.

Lost answers are real but small. Haiku omits 177 of 8,916 answers asked 60 per call and 187 with reasoning on; Qwen 3.8 omits 59 and 169. Jev, GLM, Laya and every one-per-call arm lose none.

The same postings, as images

Jev takes text only. Every posting was rendered to page images, as a screenshotted or photographed job ad would arrive, and each arm read those instead.

Each posting is wrapped at 100 characters and split into pages of 52 lines; headless Edge screenshots each page at 1000 × 1400, 34 pages for the twelve postings. GLM-5.3-flash, Haiku 4.5 and Qwen 3.8 read the page images directly. Jev reads the text that the OCR engine built into Windows recovers from the same pages, as a pipeline in front of Jev would; that OCR misreads some lines (“USA” became “CIS”) and drops Markdown labels.

OCR costs Jev nothing measurable on these pages: 0.993 against 0.995, and the OCR step runs locally. Every vision model ranks lower reading images than reading text, and the loss is largest asked sixty skills per call: Haiku falls 0.126. The Qwen 3.8 readout holds best of the vision arms, at 0.958 and $0.5911 per 1,000 decisions, 13× its text cost, because the page images are not cached between calls.

The first run

Everything below is the first run as published on 17 September, with the Qwen 3.8 arms added on 2 October. Its figures stand as measured. Two of its findings did not hold up in the rerun, and each is corrected where it appears.

The first label definition was circular

It affected every result below, so it comes first.

Scoring the 8,916 skill–posting pairs required labels; none were hand-labeled. The first definition: a skill is relevant to a posting if the posting names it verbatim.

It is also exactly what the lexical arm computes. The ground truth had been defined as one contender's output. Here is what that produced:

The top row is the label definition scoring itself; every other row is scored against the output of a substring match (str.find()).

What stays a judgement: the domains for each posting were assigned by hand, from its title and stated responsibilities. That mapping is published in full in eval_clean_labels.py with the phrase from each posting that drove each line.

The scoreboard

Labels no arm can match by string. 421 skills × 12 postings; between 37 and 248 positives per posting. Every arm is scored against the catalogue's current 421 master-labelled skills; the first run had 412, which moves the arms by about a hundredth.

Two results.

GLM beats Jev, and the margin survives testing

Ahead on 11 of 12 postings, mean difference +0.067, sign test two-sided p = 0.0063. Bootstrapped over 306,850 ranked pairs the pooled difference is +0.0597, 95% CI [+0.0581, +0.0612] — excluding zero. Haiku, by contrast, is indistinguishable from Jev (5 of 12, p = 0.77). Laya trails Jev on 11 of 12, mean difference −0.109, p = 0.0063. Neither Qwen arm separates from Jev on this metric: the logprob readout leads on 8 of 12 (+0.017, p = 0.39) and the prompted arm on 5 of 12 (+0.001, p = 0.77).

Every arm scores near chance on this metric

The best arm scores 0.68 where 0.50 is guessing. The labels are set per domain: every skill in a posting's domain is labelled relevant, so on a Security posting every security skill is a positive. A ranking that puts only the skills the posting asks for at the top does not score 1.0 against these labels; the ceiling was not measured. Jev and the free substring matcher are not statistically separated at this sample size (3 of 12 to lexical, p = 0.146). Jev leads on the pooled figure; the per-posting lead is not established.

Speed and reliability

Every arm ran on the same machine and the same postings. All but the Qwen 3.8 logprob readout ran sequentially in batches of 60; the readout answers one question per call and ran sixteen calls in flight.

572 answers were not parsed. That is 6.4% of Haiku's answers — lines that did not match the expected format, in a run where the format was one number per line at temperature 0. Nothing raised an error. Several postings came back with 518 or 636 of 743 skills scored and the pipeline carried on, the harness counted the missing lines. Jev and Laya return typed values with no text to parse, and each lost 0 answers.

Corrected in the rerun: most of these lines were answers in a format the parser did not read. Haiku writes some lines as 1. task dispatch:0.00 and some as 0.85:29. With the same settings the rerun parser dropped 597 lines reading only 1:0.00; reading both formats recovers 420 of them. 177, 2.0%, are missing from the replies.

GLM with default reasoning

GLM is a reasoning model, and at a 784-token budget, sized for 60 answer lines, it spent every token thinking and returned empty content — 0 of 60 parsed on every batch. Measured, per batch of 60:

  budget   784, thinking on      0/60    6.4s     784 out
  budget  4000, thinking on     60/60   28.5s   3,895 out   $0.00129
  budget 12000, effort low      60/60    4.7s     361 out   $0.00023

Left on default, the model billed as one-eleventh the price of Haiku costs 5.6× and takes 6× as long as itself. For GLM, the reasoning setting changed the output-token count per batch, and with it the cost per batch. (enable_thinking: false was tried; the provider ignored it. reasoning_effort is what works.) Every GLM number above uses the pinned configuration.

Qwen 3.8-flash does the same. At the 784-token budget it parsed 0 of 60 on a test batch; at 12,000 tokens with reasoning_effort: "low" it parsed 60 of 60 in 54.1s; with "none", 60 of 60 in 5.3s. Both Qwen arms use "none".

Cost

Jev uses nearly three times the tokens of GLM or Haiku and costs less than either.

Jev used 941,194 tokens against GLM's 319,672 for the same 8,916 decisions. Each noul carries its own instructions and criteria text, sixty per call, where the generative prompt lists sixty bare skill names once. The per-token price is $42 per billion input tokens, with output not priced separately.

One assumption: TypeSafe prices input and says nothing about output, so the $0.0328 counts input only. Charge Jev's 159,552 output tokens at the same rate and it becomes $0.0395 — still cheaper than both, 13.9× under Haiku.

The logprob readout sends the posting once per skill, so it reads 14× the input tokens of Jev. Every call for a posting shares the posting as a prefix, and OpenRouter served part of each repeat from cache: on a test pair, 768 of 941 prompt tokens came from cache and the second call cost 3.6× less than the first. The $1.6897 is list price with no cache discount. Rerun on one posting with billing recorded, the readout was billed $0.0290 against $0.1053 at list, with 81% of its input tokens served from cache: 0.275 of list. The rerun recorded billing on every call: the full readout was billed $0.3913, 12× Jev.

Skills the posting never names

Every model arm spends most of its ranking on skills the posting never names:

  jev      286/480   60%
  glm      289/480   60%
  haiku    297/480   62%
  qwen38   287/480   60%
  qwen38-sys1 283/480   59%
  laya    439/480   91%   (and 1.8% of its noes correct on the adjudicated rows)

  lexical cannot rank a single one of them.

Lexical cannot rank any of those picks, and the domain metric cannot grade them — domain labels are too coarse to adjudicate an individual skill, and there is no automatic ground truth for “this posting implies Terraform without saying it.”

They were graded by hand. The harness writes the cases where the arms most disagree — one confident yes, another confident no, on a skill the posting never names — to eval-disagreements.json: 240 rows, twenty per posting. Each was then judged twice, independently, one pair at a time with the full posting in view and the arms' scores withheld.

The ranking inverts

The 215 rows both passes agreed on — 103 yes, 112 no. Answering “yes” to everything scores 48%.

The Qwen 3.8 logprob readout ranks best and decides worse than Jev at a 0.5 cutoff. Its AUC is 0.853 against Jev's 0.793; a paired bootstrap over the 215 rows puts the difference at +0.060, 95% CI [+0.022, +0.101]. It says yes to 20.0% of pairs, so at 0.5 it rejects 95.5% of the noes and catches 36.9% of the yeses, for 67.4% accuracy against Jev's 73.0%. The ranking is better; the cutoff that turns it into a decision was not tuned.

Corrected in the rerun: this gap came from the rows. These 240 were selected where Jev, GLM and Haiku disagreed most, so one of the three was confidently wrong on each, and the Qwen 3.8 arms, added afterwards, were not part of the selection. On 240 rows selected from every arm's disagreement, the readout leads Jev by +0.028, 95% CI [−0.010, +0.070], which is not separated; on 480 random rows Jev leads it by 0.013, which is.

GLM and prompted Qwen 3.8 say yes least often. GLM says yes to 13.5% of these pairs and rejects 94.6% of the true noes, catching 22.3% of the skills a posting calls for; prompted Qwen 3.8 says yes to 16.3% and catches 30.1%. Jev is the only arm near balanced.

Haiku ranks below chance at 0.384 (chance is 0.5). Haiku and the adjudicators are both Claude-family models; Haiku has the lowest AUC of the six arms graded here.

Does it survive dropping the hard cases?

Restricting to the 95 rows neither pass flagged arguable, the gap widens rather than closing:

Six findings

Each has a harness in the repo that produced it. Three changed the pipeline's code. One withdrew a recommendation that had already gone out.

What it produces

One posting in, a ranked sheet out. wants is Jev; holds is the catalogue; the ranking multiplies them in code.

scored 743 skills in 9.3s (13 ms each)

  match  wants holds  skill                       alt  era
  0.89   0.98  0.85   demo environments           DTS  —
  0.84   0.92  0.85   PowerShell                  DT   —
  0.81   0.99  0.70   networking                  —    2005 to present
  0.81   0.99  0.70   pre-sales                   —    2014 to 2025
  0.78   0.95  0.70   documentation               —    2018 to present
  0.72   0.99  0.55   technical demonstrations    DTS  2009 to 2025

  wanted, and thinly held — know these before the interview:
    wants 0.99  holds 0.25  RMM
    wants 0.97  holds 0.25  RMM agent deployment
    wants 0.99  holds 0.10  endpoint management
    wants 0.98  holds 0.10  RMM and endpoint management at scale

The same run produces the second list.

The catalogue those holds come from is built the same way — a per-skill noul against each of 25 domains decides membership, and those memberships supply the graph's cross-domain edges. Before that pass ran, three domains connected to nothing at all.

Reproducing it

Four scripts, all in the repository that builds this site's CV data. Every figure on this page comes out of the results file, not off a terminal.

The 720 labelled pairs, each with the request Jev was sent, the three graders' votes and every arm's score, are published as jev-eval-dataset-2026-10-03.zip (198 KB, JSON Lines with a README), with the co-DM sets.

Two caveats. Twelve postings is a small sample for per-posting tests, which is why the sign test is reported alongside a pooled bootstrap rather than instead of it. And the postings are all from one candidate's search in one month, skewed toward pre-sales and MSP work — the domain mix reflects the roles that candidate applied to, not hiring generally.