Skill selection, measured

The trap I walked into first

This is the most useful part of the exercise, so it goes before the results rather than in a footnote.

To score 8,916 skill–posting pairs you need labels, and nobody was going to hand-label them. My first idea was the obvious one: a skill is relevant to a posting if the posting names it verbatim. Cheap, objective, reproducible.

It is also exactly what the lexical arm computes. I had defined the ground truth to be one contender's output. Here is what that produced:

Read carelessly that says a substring matcher beats every model, and Jev is second. Read correctly it says nothing at all: the top row is the label definition scoring itself, and every other row is being graded on how well it imitates str.find().

What stays a judgement, stated so it can be argued with: I assigned the domains for each posting, from its title and stated responsibilities. That mapping is published in full in eval_clean_labels.py with the phrase from each posting that drove each line, so you can disagree with a specific call rather than with the conclusion.

The scoreboard

Labels no arm can game. 412 skills × 12 postings; between 36 and 239 positives per posting.

Two things to take from this, and the second matters more than the first.

GLM beats Jev, and the margin survives testing

Ahead on 11 of 12 postings, mean difference +0.067, sign test two-sided p = 0.0063. Bootstrapped over 306,850 ranked pairs the pooled difference is +0.0597, 95% CI [+0.0581, +0.0612] — comfortably excluding zero. Haiku, by contrast, is indistinguishable from Jev (5 of 12, p = 0.77).

Everything is closer to a coin flip than anyone would like

The best arm scores 0.68 where 0.50 is guessing. Some of that is the label being coarse — a Security posting does not want every security skill, so perfect play does not score 1.0 here and the real ceiling is unknown. But it is a caution against the entire exercise, and it is why the one comparison people will want — Jev against a free substring matcher — is not statistically separated at this sample size (3 of 12 to lexical, p = 0.146). Jev is ahead. I cannot yet show that it is reliably ahead.

Speed and reliability, where Jev is not close

Every arm ran sequentially, in batches of 60, on the same machine and the same postings.

The 572 is the number worth staring at. That is 6.4% of Haiku's answers that never arrived — lines that did not match the expected format, in a run where the format was one number per line at temperature 0. Nothing raised an error. Several postings came back with 518 or 636 of 743 skills scored and the pipeline carried on, and the only reason I know is that the harness counts them. A typed response has no equivalent failure: the number is a number or the call failed loudly.

And the trap on the other side

GLM nearly produced a false headline of my own making. It is a reasoning model, and at the token budget that is ample for 60 answer lines it spent every token thinking and returned empty content — 0 of 60 parsed on every batch, which reads exactly like a model that cannot follow a format. Measured, per batch of 60:

  budget   784, thinking on      0/60    6.4s     784 out
  budget  4000, thinking on     60/60   28.5s   3,895 out   $0.00129
  budget 12000, effort low      60/60    4.7s     361 out   $0.00023

Left on default, the model billed as one-eleventh the price of Haiku costs 5.6× and takes 6× as long as itself. Per-token price lists are misleading for reasoning models unless the effort is pinned. (enable_thinking: false was tried; the provider ignored it. reasoning_effort is what works.) Every GLM number above uses the pinned configuration.

The cost claim does not survive

I expected this to be the easy win. It is the weakest part of the case.

TypeSafe publishes no pricing I could find, so Jev's cost cannot be computed — only the price at which it would break even. Its token usage is recorded, and it is high: 941,194 tokens against GLM's 319,672 for identical work. Each noul carries its own instructions and criteria text, sixty per call, where the generative prompt lists sixty bare skill names once.

Beating Haiku is plausible. Beating GLM requires Jev to cost under five cents per million tokens blended, which would make it one of the cheapest endpoints in existence. On the evidence I have, "cheaper" is not a claim I can make — and if someone at TypeSafe wants to publish a number, this is the arithmetic it lands in.

The part no metric could grade

Every model arm spends most of its ranking on skills the posting never names:

  jev      286/480   60%
  glm      289/480   60%
  haiku    297/480   62%

  lexical cannot rank a single one of them.

That 60% is the entire reason to pay for a model instead of running a substring match, and the domain metric cannot grade it — domain labels are too coarse to adjudicate an individual skill, and there is no automatic ground truth for “this posting implies Terraform without saying it.”

So I graded them. The harness writes the cases where the arms most disagree — one confident yes, another confident no, on a skill the posting never names — to eval-disagreements.json: 240 rows, twenty per posting. Each was then judged twice, independently, one pair at a time with the full posting in view and the arms' scores withheld.

And the ranking inverts

The 215 rows both passes agreed on — 103 yes, 112 no. Answering “yes” to everything scores 48%.

GLM's earlier win is explained by that last column. It says yes to 13.5% of these pairs and consequently rejects 94.6% of the true noes — excellent specificity, and it catches barely a fifth of the skills a posting genuinely calls for. For a metric built on hard negatives that caution reads as skill. For actually selecting what goes on a résumé, a recall of 22.3% is not usable. Jev is the only arm anywhere near balanced.

Haiku ranks below chance at 0.384 — worse than shuffling. It is also the arm that shares a lineage with its judge, which should have flattered it. It did not.

Does it survive dropping the hard cases?

Restricting to the 95 rows neither pass flagged arguable, the gap widens rather than closing, which is the opposite of what an artefact does:

Reproducing it

Four scripts, all in the repository that builds this site's CV data. Every figure on this page comes out of the results file, not off a terminal.

Two caveats I would raise if I were reviewing this rather than writing it. Twelve postings is a small sample for per-posting tests, which is why the sign test is reported alongside a pooled bootstrap rather than instead of it. And the postings are all from one candidate's search in one month, skewed toward pre-sales and MSP work — the domain mix is not representative of hiring generally, only of what I was actually applying to.