How good is the lead scorer, actually?

Two runs, unchanged code

Anyone quoting “86%” as the accuracy is quoting a number that would have been 82% a minute later. Reporting one run would have been the easiest and least honest thing to do.

Every error is one tier, and every error is upward

Nothing labelled hot came back cool. Nothing labelled cool came back hot. It over-promotes and never dismisses — which, for deciding who gets a call back, is the direction you would pick if you had to pick one.

The number that decides whether it is usable

Hot recall is reported on its own, because a missed hot lead is a lost instruction and a misfiled cool one is a wasted minute. Those are not the same mistake, and an overall accuracy figure hides the difference. It came out at 88.0% in both runs — the one figure that did not move — and it missed the same three enquiries each time.

Which is what makes it worth reading rather than shrugging at. All three misses are owner-side: someone who owns a named property and is weighing selling it, letting it, or asking what it costs to run. The rubric counts a named asset as a real lead; the scorer waits for a stated commitment before it will. So it systematically under-rates the listing side of the book — a bias with a direction, not noise, and a prompt change rather than a model change.

Individual verdicts are not reproducible

Eleven enquiries were scored three times inside each run. Seven of eleven came back identical in the baseline; nine of eleven in the second. Four wobbled inside a single run.

That is the finding with a product consequence attached: any feature treating one score as a fact — an alert, an auto-assignment, a number on a dashboard — is building on something that would answer differently if asked again. Which is why the tier is shown as a suggestion with its reasons beside it, and never changes an enquiry's status on its own.

A gate that can fail

A regression check compares each run against a committed baseline and exits non-zero if the scorer gets materially worse. Its tolerances come from the measured noise rather than from taste: two runs of unchanged code differed by 4.1 points, so a tighter gate on exact agreement would fail on nothing. Hot recall keeps a tighter tolerance because it earned one.

The honest consequence, stated on the page rather than buried: at this set size the gate catches large regressions only. And it was checked by degrading a copy of a results file until it failed — so it is not a light that can only ever be green.

Where the labels came from

This is the part that decides whether any of the above means anything. The labels were not produced by the model under test — a set labelled by the thing being measured only proves it agrees with itself. They were written against a rubric committed before the labels, by a different model from the one being scored, with twelve cases recorded as arguable and reported separately.

It is still not a human-adjudicated set, and the write-up says so. Seventy-three cases support no claim of significance, the class balance is deliberate rather than sampled from real traffic, and results drift as the model and prompt move — which is why every run records its model, commit and date.

The run

Labelled enquiries

7325 hot · 24 warm · 24 cool

Model scored

gpt-5.6-lunavia the live scorer

Calls per run

9573 cases + 22 repeats

Median latency

1.8smax 7.0s