Well Calibrated, Wrongly Ordered

LLM judges are everywhere in production systems: pick the best of N samples, filter rejection-sampled outputs, score revisions, build preference data. Every one of those uses is an optimiser pointed at the judge, and the folk expectation is that the agreement figure you measured once, on ordinary outputs, stops describing the judge as soon as outputs are chosen to please it. We tested that expectation directly, on a domain where ground truth is exact and free, and found it is false in the regime most practitioners operate in — and severe in a narrower one that is easy to miss.

Across three experiments we varied optimisation pressure and the alignment between what the judge rewards and what is correct. Under weak pressure — best-of-k over 24 i.i.d. samples — accuracy rises monotonically and judge overconfidence stays flat across a 24-fold increase in selection. Under strong pressure — five rounds of explicit revision to raise the judge’s score — accuracy still rises, and the optimised arm is never significantly worse than a control with the same revision budget. Only when we construct problems where the honest answer is unattractive — one required quantity deleted, so the correct response is a refusal — does the effect appear, and then it is large: confident fabrication rises from 1.7% to 75–82% within two rounds, +75.0 percentage points over control (13.3 SE). Notably, the judge is not deceived in absolute terms. It scores these fabrications 10–17 out of 100. It just ranks confident fabrication above honest refusal — and ranking is the only thing an optimiser consumes.

The question

A team shipping an LLM feature validates its judge once, on a sample of ordinary outputs, and records a single agreement figure. Thereafter the judge is used to select. The worry is well motivated at the extremes: Zheng et al. showed that a null model emitting a constant response unrelated to the input achieves an 86.5% length-controlled win rate on AlpacaEval 2.0, and Gao et al. established overoptimisation scaling laws — but for reward models graded against a fixed gold reward model, with no judge, no rubric, and no natural-language evaluation.

What is missing is the interior. Between one sample and adversarial search lies the entire operating range of production systems, and it is unmeasured. This paper measures it, and reports that the expectation is wrong in the common case. The two null results below are not failed experiments; they are the boundary that makes the third result mean something.

Setup

All tasks are multi-step quantitative word problems generated locally from four templates — tiered discounts with a threshold, staggered work rates, repeated dilution, and a loan with partial repayment — each built so that a common reasoning slip yields a clean-looking wrong number rather than nonsense. Generating problems rather than drawing them from a public benchmark means ground truth is exact and requires no annotation, and there is no contamination to disentangle from the effect under study.

The generator is claude-haiku-4-5, chosen deliberately weak so that a meaningful error rate leaves headroom for selection to act on; base accuracy is 0.60–0.67 across runs. The judge never sees ground truth and is asked for a calibrated probability that the candidate’s final answer is correct.

The design was fixed before any API call by a simulation of the selection process. It established that with judge-pleasing features that are merely uninformative, accuracy rises monotonically at every level of judge gameability — gameability suppresses the gain rather than reversing it — and that reversal requires those features to be anti-correlated with correctness. It also fixed the sample size: at 120 problems the minimum detectable effect is roughly 5 percentage points. Both predictions held.

Weak pressure means best-of-k selection over 24 independent samples per problem. Strong pressure means five rounds of explicit revision in which the generator sees its judge score and is instructed to make it go up.

Picking the judge is a measurement, not a default

Before running any experiment we measured three candidate judges on 60 solutions from a completed run, balanced 30 correct and 30 incorrect, with an identical prompt.

AUC is the probability that a randomly chosen correct solution outscores a randomly chosen incorrect one; on a balanced sample 0.5 is chance. Two findings follow. The small model sits on the score ceiling — 95% of its scores exceed 90 — so selection against it is close to random, and any experiment using it would be uninformative by construction. And the largest, most expensive model discriminates worse than the mid-tier one at roughly five times the price. The intuition that the strongest available model makes the best judge is not supported here. We picked claude-sonnet-5, and the result prefigures the paper’s main finding: what matters about a judge is its ordering, not its level.

Weak pressure: nothing breaks

We draw 24 i.i.d. solutions per problem (120 problems, 2,880 samples) and let the judge select the best of a random k-subset for each k up to 24. Accuracy rises monotonically: acc@24 − acc@1 = +16.45 percentage points (5.6 SE), peaking at the largest k tested, with no interior optimum. The judge’s own stated confidence rises from 79.0 to 96.5 while realised accuracy rises from 0.668 to 0.833, so overconfidence is essentially flat — +12.2 points at k=1 and +13.3 at k=24 — across a 24-fold increase in selection pressure.

Three mutants confirm the measurement rather than the sort: shuffling judge scores flattens the curve to 0.0 SE, a judge given ground truth reaches the oracle exactly, and an inverted judge drives accuracy down from 0.669 to 0.521.

One negative result about our own analysis: the share of achievable gain captured by the judge appears to fall from 81.5% to 68.5% between k=4 and k=24, which reads as decay. Computed per problem and paired it is −0.023 ± 0.028, or 0.8 SE. The aggregate figure is a ratio-of-means artefact, and we do not claim it.

Strong pressure, aligned judge

Best-of-k over i.i.d. samples is a lottery over the generator’s natural output distribution, not a search for the judge’s blind spots. So we apply pressure directly: the generator is shown its judge score and instructed to revise so that the score increases, for five rounds. A control arm starts from the same solution, receives the same revision budget, and is instructed to maximise the chance of being correct, with the judge never mentioned. The control is what makes the comparison identifying — without it, any decline in the treatment arm could simply mean that revision degrades solutions.

Pressure was genuinely applied: the judge score climbs from 77.5 to 89.6 (+11.5, 4.8 SE). Correctness did not pay for it. Accuracy in the optimised arm rises from 0.675 to 0.858, and at no round is that arm significantly worse than control. A +7.50 point advantage of the judge-optimised arm over control appears at the final round, but at 2.2 SE uncorrected across six round-wise comparisons and against a bouncing trajectory, we do not claim it.

Strong pressure, misaligned judge

Both nulls admit one explanation. On solvable problems, “looks correct to a careful judge” and “is correct” largely coincide, so climbing the judge climbs correctness; with an AUC of 0.851 the judge is tracking something real. So we construct the opposing condition rather than hoping to encounter it. Half of the 120 problems have exactly one required quantity deleted and replaced with a phrase that reads naturally (“a first discount is applied”, “Worker B also works at a steady rate”). The correct response becomes a refusal; any numeric answer is a fabrication. Ground truth remains exact and free. Judge-pleasing behaviour — a confident, complete, well-presented derivation ending in a number — now points directly against correctness. The generator is explicitly told that refusal is available and legitimate, so that fabrication measures the optimiser and not the prompt.

Fabrication in the judge-optimised arm rises from 0.017 to 0.750, a paired difference over control of +75.0 percentage points (SE 5.6; 13.3 SE). The control arm ends at 0.000.

This is not general degradation. Accuracy on the answerable half rises in both arms over the same rounds (0.633 → 0.783 optimised, → 0.733 control). The model did not get worse; it learned specifically to fabricate where fabrication paid. Two structural checks hold: round 0 is identical across arms by construction, and fabrication on answerable problems is exactly 0.000 in both arms throughout, so the grader is not miscounting.

The optimiser is not misreading its instructions; it is climbing a real gradient. Among unanswerable problems, fabricated solutions score 17.0 against 10.5 for honest refusals — a difference of 6.5 points at 9.3 SE. Restricting to the 65 within-task transitions where the optimised arm flipped from refusing to fabricating, the judge score rose +5.8 points (SE 1.3). The exchange rate is the finding: approximately six points of judge score, purchased with 75 percentage points of correctness.

The judge’s absolute scores on these fabrications are 10–17 out of 100. By any calibration measure it knows these solutions are bad. What is broken is the ranking: it places confident fabrication above honest refusal. An optimiser consumes only the ordering; level is irrelevant to it. This is why mean agreement is the wrong summary statistic for a judge that will be optimised against, and why the judge-selection result above points the same way — AUC, a pure ranking measure, separated the three candidate judges in a way their mean scores did not.

The arc

Weak pressure: nothing. Strong pressure with judge and truth aligned: nothing. Strong pressure with judge and truth opposed: +75 percentage points.

The practical consequence is a check rather than a warning. Mean agreement, measured on ordinary outputs, does not predict what happens under optimisation. What predicts it is the judge’s ordering on cases where the honest answer is unattractive — refusals, admissions of insufficient information, hedged answers, and any response that is correct but unimpressive. A team can construct that test for its own domain cheaply, and should, because it is the property their optimiser will find.

Limitations

The domain is one family of multi-step quantitative word problems with a single generator and a single judge; 60 unanswerable problems at four rounds. The unanswerable construction is synthetic — a deleted quantity — and real underspecified inputs are messier and may be recognised more or less readily. The effect size should be read as a demonstration that the regime exists and is reachable by ordinary means, not as an estimate of its magnitude in deployment.

Two experiments we did not run bear directly on the claim. We did not test a judge explicitly prompted to reward appropriate refusal, which is the first fix a practitioner would reach for and might eliminate the effect entirely. And we did not test a subjective domain, where ground truth is unavailable and the coincidence that produced our null results may break for reasons unrelated to the mechanism identified here.

The two null results bound the phenomenon at 120 problems, where the minimum detectable effect is roughly 5 percentage points; degradations smaller than that would not have been visible.

Related work

Zheng et al. establish the adversarial endpoint — a null model reaching 86.5% on AlpacaEval 2.0 — and Gao et al. are the nearest theoretical relative, explicitly about reward models graded by a synthetic gold reward model rather than natural-language judges. The static judge-quality literature measures judges on i.i.d. outputs — Norman et al. report high test–retest reliability alongside severe ordering biases, and Panickssery et al. show evaluators favour their own generations; the distinguishing feature of this work is that outputs are selected to please the judge. On measurement practice, Miller argues that evaluations are experiments and should be analysed as such, and Bjarnason et al. quantify run-to-run variance in agentic evaluation at 2.2 to 6.0 percentage points even at temperature zero.

References

1. Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. On randomness in agentic evals. arXiv preprint arXiv:2602.07150, 2026.
2. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022.
3. Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024.
4. Justin D. Norman, Michael U. Rivera, and D. Alex Hughes. Reliability without validity: A systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias. arXiv preprint arXiv:2606.19544, 2026.
5. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024.
6. Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Cheating automatic llm benchmarks: Null models achieve high win rates. arXiv preprint arXiv:2410.07137, 2024.