Here is the quiet load-bearing assumption of 2026: with models shipping faster than humans can evaluate them - one public testing effort ran 25 unique models in Q3 2025, 83 in Q4, 44 more by mid-March - the industry has standardised on using AI to grade AI. LLM-as-judge is now how leaderboards rank, how agents get certified for production, how “safe” gets stamped. So the most useful experiment this desk read all quarter is the one that handed a single agent’s work trace to six frontier models with identical grading criteria - and published the reasoning, not just the scores.
The result: consensus on the number, chaos underneath
The scores looked like agreement - all six landed between 90 and 100 on the IT-service-desk trace. The reasoning did not survive contact with itself. One judge docked points for an undocumented verbal approval; another flagged prerequisite-verification sequencing; a third graded tool-call correctness; a fourth wanted ticket metadata. Each model pulled different evidence, built a different theory of failure, and applied a different working definition of “good.” The detail this desk keeps returning to: the judges that investigated least scored highest - the shallowest reviewer, eight steps of checking, awarded the perfect 100. Diligence and generosity were inversely correlated, which is exactly the failure mode you would design if you wanted evaluations that feel rigorous and flatter everyone.
A score is a compression of a judgment. Six judges compressing six different judgments into the same number is not agreement - it is collision.
Why this lands now
Because the volume makes human grading structurally impossible and the stakes make bad grading structurally expensive. Seven frontier models shipped from the three leading labs in one 78-day stretch this spring; benchmark suites now rebuild themselves quarterly against contamination; enterprises are wiring agents into workflows where the “evaluation” is the only gate between a model update and production. Every one of those pipelines has a judge in it. The experiment’s implication is not that AI judges are useless - it is that a judge’s score without its reasoning trace is an unverifiable claim, and this publication has a standing policy on those.
THE DESK’S MINIMUM STANDARD FOR AI-GRADED RESULTS
Treat a judged score as evidence only when four things ship with it: the judge model and version, the rubric verbatim, the full reasoning traces, and inter-judge agreement measured on the reasoning - not the number. A 95 with hidden reasoning is a vibe with decimals.
What would change our mind
Published inter-judge studies where the failure theories converge, not just the scores; judging panels with disclosed disagreement rates, treated like error bars; and evaluation providers versioning their judges the way serious labs version their models. Some of this is emerging - the better public trackers now lock model versions and publish per-run variance. Until it is normal, the reading rule stands: when AI grades AI, audit the grader first.
THE DESK’S BOTTOM LINE
The judge problem is not a corner case; at current release velocity it is the evaluation system. Identical scores concealed six incompatible theories of quality, and the laziest judge was the kindest. Demand the reasoning or discount the number.
