How we know the AI is right: dueling judges

The first time our eval suite waved a new model through, the question that stopped us wasn't about the model. It was about the grader. Our "eval" was an LLM judge scoring another LLM's output - so a passing grade proved one of two very different things: the output was good, or the judge was easy. We had no way to know which.

That's the quiet crisis in every LLM-judge setup, and most teams resolve it by not thinking about it. We resolved it by making the judges fight.

The judge, and the judge's judge

The primary judge is deliberately cheap and fast: it grades outputs on simple 0/1/2 rubrics - wrong, partially right, right - across a labeled evaluation pool. Resumable, parallel, run all the time. On its own, exactly as trustworthy as any single model grading another: unknown.

So a second judge audits the first. It's a different, stronger model, it re-grades a sample of the same outputs against the same rubric, and it shares nothing with the primary - no sight of the first judge's scores, no shared conversation, nothing to anchor on. Then we compute Cohen's kappa - the boring, fifty-year-old agreement statistic - between the two.

Kappa turns "do we trust the grader?" into a number with teeth. High agreement means the rubric is tight enough that two different models read it the same way - which is the strongest signal available that a model or a human would too. Low agreement means the rubric is ambiguous, and this is the part that changed how we work: when the judges disagree, we fix the rubric, not the model. Almost every disagreement traces to an underspecified grading criterion, and every rubric tightened this way makes all future grades cheaper to trust.

Floors, baselines, and the flip gate

Around the judges sits the machinery that makes the grades operational:

1. Locked metric floors per service. Not vibes - numbers with names: anchor facts at 100% accuracy (an anchor fact wrong is a customer-visible lie), structured slot extraction at ≥80% exact match, narrative claims graded on the 0/1/2 grounding scale. A change that dips any floor doesn't ship.
2. Versioned baselines. The eval pool and its scores are locked and versioned; "better" always means better than the locked baseline, not better than whatever ran last Tuesday.
3. The model-flip gate. Swapping a model - any tier, any task - requires at least 50 labeled examples, a paired comparison on identical inputs (a skeptic agent caught our original unpaired design being statistically underpowered), no scorer regressing beyond its noise floor, then a 7-day single-tenant canary with a feature-flag rollback measured in minutes.

Real numbers

1. Two independent judges - a fast grader audited by a stronger model - with Cohen's kappa as the trust metric between them.
2. 0/1/2 rubrics with locked floors: 100% on anchor facts, ≥80% exact-match on structured slots, graded grounding on narrative claims.
3. ≥50 labeled examples, paired comparison, 7-day canary before any model flip reaches customers.
4. Baseline locked and versioned - regressions are measured against a fixed point, not a moving memory.

Where the humans sit

Humans write the rubrics and label the seed examples - grading criteria are product decisions, and they're reviewed like code. Humans read the kappa reports and decide whether a disagreement means a broken rubric or a genuinely hard case. And no model flip reaches customers on green numbers alone: the gate produces evidence, a person makes the call. The judges never merge anything.

Steal this

Audit your judge before you trust a single number it produces: sample fifty of its grades, re-grade them with a different model that can't see the first one's answers, compute kappa. It's an afternoon. If agreement is high, you've earned real confidence in every eval you run after. If it's low, you just learned your evals were theater - which is worth knowing before the model swap, not after. And when judges disagree: fix the rubric, not the model.

Next week the build track meets the reviewer that isn't human. Then back here: graded output is expensive output - the event-sourced trick that stopped us regenerating anything twice.

---

This post is part of the product track of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.