LLM-as-a-Judge Bias: We Audited 10 AI Judges (839 Calls)

As part of the Startrise LLM Benchmark, we made ten frontier models grade each other’s frontend work (839 blind scoring calls across twelve tasks) to measure LLM-as-a-judge bias directly. The most biased judge, Claude Haiku 4.5 (self-bias +7.04), was also the worst-performing builder in the field. The harshest judge, Claude Opus 5 (leniency −7.25), still favoured its own work by +2.87. Three models — Claude Sonnet 5, Grok 4.5, and Kimi K3 — scored themselves below what the panel gave them. And because the benchmark’s winner sat on its own judging panel, we report the top of our own leaderboard as a tie.

Twelve Models, Twelve Briefs is the flagship study — the leaderboard, the per-task winners, the bug list. This post covers the one part of that study nobody else publishes: the audit of the judges themselves.

What is self-preference bias in LLM-as-a-judge?

LLM-as-a-judge means using a language model to score another model’s output. It’s the pattern behind most automated eval pipelines, agent QA loops, and public leaderboards in 2026. Self-preference bias is the failure mode where the judge systematically scores its own output higher than independent judges score it. In the Startrise LLM Benchmark’s 839-call audit, measured self-bias among ten frontier judges ranged from +7.04 (Claude Haiku 4.5) to −1.39 (Claude Sonnet 5).

The research record says the bias is real and mechanical, not vain. Self-Preference Bias in LLM-as-a-Judge (arXiv 2410.21819) found that “GPT-4 exhibits a significant degree of self-preference bias” and traced the mechanism to perplexity: LLMs “prefer texts more familiar to them”. And nothing is more familiar to a model than its own output. Follow-up work, Quantifying and Mitigating Self-Preference Bias of LLM Judges (arXiv 2604.22891), defines it as a directional deviation: judges can favour or disfavour their own work. Our audit found both directions in the wild.

The stakes are simple. If a judge model gates your CI, approves your agent’s outputs, or ranks a public leaderboard, its bias is multiplied by every call it makes — silently.

How we measured it: 839 blind cross-judgings

Separately from the Startrise LLM Benchmark itself, we made every available model in the roster judge every other model’s work across all twelve tasks: 839 scoring calls. Every submission was anonymised and shuffled; no judge ever learned authorship. From those calls we computed six numbers per judge (plus the raw self-score the table reports alongside them):

• Mean given — the judge’s average score across the field.
• Leniency — mean given relative to the panel average. Positive grades soft; negative grades hard.
• Self-bias — how much better the judge scored its own work than the rest of the panel did, after subtracting that judge’s leniency.
• Vendor bias — favouring same-family models beyond general leniency.
• Discrimination — the spread between a judge’s best and worst scores. A judge that clusters everything mid-scale isn’t judging.
• Agreement — distance from panel consensus; lower is better.

The leniency subtraction is the step most write-ups skip. “Grades everyone generously” and “grades itself generously” are different failures, and unless you remove the first you’ll misdiagnose the second. Haiku’s +7.04 self-bias sits on top of its +6.92 leniency.

One honesty note on the count. The study prose says every model in the roster judged; the published audit table reports ten judges: DeepSeek V4 Pro and Qwen 3.7 Max are absent, and GLM 5.2 judged the field but has no self-score. We verified against the run data at the benchmark: ten is the number the data supports.

Why judge bias matters this much here: in the parent benchmark, a three-judge panel (three labs, three lenses, median per axis) carries most of the signal. The judge column spans 32.7–81.3 while the automated technical column spans just 81.1–94.4. When the judges are the instrument, judge bias is instrument error. (One run, one task family, N=1; the full caveat ledger lives in the flagship.)

The audit table: every judge, every bias

Ten judges from five labs, seven metrics, 839 blind cross-judgings in the Startrise LLM Benchmark: Claude Haiku 4.5 is simultaneously the most lenient judge (+6.92) and the most self-favouring (+7.04); Claude Opus 5 is the harshest (−7.25) and still favours itself (+2.87); GPT-5.6 Sol carries the only clear vendor-loyalty signal (+1.98); and three judges rated their own work below the panel’s verdict.

Run 2026-07-26-0159. Leniency is relative to the panel average; self-bias is the own-work premium after leniency is subtracted; agreement is distance from consensus (lower is better). Full data at the benchmark.

Four stories live in that table.

Haiku’s double crown. Claude Haiku 4.5 is the only judge to lead both the leniency column (+6.92) and the self-bias column (+7.04), a combination damning enough to get its own section below.

Opus, the hanging judge. Claude Opus 5 grades the field 7.25 points below the panel average (the harshest judge tested), yet rates its own work +2.87 above what the other judges gave it. In the study’s words: it holds everyone to a high bar and itself to a slightly lower one.

Sol’s family loyalty. GPT-5.6 Sol favours its GPT-5.6 siblings by +1.98 beyond its general leniency. Small in absolute terms, but it’s the only clear vendor-loyalty signal in the data. Sol also carries the second-highest self-bias in the field, +3.33.

The self-critics. Three models scored their own work below what the panel gave it: Claude Sonnet 5 (−1.39), Grok 4.5 (−0.93), and Kimi K3 (−0.83). Genuine self-criticism, and Sonnet 5 is the most self-critical model tested. The remaining soft graders, GPT-5.6 Luna (+5.32 leniency) and GLM 5.2 (+4.11), grade generously across the board without a comparable self-premium.

The worst builder was the most biased judge

Claude Haiku 4.5 finished last in the Startrise LLM Benchmark at 44.8 overall (last place on eight of twelve tasks) and was also the most lenient judge in the audit (+6.92) and the most self-favouring (+7.04). It scored its own submissions 50.63, the lowest raw self-score of any judge. Even that was still seven points more generous than the panel’s verdict on the same work.

The study’s frame, verbatim: a weak model that grades generously and grades itself most generously of all is the exact profile you must keep off a panel.

This matters because of how judges get chosen in practice. Judge calls are high-volume, so teams reach for a small, cheap model to keep the eval bill down. In this data, the budget-tier judge was the biased judge — at least here, at least on this task family. If your QA gate runs on the cheapest model you could find, the audit says that choice deserves its own audit.

The fairness line, which we hold to: Haiku 4.5 is a fine model for its price and tier. Grading frontier frontend work is simply not its domain.

Judging skill is not building skill

The best-calibrated judge in the Startrise LLM Benchmark audit was a mid-table builder. GPT-5.6 Terra finished 6th on the leaderboard, yet as a judge it pairs a 17.30 discrimination (the second-widest spread between best and worst work in the field, behind only GPT-5.6 Luna’s 17.65) with strong agreement (3.94; lower is better). Luna’s wider spread comes with far weaker agreement (6.28) and +5.32 leniency; Terra distinguishes quality and tracks the consensus, which makes it the best-calibrated judge in the roster.

The inverse holds too. Claude Opus 5, the strongest builder in the study, is the harshest judge (−7.25) with a +2.87 self-premium. Kimi K3, the second-strongest builder, is a mild self-critic (−0.83) with tight agreement (3.48) but middling discrimination (16.63). Skill at making the thing and fairness at grading the thing are separate, separately measurable properties.

The research agrees. arXiv 2604.22891 reports that “advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB”. That’s their finding, on their tasks, and our audit lands the same way. Don’t staff your panel with your strongest generator by default. Measure calibration first.

Choosing a model to build with is a different question with different answers. That lives in our best LLM for frontend buying guide, not here.

Our conflict of interest — and what we did about it

Claude Opus 5 won the Startrise LLM Benchmark and also sat on its three-model judging panel as the Craft judge. Its measured self-bias is +2.87. That’s why our headline result reads “Opus 5 and Kimi K3 tied at the top” (82.3 vs 79.8), not “Opus wins.”

The full confession: one third of the panel was a competitor grading itself, and two more competitors held the other seats — Grok 4.5 as the Brief judge and GPT-5.6 Terra as the Engineering judge. Every submission was anonymised and shuffled, and no judge ever learned authorship. That mitigates the problem; it doesn’t eliminate it. Blinding hides names, not style, and the familiarity mechanism from arXiv 2410.21819 survives anonymisation.

The numbers cut both ways, and we report both. Opus’s +2.87 self-bias is small, and Opus is also the harshest judge in the field, so its seat suppressed scores more than it inflated its own. But +2.87 is not zero, and one third of the panel scoring the winner is a structural weakness in the design, not a rounding error. So we priced it into the claim: Opus 5’s 2.5-point margin over Kimi K3 is inside the noise floor this conflict creates, and the honest reading is a tie.

A leaderboard that won’t audit its own judges is asking you to trust what it refused to measure.

How to design a judge panel you can trust

A panel’s trustworthiness isn’t a vibe. It’s a property you can compute before you rely on it. Every rule below cites either a number from our 839-call audit or a named paper.

1. Blind everything. Anonymise and shuffle submissions; never leak authorship, filenames, or model-flavoured metadata. Bias you can measure is manageable; an authorship leak is bias you can’t measure afterwards.
2. Cross lab lines — but don’t expect miracles from headcount. Sol’s +1.98 sibling premium is what a same-family panel bakes in permanently. And diversity has limits: Nine Judges, Two Effective Votes (arXiv 2605.29800) found a nine-judge panel spanning seven model families delivers “only about 2 independent votes’ worth of information” because the models make the same mistakes on the same items, and “the best single judge matches or outperforms the full panel.” That’s on NLI datasets, and our frontend context differs, but it kills the just-add-judges reflex.
3. Aggregate with median per axis, on an odd panel. One outlier judge cannot drag the result, and the aggregate is always a score a real judge actually gave. Flag genuine disagreement (we mark any 4+ axis spread SPLIT PANEL) instead of smoothing it away.
4. Measure leniency and self-bias separately, before trusting any judge. A cross-judging harness is a weekend of work; a wrong leaderboard is not. Without the decomposition you’d read Haiku’s generosity as kindness rather than +6.92 leniency stacked under a +7.04 self-premium.
5. Keep weak-and-lenient models off the panel. The worst builder in our field was the softest, most self-favouring judge in it. Cheap judge calls are only cheap until they gate the wrong thing.
6. Disclose conflicts and price them into your claims. Our winner judged itself; our headline says “tied.” If a conflict doesn’t change your conclusion, you haven’t priced it.
7. Keep a human gate where consequences are real. Our blend weights blind human review at ×0.30, and 106 of 144 cells carried it. How to wire that gate without stalling the pipeline is its own guide: risk-tiered approval workflows for AI agents.

Your product probably has an unaudited judge in it

If an LLM grades, filters, ranks, or approves anything in your pipeline (support triage, content moderation, model-graded CI, lead scoring), you’re running a judge nobody has audited. This post is what an audit finds: leniency you didn’t choose, self-preference you can’t see, a conflict of interest somewhere in the loop.

That’s billable work for us, and we do it in both directions. We wire the QA gates, human review queues, and audit logs that make automated judgment trustworthy (AI Monitoring & Guardrails, from $2,500), and we red-team the AI systems themselves, prompt injection to tool abuse, in an AI Security Audit from $4,500. The agent pipelines these judges sit inside are our AI automation practice.

We audit AI systems for a living. This post is what it looks like when we turn that on our own.