Can AI judges be accurate yet disagree on the errors that matter?
Yes. In a new unreviewed study of ten reasoning models, average accuracy stayed high on a 600-item answer-checking benchmark, while prompt and model choices sharply changed whether errors were too lenient or too strict. The result argues against trusting one headline accuracy score—or one AI judge.
The short version
- What happened
- The preprint compared ten reasoning models across 120 judge designs for sentiment and toxicity ratings and 60 designs for a 600-item answer-checking task.
- Who it affects
- People affected by automated scoring may face different false-acceptance or false-rejection risks even when systems show similar average accuracy.
- What remains unproven
- The paper is a preprint submitted on 4 October 2026 and has not been peer reviewed.
Keep in mind
The paper is a preprint submitted on 4 October 2026 and has not been peer reviewed.
The experiments cover two scalar-rating tasks and one answer-classification task, not the full range of real evaluation work.
Answer-classification ground truth used at least 90% model-judge consensus for 553 of 600 items; only 47 ambiguous items were manually resolved.
Human rating labels contained non-zero disagreement, particularly for toxicity, where Krippendorff’s alpha was 0.68.
Candidate answers came from a single provider’s model, introducing a possible provider-adjacent preference for one judge.
Descriptive thresholds such as a 0.2-point bias are not established standards for acceptable judge reliability.
Source analysis · 2 linked sources. Primary analysed source date: 4 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: similar accuracy can conceal different failure patterns
An AI judge can look accurate on average and still make a meaningfully different kind of mistake from another judge. Across 60 designs used to classify whether 600 generated answers were correct, the study reports average accuracy of 96.5%, with design-level results ranging from 92.7% to 98.3%. Yet prompt detail and model identity changed whether errors were false positives—wrong answers accepted as correct—or false negatives—correct answers rejected.
That distinction matters whenever automated evaluation influences a model ranking, a safety claim or a release decision. A lenient judge can inflate apparent performance; a strict judge can suppress it. The study’s practical message is therefore narrower than ‘AI judges work’: headline accuracy is insufficient, ambiguous cases deserve special attention, and evaluation should report error direction and use more than one judge where stakes are high.[1]
What the researchers actually tested
The University of Stuttgart researchers tested ten reasoning models from several providers. For scalar rating, they built separate sentiment and toxicity benchmarks. A model generated 600 sentences for each dimension; after reserving examples for in-context prompts, the evaluation sets contained 501 sentiment items and 521 toxicity items. Each sentiment item received ten human ratings and each toxicity item received 15 ratings on a seven-point scale.
For answer checking, the researchers drew 600 difficult, non-image, non-multiple-choice questions from Humanity’s Last Exam. Candidate answers were generated by one model, then the ten judges saw the question, the candidate response and the verified reference answer and returned a binary correct-or-incorrect verdict. Public study materials are linked through OSF, making the benchmark and prompts available for scrutiny and replication.[1][2]
The design comparison was unusually broad
For sentiment and toxicity, the study crossed ten models with three prompt styles and four output scales: 1–7, -3–3, 0–1 and 0–100. That produced 120 judge designs per rating benchmark, or 240 evaluated designs across the two tasks. The minimal prompt supplied the task and scale; short and long variants added examples, scoring anchors, explicit principles and, for the longest version, a step-by-step procedure.
For answer checking, the researchers crossed the ten models with three prompt-detail levels and two reasoning-effort settings, producing 60 designs. They measured not only accuracy, but leniency, error switching and performance on subsets defined by agreement. The 600 items split into 429 on which all designs agreed, 124 with at least 90% agreement, and 47 ambiguous cases resolved manually.[1]
Model choice mattered more than the headline average suggests
Changing only the model produced an average 1.2-percentage-point gap in answer-checking accuracy, but a 23.2-point gap in leniency. Across ambiguous items, two otherwise matching designs that used different models disagreed on 36.5% of verdicts on average, and the accuracy gap rose to 12.7 points. Model identity was also the main source of variation in the rating tasks and often set whether a judge tended to over-rate or under-rate compared with human scores.
No model was consistently best across the rating designs. Five different models placed first across the sentiment configurations and six did so across toxicity configurations. The authors recommend aggregating two or three judges or designs. That is a sensible robustness check, but it is not a guarantee of truth: correlated models can share blind spots, and majority voting is only as good as the evidence and independence behind the voters.[1]
Longer prompts changed strictness more than accuracy
On the answer-checking task, longer prompts did not materially improve the overall accuracy average. They did, however, make judges stricter: the proportion of errors that were false positives fell from 81.3% with the base prompt to 52.4% with the long in-context prompt. The maximum leniency shift attributed to prompt detail was 28.9 points; switching models produced a maximum shift of 56.1 points.
Lower reasoning effort showed little average effect on accuracy or leniency in this particular task. That finding may matter for evaluation cost, but it should not be generalized to other reasoning workloads. The experiment varied only two effort settings on a constrained binary classification setup in which each judge received a reference answer; it did not test open-ended grading without a reference or consequential real-world decisions.[1]
Subjective ratings were less stable than sentiment scores
Human agreement was stronger for sentiment than toxicity: Krippendorff’s alpha was 0.86 for sentiment and 0.68 for toxicity. The model judges showed the same broad pattern. Average deviation from the human reference was small across rating designs, but toxicity was more sensitive to prompt and scale choices. A centered -3-to-3 scale was especially problematic because zero could be interpreted as ‘neutral’ rather than merely the numerical midpoint.
This is important for safety evaluation because concepts such as harmfulness, toxicity and policy compliance are partly judgement-laden. A benchmark can make those judgements look mechanical while hiding contested definitions and annotator uncertainty. The paper’s data support closer inspection of subjective and borderline cases, not replacing human governance with a larger panel of machines.[1]
The ground truth is the study’s most important limitation
The rating benchmarks used multiple paid human annotators, informed consent and ethics approval, but human labels still contained measurement uncertainty. More consequentially, the answer-checking ground truth was partly built from the evaluated judges themselves. When at least 90% of the 60 designs agreed, their majority verdict became the label; only the remaining 47 items were manually resolved. That makes the reported 96.5% accuracy partly dependent on consensus within the system being assessed.
The authors acknowledge this weakness. It can make broad agreement look like accuracy even if many models share the same mistake. Candidate answers also came from one provider’s model, creating a possible provider-adjacent preference in another model from that provider. The results therefore describe internal robustness across these benchmarks and designs; they do not establish that AI judges can replace independent expert evaluation.[1]
What this changes for people who rely on AI evaluations
For researchers, procurement teams and safety reviewers, the immediate implication is to ask what sits behind a single evaluation score. Reports should identify the judge model and version, disclose the exact prompt and scale, separate false acceptance from false rejection, and show results on ambiguous subsets. Repeating the evaluation with independent models can reveal sensitivity that one configuration would miss.
For the public, the finding helps explain why claims that one AI system is ‘safer’ or ‘more accurate’ can change when the evaluator changes. That does not mean every AI benchmark is arbitrary. It means methodological choices are part of the evidence and should be visible. Decisions affecting access, moderation, employment, health or public services still need accountable human review and domain-specific validation.[1]
Funding and what would change the assessment
The work was supported by the Baden-Württemberg Ministry of Science, Research and the Arts through University of Stuttgart programmes. The paper reports ethics approval for the human-rating study and average annotator compensation of £9.95 an hour. It is an arXiv preprint, so its methods and interpretations have not yet passed journal or conference peer review.
Confidence would rise with independent reproduction across more domains, languages and open-ended tasks; ground truth established without the evaluated judges; preregistered thresholds; and expert adjudication of every disputed answer. Real deployment studies should test whether multi-judge evaluation actually catches harmful regressions, improves release decisions and remains reliable as models and provider APIs change.[1]