Change the rating scale, change the AI verdict
Across political, sentiment and toxicity tasks, changing one scale feature shifted about one in five category assignments. The unreviewed audit warns against treating LLM scores as stable measurements.
The short version
- What happened
- The experiment crossed two formats, four scale lengths, two signs, three tasks, two prompting strategies and five models.
- Who it affects
- A person or post could be placed in a different risk, sentiment or political category because an analyst changed the numbering rather than the underlying text.
- What remains unproven
- Unreviewed preprint; three English-language datasets and constructs; five model versions; no evidence that one scale is universally correct or that every downstream decision changes.
Keep in mind
Unreviewed preprint; three English-language datasets and constructs; five model versions; no evidence that one scale is universally correct or that every downstream decision changes.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A scoring choice moved roughly one verdict in five
Language models did not give stable text scores when researchers changed only the ruler used to ask the question. In an unreviewed experiment, switching between continuous and ordinal scales changed the assigned low, middle or high category for 18.8% of texts. Changing the number of points changed 18.2%, while moving between signed and unsigned scales changed 20.3%. Average normalised-score shifts were about a tenth of the full scale. For a moderation team or social-science project, that is large enough to alter which posts, parties or people appear extreme.
The design crossed two formats, four lengths—3, 5, 11 and 21 points—and signed versus unsigned numbering. Researchers repeated those conditions for political leaning, movie-review sentiment and toxicity, under zero-shot and rubric prompts, using five proprietary and open models. The English-language datasets contained 1,700 media-bias texts, 5,000 movie reviews and 5,000 toxicity examples. This is a systematic sensitivity test, not a claim that every model decision is wrong.[1]
Toxicity was especially vulnerable to the ruler
Toxicity produced the biggest category changes: between 29% and 40% depending on which scale feature moved. On continuous scales, changing the sign altered 24.4% of categories, compared with 16.3% on ordinal scales. The paper suggests one reason is conceptual. Political leaning and sentiment run between two opposing poles, while toxicity is a one-directional construct from absence to greater presence. A signed scale may tempt a model to treat zero as the boundary for harmlessness even when the verbal labels imply something else.
More points reduced the numerical effect of switching formats, but did not make the resulting categories more stable: 18.2% to 20.1% still changed at every scale length. A written rubric also failed to reduce sensitivity relative to a zero-shot instruction. Extra reasoning helped a little for three open models, but the improvements were small and inconsistent. The audit therefore challenges a familiar comfort: a detailed prompt can make a measurement look disciplined while leaving the result dependent on arbitrary scale design.[1]
The distinction that matters
Three scale edits, three routes to a different category
Each comparison changed one prompt feature while holding the task, text and other scale features fixed.
Format
Continuous ↔ ordinal
18.8% switched category
Range
3, 5, 11 or 21 points
18.2% switched
Sign
Signed ↔ unsigned
20.3% switched
This is a robustness warning, not a universal error rate
The study compares each model with itself under minimally different prompts; it does not show that the new category is less accurate than the old one. Human labels were used for descriptive comparison, not as a complete causal explanation of why scores moved. Only three tasks, three English datasets and five current models were tested. Political categories were heavily concentrated in the middle, sentiment was more polarised, and toxicity had its own class distribution, so part of the difference may come from those datasets rather than the underlying concept alone.
No funding or competing-interest statement appears in the accessible preprint. Before an institution uses LLM scores to flag abuse, measure ideology, rank applicants or compare populations, it should report the exact scale wording and rerun the analysis with plausible alternatives. Decisions that change under a defensible relabelling need human review or a more stable measurement method. Confidence would rise with preregistered replication across languages, domains, model versions and human-designed scales, plus tests of downstream decisions rather than scores alone. The finding is not that numeric prompts are useless; it is that their apparent precision can conceal a fragile choice of ruler.[1]