The odds stayed the same—why did AI advice change?
Across seven models and 240 synthetic high-stakes scenarios, complementary gain and loss wording changed the recommendation category in 32.4% of complete comparisons. Almost all shifts involved uncertainty rather than a direct yes-to-no reversal.
The short version
- What happened
- FrameBench-HPS generated 2,880 prompts from 240 synthetic scenarios across healthcare, finance, education or career, and legal or consumer-rights domains.
- Who it affects
- People asking about treatment, money, education or legal remedies may receive more hesitant or forceful language depending on equivalent wording. Interfaces should show the underlying probability and avoid presenting one phrasing as neutral.
- What remains unproven
- This is an unreviewed synthetic benchmark, not an evaluation of real decisions, outcomes or professional advice.
Keep in mind
This is an unreviewed synthetic benchmark, not an evaluation of real decisions, outcomes or professional advice.
Scenarios used four probability levels and controlled templates; broader uncertainty and multi-turn conversations were not tested.
Human rating covered 140 pairs, and inter-rater agreement was moderate or lower across the three qualities.
The paper reports no separate funding or competing-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent probabilities changed commitment
Researchers built 240 synthetic scenarios spanning healthcare, finance, education or career, and legal or consumer rights. Each appeared at 20%, 40%, 60% and 80% positive-outcome probabilities, with neutral, gain and mathematically complementary loss wording. Seven models returned a recommendation category, risk score, actionability and confidence, producing 2,880 prompts before repeated runs and robustness checks.
Among 6,644 complete model-scenario-probability groups, gain versus loss wording changed the recommendation category 2,153 times, or 32.41%. Yet 98.1% of those changes involved the uncertain category; direct yes-to-no reversals occurred in only 42 groups, 0.63%. Loss framing raised model-assigned risk by 0.285 points on average and made answers more directive, even though the underlying odds and scenario facts were unchanged.[1]
People noticed the tone shift, but decision quality was not tested
Human annotators rated 140 paired responses, balanced across domains, probabilities and models. Loss-framed answers scored 0.27 points higher for risk presentation and 0.32 higher for directiveness. Agreement was moderate at best: ordinal alpha was 0.572 for risk, 0.480 for directiveness and 0.419 for expressed certainty. A held-out paraphrase test covered 32 scenarios across three model families and found the effect persisted beyond fixed templates.
The scenarios are synthetic and use only four probability levels. The study measures consistency, not whether the recommendation is correct, beneficial or harmful. Multi-turn context, personalised risk and real professional decisions were outside scope. Rewriting both frames into a canonical probability reduced sensitivity, which suggests a practical control, but it needs prospective testing with domain experts and real users before it can be treated as a safety intervention.[1]
The distinction that matters
One probability, three presentations
The facts stay fixed while the sentence around the probability changes.
Gain
State the chance of the favourable outcome
Foregrounds success
Loss
State the complementary chance of failure
Foregrounds harm
Compare
Hold scenario and probability constant
Measures framing sensitivity