Could a better benchmark score hide a poisoned training dataset?
A preprint shows how a malicious benchmark provider could supply training data that improves a model's public score while installing a hidden trigger. All 30 tested task–model–scenario combinations gained benchmark performance and retained high attack success.
The short version
- What happened
- Researchers tested five attack scenarios across MMLU, TweetEval and FalseReject using Qwen2.5-3B and Gemma 3 4B: 30 task–model–scenario configurations in total.
- Who it affects
- Model teams and downstream buyers should ask who created an evaluation, who supplied the improvement data and whether independent tests were run after fine-tuning. A higher score is not, by itself, evidence of a safer model.
- What remains unproven
- The paper is an unreviewed preprint and has not been independently replicated.
Keep in mind
The paper is an unreviewed preprint and has not been independently replicated.
Only two small open model families and three NLP tasks were tested.
Five attack scenarios were constructed; no real benchmark operator or released model is alleged to be compromised.
The threat model assumes an adversary can influence both a valued benchmark and follow-on training data.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The attack turns benchmark improvement into the lure
BenchDoor assumes a benchmark creator or data supplier can publish a test and later offer examples that appear to help developers improve against it. Some examples contain a trigger linked to an attacker-chosen output. The researchers applied this threat model to three tasks—MMLU multiple choice, TweetEval sentiment and FalseReject open-ended answers—and two open model families. Poison rates were 5%, 10% and 2% respectively.[1]
The hidden behaviour survived all 30 tested configurations
Across five scenarios, three tasks and two models, all 30 configurations improved the target benchmark and produced held-out attack success between 82.33% and 100%. The authors then generated synthetic training data and removed examples similar to the benchmark. Attack success still reached at least 90.33%. That matters because the visible score gain can reward the very training step that carries the backdoor.[1]
The distinction that matters
How a benchmark can become a delivery channel
A score improvement can conceal the origin of the training signal.
Benchmark
Provider defines a valued public test
Developers optimise toward its score
Dataset
Helpful examples include a hidden trigger
Fine-tuning installs a backdoor
Verification
Independent data and evaluations
Break the single-provider dependency
Supply-chain controls matter more than one decontamination check
The study covers small models, three NLP tasks and artificial scenarios. It does not show a real benchmark provider poisoning production systems, and the paper is not peer reviewed. Developers can reduce risk by separating benchmark ownership from training-data supply, verifying dataset provenance, scanning for triggers, testing on independent evaluations and reviewing score jumps that depend on a narrow external dataset. Evidence from larger models or a documented live incident would strengthen the conclusion.[1]