Can 96.46% AI-text detection travel?
A peer-reviewed benchmark separates human, AI-generated and AI-rephrased reviews and posts with high in-distribution accuracy. But the test reuses the same three generator families and two domains as training, while acknowledged overlap and corpus inconsistencies counsel against real-world verdicts.
The short version
- What happened
- The public corpus contains 161,788 English samples: 46,844 human-written, 57,247 AI-generated and 57,697 AI-rephrased Amazon reviews and Twitter posts.
- Who it affects
- A three-way benchmark better reflects assisted editing than a simple human-versus-AI label.
- What remains unproven
- All partitions use the same GPT/ChatGPT, DeepSeek and Kimi generator families; unseen-generator generalisation was not evaluated.
Keep in mind
All partitions use the same GPT/ChatGPT, DeepSeek and Kimi generator families; unseen-generator generalisation was not evaluated.
Only English Amazon reviews and Twitter posts were included; other genres, languages and code-switching are untested.
The AI-rephrased class reflects one few-shot prompting procedure, not the full range of AI-assisted editing or human–AI co-writing.
The paper acknowledges 106 train–test duplicate texts and 18 original–derivative pairs split across partitions.
Corpus counts were corrected from earlier versions, and the article contains a conflicting 180,000-sample description in its data-availability statement.
Ensemble confidence intervals come from one split and do not account for variation across training seeds.
Source analysis · 2 linked sources. Primary analysed source date: 5 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The benchmark asks a harder three-way question
Most automated text-origin studies reduce authorship to two labels: human or AI. Najma Sadia and colleagues instead created a third class for AI-rephrased material—human text whose wording and syntax were altered by a model while its meaning was preserved. That distinction matters for journalism, education and platform moderation because assisted editing is common and may sit between fully human drafting and fully synthetic generation.
The deposited English-language corpus contains 161,788 samples: 46,844 human-written, 57,247 AI-generated and 57,697 AI-rephrased. Human material came from Amazon product reviews and Twitter posts. GPT/ChatGPT, DeepSeek and Kimi generated new reviews and posts from zero-shot instructions, then rephrased human originals from three few-shot demonstrations. The authors retained texts from 10 to 512 words, cleaned and normalised them, and divided them into 113,249 training, 16,180 validation and 32,359 test samples. This is a sizeable, reproducible semi-synthetic benchmark, not a representative sample of everything people publish.[1][2]
Transformers and stacking score highly inside the benchmark
For classical models, the researchers built a 168-dimensional representation: 68 stylistic, lexical, linguistic, syntactic and stylometric features plus 100 dimensions from RoBERTa and MiniLM embeddings. Logistic regression and a linear support-vector machine each reached about 74% accuracy, while random forest reached roughly 80%. End-to-end transformer fine-tuning was much stronger: RoBERTa-base reported 95.49% accuracy and a 95.50% weighted F1 score; DeBERTa-v3-base reported 94.88% accuracy and a 94.93% weighted F1.
Averaging transformer probabilities produced 95.66% accuracy. A stacking system then fed model outputs to several meta-learners, with random forest producing the headline 96.46% result. Its reported F1 score for the AI-rephrased class was 0.96. Ablations suggested handcrafted features added information beyond embeddings for the classical models. These are technically useful comparisons under a shared split, but the decimal precision should not be mistaken for confidence about a school essay, newsroom submission or future model that was never represented in the corpus.[1]
The test stays inside the same generators and domains
The central limitation is distribution shift. All three partitions draw from the same GPT/ChatGPT, DeepSeek and Kimi families and from the same Amazon-review and Twitter-post domains. The study did not hold out one generator family, train on one domain and test on the other, or evaluate Claude, Gemini, Llama, Mistral, essays, news, blogs, technical writing, multilingual text or code-switching. The authors explicitly describe the figures as in-distribution performance and identify unseen-generator evaluation as the primary next step.
That boundary is crucial for use. A detector may learn stable artefacts of the selected prompts, decoding settings, cleaning rules or platforms rather than a durable signature of machine authorship. AI-rephrased examples were made with one few-shot procedure—three demonstration pairs, temperature 0.5 and top-p 0.9—so the class does not cover every editing workflow or human–AI collaboration. A score produced outside this design is an extrapolation, not a validated probability that a person used AI.[1]
Acknowledged overlap and corpus corrections narrow confidence
A post-split inspection found 106 test texts, 0.33% of that split, also present in training, mostly short repeated Amazon reviews. The authors also found 18 cases in which an original and a synthetic derivative landed in different partitions. They call the contamination minor and unlikely to change the reported result materially, but group-level splitting before synthetic generation would have been the safer design. Even small leakage is relevant when performance differences between top models are below one percentage point.
The paper also corrects inaccurate corpus statistics from earlier manuscript versions. It says the deposited dataset contains 161,788 samples and notes that short human Amazon reviews were partly retained despite the stated ten-word filter. Yet the data-availability section separately describes a balanced corpus of 180,000 samples. One RoBERTa figure caption also says its displayed human row came from an earlier test split, although the summary metrics were calculated on the corrected split. Transparency about those discrepancies is welcome; it also means independent replication from the deposit and code is necessary before treating 96.46% as a stable benchmark.[1][2]
A benchmark score should not become an accusation
False authorship claims can affect grades, employment, reputation and access to publishing. This study did not test a disciplinary workflow, estimate false-positive rates for particular demographic or language groups, or examine what happens when people revise model output heavily. It does not justify using a detector as sole evidence of misconduct. Even a high average accuracy can hide consequential errors, and the relevant error rate will change as real-world class balance and writing style depart from the benchmark.
The constructive use is narrower. Researchers can use the open corpus to compare three-way methods, probe why rephrased text is difficult and design stronger out-of-distribution tests. Publishers or platforms considering detection should demand calibrated scores, transparent abstention, appeal routes and corroborating provenance evidence. Cooperative signals—document history, disclosed assistance or cryptographic provenance—answer a different question from stylistic classification and may be safer than inferring authorship from text alone.[1]
What would change the assessment
The decisive experiments are leave-one-generator-out and leave-one-domain-out evaluations with deduplicated, group-split data fixed before training. A prospective test should collect naturally occurring human, fully generated, lightly edited and heavily co-written material without revealing labels to developers, then report calibration and class-specific false positives across genres, languages and writer groups. Multiple training seeds, locked external test sets and comparisons with simple baselines would show whether the stacking gain is durable.
The work was supported by the Republic of Korea’s ANCHOR programme through the Jeju ANCHOR centre; the authors declared no competing interests and disclosed using ChatGPT to improve manuscript language and clarity. The present evidence supports the benchmark contribution and the conclusion that rephrased text can be separated well under matched conditions. It does not establish a universal detector, a reliable verdict on an individual document or generalisation to the next generator. Until those external tests arrive, the headline number belongs inside its test enclosure.[1][2]