The same facts were wrong 15.9% of the time in Uzbek—and model choice narrowed the gap
Eight open models answered the same 100 questions in English, Russian and Uzbek. Language-adapted 7–8B models performed far better in Uzbek, but this was a model audit, not a study of students or learning.
The short version
- What happened
- Eight open-weight models answered 100 matched factual questions in English, Russian and Uzbek: 2,400 deterministic prompts in a fully crossed design.
- Who it affects
- A student working in Uzbek may receive a confident false fact from the same model that answers correctly in English. Institutions can test and choose tools in the language actually used, rather than treating English performance as a proxy.
- What remains unproven
- 100 short-answer questions
Keep in mind
100 short-answer questions
Three languages
Eight open-weight models
No proprietary frontier systems
One deterministic sample per prompt
Refusals excluded from error denominator
Machine-assisted translations
No student or learning outcome
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The language of the question changed the answer risk
Eight open language models made factual errors on 15.9% of scored Uzbek answers, compared with 10.9% in Russian and 3.5% in English, in a peer-reviewed Frontiers in Education study published on 29 October. The 95% confidence intervals were 13.5%–18.6% for Uzbek, 8.8%–13.4% for Russian and 2.4%–5.0% for English. The overall language difference had a medium reported effect size, although the practical gap varied sharply by model and topic.
That matters to a student using a chatbot as an informal study aid: the answer may sound equally fluent while being more likely to name the wrong person, number or scientific fact. The widest pooled English-to-Uzbek gaps appeared in Central Asian regional knowledge, general science and attribution of works or discoveries. This experiment measured model answers, however; it did not observe students, assignments, trust or learning.[1]
Every fact was tested across three languages and eight models
Researchers at Kokand University built 100 short-answer questions with unambiguous answers: 16 geography, 17 science, 17 history, 16 elementary mathematics, 17 attribution and 17 Central Asian regional items. Each question was asked in English, Russian and Uzbek to eight open-weight models ranging from one to 12 billion parameters, producing 2,400 prompts. Russian and Uzbek translations were drafted with machine assistance and checked by native-speaking authors. Models produced one greedy-decoded answer at temperature zero.
Scoring used accepted answer forms rather than another language model. Misspellings and recognisable transliterations counted as correct; wrong or unresolvable referents did not. Authors manually reviewed every automatic incorrect or refused verdict. Refusals were excluded from the denominator—a material choice because the one-billion-parameter Uzbek-adapted model returned only a question mark on 83 of 100 Russian prompts. Its apparent error rate therefore does not capture that severe non-answer behaviour.[1]
The distinction that matters
One question bank, three language conditions
The fully crossed design kept the fact and model fixed while changing the language of the prompt.
English
800 model-question combinations
3.5% pooled factual error
Russian
800 model-question combinations
10.9% pooled factual error
Uzbek
800 model-question combinations
15.9% pooled factual error
Adaptation helped, but the evidence stays narrow
Two Uzbek-adapted mid-sized models recorded 2% and 8% error in Uzbek. By contrast, general Gemma 3 4B and Qwen3 4B models reached 24% and 25%; a larger general Gemma 3 12B reached 5%. The pattern suggests that targeted language exposure and general capability both matter. It does not prove that any named model is safe for coursework, because the bank tests short facts rather than reasoning, citation, dialogue or subject teaching.
The audit excluded proprietary frontier systems, covered only three languages and sampled each prompt once. Translation artefacts may remain, and the authors inferred educational consequences without a learner study. They reported no funding or financial conflicts and disclosed generative-AI help with drafting and language editing. A university can use the released question bank as a starting point, but a stronger assessment would add local curriculum questions, count refusals as a separate failure, involve independent scorers and test whether students detect and correct errors before those errors enter assessed work. Evidence that students reliably verify the replies—or that adaptation fails on a broader local syllabus—would materially change the practical assessment.[1]