Web agents worked far worse in Amharic and Urdu—what did the benchmark reveal?
Eight models attempted 3,000 localized task instances across six simulated sites. English-only evaluation overstated reliability, especially for open-weight agents and pages with pop-ups.
The short version
- What happened
- LoWAgent contains 600 unique tasks localized into English, Bengali, Hindi, Urdu and Amharic across six simulated websites, creating 3,000 task instances.
- Who it affects
- A person using an agent in Amharic, Urdu, Bengali or Hindi may receive a less reliable service even when the website offers an equivalent localized task.
- What remains unproven
- This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
Keep in mind
This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
Source analysis · 1 linked source. Primary analysed source date: 19 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent tasks produced unequal results across languages
A new benchmark suggests that testing web agents only in English can substantially overstate how well they will serve people using other languages. LoWAgent uses six simulated sites—shopping, a forum, used cars, rail tickets, government services and airline booking—with the same code and underlying state in English, Bengali, Hindi, Urdu and Amharic. Its 600 unique tasks become 3,000 localized instances, including 750 visually grounded cases.
Eight models attempted the tasks. Gemini 3.1 Flash-Lite had the smallest reported gap, losing 1.5 to 7.3 percentage points outside English. GPT-5.6 Luna lost 16.6 points in Amharic, while Llama-3.1-8B and GPT-OSS-20B lost 34.5 and 33.3 points respectively. Visual tasks were harder: the best model reached about 60% in English versus 86.4% on text-only English tasks.[1]
Common pop-ups exposed another reliability gap
The team added a cookie notice and an advertising pop-up that agents had to dismiss. Gemini's success fell from around 80% across languages to below 20%; several open models lost more than half their baseline performance. Post-training two open models on 450 tasks improved held-out success by roughly five points overall, with larger gains in the four lower-resource languages.
For people using agents to book travel, find public services or shop, the result means language and interface conditions should be tested together before deployment. Native speakers reviewed the Fable-assisted localization and every task was manually executed, but the websites were artificial and did not include the full diversity of real layouts, dialects, assistive technologies or network conditions. Independent live-site evaluation and user studies would change the assessment.[1]
The distinction that matters
What changed while the task stayed the same
The benchmark held site logic and ground truth constant while varying language and interface obstacles.
Language
Same task localized five ways
Success gaps become measurable
Interface
Cookie and advert interruptions appear
Agents must recover before acting
Training
Small multilingual subset is used
Held-out performance improves modestly