Can an AI agent run a neighbourhood shop? Four models fell short
In an unreviewed Indian simulation, the best model earned about one-eighth of a scripted shopkeeper's profit. The test exposes language and accounting failures, not outcomes in a live business.
The short version
- What happened
- Four open-weight models each ran three 30-day shop simulations; the best averaged Rs 13,683 profit against Rs 103,434 for a fixed-rule reference.
- Who it affects
- Small retailers could bear the cost of missed orders, incorrect cash reports and language failures if an agent is trusted without reconciliation.
- What remains unproven
- Unreviewed preprint; one simulated Delhi shop; four model builds; three seeds per model and no live customers, workers or suppliers.
Keep in mind
Unreviewed preprint; one simulated Delhi shop; four model builds; three seeds per model and no live customers, workers or suppliers.
The scripted reference had direct stock access and language-matching capabilities that the model manager did not share.
Simulated messages and purchasing behaviour cannot establish real profit, adoption, safety, workload or employment effects.
No funding or competing-interest statement was found in the accessible paper, and the linked trace archive could not be independently opened during review.
Source analysis · 2 linked sources. Primary analysed source date: 16 October 2026. Earliest linked supporting source: 7 June 2024.
Read the full report · 5 min The evidence, methods, implications and limitations
The best model remained far behind the scripted reference
Four open-weight language models struggled to run a simulated Indian neighbourhood shop for a month, according to an unreviewed preprint submitted on 16 October. Each model was tested across three 30-day seeds, producing 12 simulated shop-months and 360 model-managed days. The model filled four roles—manager, cashier, helper and bookkeeper—while customers and suppliers communicated in English, Hindi and Hinglish. Demand was calibrated using India's 2022-23 Household Consumption Expenditure Survey.
A fixed-rule shopkeeper averaged Rs 103,434 profit. The best model, Qwen3.8-27B, averaged Rs 13,683—about one-eighth as much. Gemma 4 31B averaged a Rs 3,730 loss, while gpt-oss-20b and LFM2.5-8B-A1B lost Rs 193,411 and Rs 126,554 respectively. An unmanaged shop lost Rs 114,995. The three runs also varied widely: Gemma ranged from a Rs 37,226 profit to a Rs 40,762 loss, so model rankings should not be read as stable league-table positions.[1][2]
Language and reporting errors reached daily operations
The simulation makes failure concrete. In pooled supplier-email handling, Qwen failed 5% of messages, Gemma 7% and gpt-oss 46%. The last model failed on 23% of English messages, 51% of Hindi messages and 59% of Hinglish messages. LFM2.5 was excluded from that comparison because its model card did not support the Indic languages used. An English-only control recorded no failures across 459 messages, indicating that multilingual operation—not merely email volume—was a central difficulty in this setup.
Bookkeeping also differed. Qwen and Gemma reproduced daily sales exactly in the reported comparison, whereas gpt-oss misstated sales on 6% of days and LFM2.5 on 27%; the largest one-day discrepancies exceeded Rs 15,000. For a small retailer, those failures could mean missed stock, unpaid suppliers or decisions based on the wrong cash position. The study does not show that a human shopkeeper would accept those mistakes, or that a person supervising the system could not catch them.[1]
The distinction that matters
What the benchmark compared
The three conditions did not receive identical capabilities, which matters when interpreting the profit gap.
Language-model manager
Acts through simulated staff reports
Four models × three 30-day seeds
Fixed-rule reference
Reads stock directly and matches supplier language
Highest average profit
Unmanaged shop
No active replenishment or coordination
Loss-making reference
A controlled benchmark is not a field trial
The test is useful because it joins ordering, pricing, staffing, cash and communication over time rather than asking isolated questions. But it represents one simulated Delhi shop, one catalogue, one supplier schedule and one festival calendar. Customer and supplier language was itself model-generated. Only four specific open-weight builds were tested, with three seeds apiece, and the paper has not been peer reviewed. The authors did not report a funding or competing-interest statement in the accessible paper.
The fixed-rule reference is also a demanding but uneven comparator: it could read stock directly and communicate in the supplier's language, while the model manager depended on reports from its simulated staff. The result therefore shows that these configurations failed under the benchmark's constraints; it does not prove that every AI agent will fail or that the profit gap measures real-world productivity. Confidence would rise with more seeds and shops, preregistered field tests, human-supervised comparisons, representative language users and measured error recovery. Until then, shop owners should treat agent autonomy as an experiment requiring reconciled accounts and human control—not a replacement proven ready for the till.[1]