Tlaxcala’s assistant invented no fee in 40 audited AI answers—but half of clarifying users left
New analysis today of an unreviewed 27 October preprint: 380 production traces suggest deterministic routing can contain numeric errors, while slow generative replies and a 53% clarification drop-off expose the human cost of friction.
The short version
- What happened
- The Tlaxcala government assistant covered 1,143 procedures and 924 offices; its study analysed 380 organic production traces from 312 sessions over eight weeks.
- Who it affects
- A public assistant can avoid fabricating a fee yet still fail someone who waits half a minute or leaves an unclear menu. Completion, accessibility and recovery when the system is wrong matter as much as answer accuracy.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
One state portal
380 traces and 312 sessions
Only 40 generative production answers
Raw traces not public
Author designed and evaluated the platform
No service-completion outcome
Low traffic and CPU queueing
Limited fairness probe
Source analysis · 1 linked source. Primary analysed source date: 27 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The safest route usually avoided generation
A public-service assistant operating on the Tlaxcala state portal did not invent a fee, deadline or other number in 40 audited generative answers, according to an unreviewed Mexican preprint submitted on 27 October. The more important design choice was what happened before generation: 340 of 380 production traces, or 89.47%, were answered through deterministic retrieval. Only 40 reached the language-model synthesis layer.
The service covered 1,143 government procedures and 924 offices. Researchers examined 380 organic traces from 312 sessions between 15 February and 15 April 2026, alongside a separate 200-query stratified benchmark. The complete system reached 86.50% strict top-one accuracy on that benchmark—ten percentage points above a simpler binary gate—and averaged 86.41% across five folds, with a wide 7.50-point standard deviation.[1]
Zero observed inventions is not zero risk
Two analysts who were not involved in development audited the production answers blind. They found zero numeric inventions among the 40 generative cases, and the paper reports that all 40 synthetic attempts to inject an unsupported number were intercepted. Because the production denominator is small, however, zero out of 40 still corresponds to a reported 95% upper bound of 8.81%. The guardrail also checks number provenance, not whether the system retrieved the wrong neighbouring procedure, used a stale rule or swapped one concept for another.
The user journey was less reassuring. Thirty queries triggered a clarification menu and only 14 people replied, meaning 53.33% disappeared before the next turn. All 14 who continued converged on a procedure, but counting only completers would hide the attrition. Speed was similarly uneven: the global median was 0.665 seconds, yet the 95th percentile was 30.412 seconds because generative work queued on CPU infrastructure.[1]
The distinction that matters
Three routes, three evidence standards
The system tried to match the answer method to how much source evidence it had.
Exact record
Return deterministic source text
340 of 380 production traces
Ambiguous request
Ask the citizen to choose a facet
14 of 30 users replied
Supported synthesis
Generate with numeric verification
0 inventions observed in 40 audited answers
A promising operational study with an embedded evaluator
The author was both the platform's technical architect and the paper's investigator. The paper declares no commercial or financial conflict and describes the infrastructure as state-owned and open-source, while using independent analysts for the audits. Even so, raw citizen traces were not released, so outsiders cannot reproduce the production analysis or inspect who abandoned the service. A fairness probe also passed only six of ten pairs when socioeconomic traits were embedded naturally, and the assistant gave a falsely specific answer in four of 20 tests about unindexed state branches.
For public-service teams, the evidence supports constrained routing and human review—not a claim that sovereign infrastructure makes an assistant reliable. The next study should publish privacy-preserving trace samples, test accessibility and language variation with citizens, measure whether people complete the underlying service, and compare latency, error and abandonment against a non-AI search route.[1]