Can a cheaper AI API relay change what you get—or what you pay for? A 20-service audit found warning signs, not proof of fraud
Researchers saw answer-pattern differences, unexpected tool calls and five services reporting thousands more input tokens than the official APIs for identical requests. But the non-random, two-round audit could not see which models or wrappers ran behind each relay, and its price-group comparison did not reach conventional statistical significance.
The short version
- What happened
- The researchers selected 20 accessible relays advertising GPT-6 Astra or Claude Opus 5: ten below official prices and ten at the same or a higher price. Selection was practical rather than random, so it cannot estimate how common problems are across the relay market.
- Who it affects
- Developers and small organisations may gain access or simpler billing through a relay, but can face hidden context, unstable behaviour or effective costs that the advertised unit price does not reveal.
- What remains unproven
- The work is an unreviewed preprint with a non-random sample of 20 accessible relays and two advertised model families.
Keep in mind
The work is an unreviewed preprint with a non-random sample of 20 accessible relays and two advertised model families.
The lower-price versus same-or-higher-price comparison produced p=0.0698, above the conventional 0.05 threshold, and the two groups contained only ten services each.
Seven of 35 repeated relay-model pairs changed primary flag status between rounds, showing that a single snapshot is not a stable certification.
Output comparisons cannot identify the hidden backend model or distinguish substitution from wrappers, routing, version changes, sampling or other implementation choices.
The audit did not test retention, API-key misuse, internal gateway software, contractual compliance or representative production workloads, and no tool call was executed.
Source analysis · 2 linked sources. Primary analysed source date: 5 November 2026.
Read the full report · 11 min The evidence, methods, implications and limitations
The answer: relays can produce observable differences, but this audit cannot say why
A new Canadian preprint finds that some third-party services selling access to leading language models behaved differently from official provider APIs under matched prompts. Across a deliberately selected group of 20 relays, the researchers saw unusual answer patterns, tool calls that appeared despite an instruction to suppress tools, and reported input-token counts far above the official endpoints for the same requests. Those observations justify buyer caution and better logging. They do not prove that a relay substituted a cheaper model, inflated a bill deliberately or acted maliciously.
The boundary is important because an API relay is a middle layer. It may add a system prompt, safety wrapper, translation, retry policy, cache, routing rule or its own token-accounting convention before and after a request reaches a model provider. A user sees the advertised model name and the returned response, but ordinarily cannot inspect those internal steps. The experiment compares what entered and left the public interface; it does not identify the actual backend model or reveal the operator's intent.
The paper is also an unreviewed preprint rather than a peer-reviewed market survey. Its 20 services were chosen because researchers could find, access and pay for them after consulting model recommendations, community lists and web search. Ten advertised prices below official rates and ten charged the same or more. That construction creates an informative stress test, not a probability sample. A flagged share such as seven out of ten must not be read as the prevalence of unreliable low-cost relays worldwide.[1][2]
How the researchers compared answers without claiming to know the hidden model
The team targeted services offering GPT-6 Astra or Claude Opus 5 and built prompt families intended to reveal stable behavioural differences rather than simply ask whether an output looked good. Tests covered self-description, factual and reasoning responses, coding tasks, instruction following and repeated setup queries. Official provider APIs supplied reference behaviour. The researchers then looked for response patterns outside the reference envelope, while acknowledging that sampling variation, undocumented provider updates and relay-side prompting can all move an answer without model substitution.
In the first round, seven of the ten below-official-price relays and two of the ten same-or-higher-price relays had at least one primary flag. A two-sided Fisher exact test produced p=0.0698. That result is compatible with a meaningful association, but it does not cross the conventional 0.05 threshold and the tiny groups produce wide uncertainty. Price may be a useful screening signal; this study does not establish that a cheap relay is more likely to misrepresent its service.
Repeat measurement exposed another limit. The first round covered 36 available relay-model pairs. Eleven to 18 days later, the researchers repeated 35 pairs. Six services remained flagged across both rounds, but seven of the 35 relay-model pairs gained or lost a primary flag. A service can change its routing, prompts or upstream access; an official model can also change. A one-day certification would therefore be weak. Buyers need ongoing canary tests and change records rather than a permanent trust label based on one snapshot.[1][2]
The distinction that matters
What the relay audit observed—and what remained hidden
The experiment can compare the public interface with official endpoints, but it cannot see the relay's internal route or intent.
Answer tests
Compare repeated response patterns
Observable difference, not model identity
Tool tests
Suppress tools and inspect returned objects
Four transient calls; none executed
Usage tests
Send identical requests and compare meters
Five relays reported extra input tokens
Balance checks
Reconcile six accessible accounts
Charges matched each relay's own meter
Unexpected tool calls appeared, but none executed and none repeated
The study screened 3,332 code and setup responses across the two rounds for prompt or content injection and for tool behaviour. It confirmed no prompt or content injection in that corpus. In the first round, four relays nevertheless returned tool-call structures even though the test explicitly suppressed tool use. Those calls were not executed, and none of the four behaviours recurred in the second round. The finding is therefore a transient interface warning, not evidence that a tool harmed a system or that a relay conducted an attack.
For developers, the practical consequence is still serious. Tool permissions should be enforced by the application that can execute a call, not trusted to a natural-language instruction sent through an intermediary. A client can reject any tool-call object when tools are disabled, allow-list names and arguments when they are enabled, require confirmation for consequential actions and run tools with the least privilege available. A relay response should be treated as untrusted input even when it claims to come from a familiar model.
This distinction prevents two opposite errors. It avoids exaggerating four non-executed, non-repeated outputs into a successful breach, and it avoids assuming a provider-side 'no tools' setting is a complete security boundary. The paper tested public responses, not the relay's data retention, access to submitted API keys, internal gateway software or handling of sensitive prompts. Organisations sending health, legal, customer or source material need contractual and technical assurance on those issues separately.[1][2]
The token discrepancy is measurable; its cause is not
Five relays reported between 4,200 and 60,000 more input tokens than official APIs for identical requests. That is the paper's clearest operational discrepancy because token usage can affect cost and capacity planning. Yet reported tokens are not a direct view into the backend. A relay may prepend a long system message, include prior context, retry internally, count cached material differently or use a tokenizer and accounting boundary that differs from the official provider. The experiment could not observe which explanation applied.
The researchers could inspect account balances for six services. In those cases, the balance changes matched the usage reported by the relay and its posted price to the cent, including one service with extra reported input tokens. That consistency shows that the service's billing followed its displayed meter; it does not validate the meter's relationship to the official provider or prove intentional overcharging. A customer can be billed exactly according to an opaque rule that is still unsuitable for comparison shopping.
Procurement should therefore request a written definition of billable input: whether it includes relay instructions, retries, caches, moderation passes or hidden context; which tokenizer applies; and how disputes are resolved. Technical teams can send fixed canary prompts to the official endpoint and the relay, save raw request and response metadata, compare reported tokens and calculate effective cost per accepted task. The relevant outcome is not merely the advertised price per million tokens but the total price, latency and error burden for a verified result.[1][2]
What the design can and cannot establish
The design is valuable because it combines several observable layers—answers, tool structures and billed usage—and repeats the experiment rather than treating one anomaly as permanent. It also anonymises the services as R01 and similar labels, which keeps the paper focused on method rather than making an accusation the data cannot support. But anonymity means a buyer cannot use the paper as a shopping list, and the authors do not supply an independent audit of each operator's infrastructure or contracts.
The sample is small, non-random and limited to two advertised frontier models. Service availability and payment access shaped inclusion. The prompts are controlled probes rather than a representative mixture of production workloads, languages, regions and long conversations. Official API behaviour is a reference, not a biological fingerprint: two legitimate deployments of the same family can differ because of versions, system prompts and sampling. Multiple flags and repeated consistency strengthen concern, but no output-only test can conclusively identify a hidden model in every setting.
The accessible manuscript does not provide a funding acknowledgement. The authors list Queen's University in Kingston, Ontario, and the paper does not report a competing-interest statement in the sections reviewed. Absence of a visible declaration is not evidence that no support or interest exists. Peer review, a fuller disclosure and independent replication would strengthen confidence in both the methods and the interpretation.[1][2]
What this means for teams buying model access
A relay can be useful. It may simplify payment, provide regional access, aggregate models or offer resilience when official services are unavailable. The study does not show that all relays are unsafe or that lower prices are illegitimate. It shows that the advertised model label and unit price are not enough evidence for a consequential workload. Teams should decide what they need to verify: model provenance, retention, geographic processing, tool controls, uptime, effective price, or answer consistency. Different uses carry different thresholds.
For low-risk experimentation, transparent logging and a spending cap may be proportionate. For code deployment, financial decisions, patient data, public services or confidential reporting, the organisation should require a named contracting entity, data-processing terms, deletion and retention rules, incident reporting, subprocessor disclosure and a way to test or attest the upstream route. Client-side policy must remain authoritative when a returned object requests a tool or action. Secrets should be scoped and rotated, and the relay should not receive broader credentials than the task requires.
Confidence in the market-level claim would rise with a preregistered, randomly sampled international audit across more models, languages and workload types; repeated testing over months; and independent access to relay logs or cryptographic provider attestations. It would rise further if billing definitions were tested against disclosed wrapper tokens and retries. The assessment would change sharply if named services and upstream providers jointly verified routing and accounting, or if controlled evidence showed deliberate substitution. Until then, the responsible conclusion is narrower: public-interface tests can reveal discrepancies worth investigating, but cannot by themselves tell a buyer exactly what happened behind the middleman.[1][2]