Can an AI assistant reveal a private condition without naming it? A synthetic benchmark found indirect clues survived privacy prompts
Across five models and 950 synthetic persona-task instances, privacy-aware instructions sharply reduced explicit disclosure but left attributes inferable in 16.2% to 37.4% of top-three tests. The benchmark used simulated data and an LLM recipient, not real communications.
The short version
- What happened
- InferPriv combines 10 synthetic personas, 10 sensitive attributes and 10 routine tasks into 950 instances, with five sampled replies per protected persona-attribute-task cell.
- Who it affects
- People using assistants for work, travel or bookings may disclose sensitive facts through helpful details they did not choose to share, even when the assistant avoids naming the fact.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Synthetic personas, memories and tasks
One LLM acts as recipient and utility judge
Small English-language scenario set
Worst-case any-of-five and top-three metrics
No real users, recipients or measured harm
Source analysis · 1 linked source. Primary analysed source date: 25 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark looks for clues, not only secret words
An unreviewed US preprint tests a privacy failure that ordinary keyword checks can miss: an assistant may avoid saying that a user has diabetes yet request a refrigerator for insulin, allowing a hotel coordinator to infer the condition. InferPriv pairs 10 synthetic personas with 10 sensitive attributes—including health conditions, religion and financial distress—and 10 routine coordination tasks. The final set contains 950 instances; implausible persona-attribute combinations are excluded.
Each persona has 27 ordinary memory statements plus three records describing one sensitive attribute. The researchers identified 98 attribute-task pairs where the sensitive records were unnecessary to complete the request. Five frontier models then drafted five replies for each protected cell. A separate language model, shown only the outgoing message, made up to three guesses about the writer. A control run without the sensitive records was subtracted to reduce chance guessing and background confounding.[1]
Privacy instructions stopped names more readily than implications
With a privacy-aware prompt, explicit disclosure across five replies fell to 0% for GPT-6.1-Sol and Gemini-3.8-Flash and 0.9% for Sonnet-5.5. Yet the correct attribute still appeared among the simulated recipient’s top three guesses in 25.2%, 16.3% and 16.2% of adjusted protected cells respectively. DeepSeek-V4.1-Flash and Muse-Spark-1.3 retained higher explicit and inferential rates. Task-completeness scores stayed between 90.2% and 96.2% under the privacy prompt.
The metric is deliberately conservative from a privacy perspective: if any of five sampled replies leaks, the cell counts as leaking. Top-three guessing gives the recipient several chances, so the top-one results are also important. They ranged from 13.0% to 33.4% under the privacy prompt. Increasing reasoning effort for one model lowered explicit disclosure but did not remove indirect leakage, and removing the sensitive attribute’s name from memory records barely changed its privacy-aware result.[1]
The distinction that matters
Direct versus inferential disclosure
A reply can keep the attribute unnamed while still communicating enough clues for someone to infer it.
Direct
Reply names diabetes
Keyword disclosure is visible
Inferential
Reply requests insulin storage
Condition may be guessed
Safer path
Minimize context and ask before sending
Needs real-user validation
This is a synthetic stress test, not observed surveillance
No people sent these messages and no real recipient inferred a health condition, religion or financial status. Personas, memories and tasks were synthetic; the same model family acted as the curious recipient and the task-completeness judge. The test covers a small set of English-language coordination scenarios, and its lexical matching may miss paraphrases or count broad guesses differently from a human. The paper does not state funding or competing-interest disclosures.
Even with those limits, the design isolates a practical distinction for product teams: direct disclosure and inferential disclosure are different tests. Before an assistant sends a message to an employer, hotel or service provider, it may need to minimize which memories enter the drafting context, express requirements without revealing their cause and ask the user before transmitting a revealing accommodation. Real-user studies, human-recipient evaluations and repeated tests across languages and tasks would show whether this benchmark predicts actual privacy loss.[1]