Could a privacy audit falsely accuse a correct AI system?
An unreviewed statistical study found that choosing the best attack threshold on the same data pushed a nominal 5% false-alarm rate as high as 19.4% in controlled simulations. Recomputing six prior audit setups with valid corrections cut reported lower bounds by 5% to 93%.
The short version
- What happened
- The authors simulated 10,000 audits in each of 12 Gaussian-mechanism cells, spanning four privacy parameters and 100, 1,000 or 10,000 samples per hypothesis.
- Who it affects
- A false privacy violation can waste investigation time or mislead a regulator; an overconfident audit can also obscure how much uncertainty remains. Teams handling health, financial or public-service data need the threshold-selection rule documented before treating an empirical bound as certified evidence.
- What remains unproven
- The paper is an unreviewed preprint and applies established statistical tools rather than proving a new correction theorem.
Keep in mind
The paper is an unreviewed preprint and applies established statistical tools rather than proving a new correction theorem.
The known-truth coverage result uses an idealised Gaussian mechanism with an optimal attack.
For real-model audits the true leakage parameter is unknown, so the paper measures how bounds change, not their empirical coverage.
All tested valid methods are conservative, and no single method dominated every protocol.
The reanalysis covers selected configurations from six prior works, not every privacy-audit implementation.
Funding includes public programmes, CIFAR and resources partly provided by companies sponsoring the Vector Institute; no specific commercial sponsor is tied to the reported conclusion in the accessible paper.
Source analysis · 1 linked source. Primary analysed source date: 19 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The problem begins when an auditor looks before choosing
A membership-inference audit tries several score thresholds to distinguish whether a record was in training data. A confidence interval is valid when its target is fixed independently of the observations. If the auditor inspects the same scores, picks the threshold that yields the largest privacy-leakage lower bound and then reports the original confidence level, that selection is hidden from the calculation. The paper adapts established multiple-comparison tools—fixed thresholds, union bounds, Dvoretzky–Kiefer–Wolfowitz bounds, holdout splits and K-fold selection—to keep the claimed coverage valid.[1]
Known-truth simulations make the false-alarm rate measurable
For Gaussian mechanisms where the true privacy parameter is known, the authors ran 10,000 audits per cell at four parameter values and three sample sizes. At 95% confidence, the invalid maximum-threshold procedure exceeded the 5% nominal error rate in nearly every cell with at least 1,000 samples. Its worst cell reached 19.36% miscoverage at 10,000 samples per hypothesis. Every tested valid alternative stayed below 5%. More samples tightened estimates but did not repair the unaccounted-for selection; in this setup, the false-confidence problem became more visible.[1]
The distinction that matters
Where selection bias enters a privacy audit
The same attack scores should not both select the winning threshold and justify its untouched confidence claim.
Score
Run membership-inference attacks
Produce error rates across thresholds
Select
Choose the strongest-looking threshold
Creates a data-dependent comparison
Correct
Use a prespecified, simultaneous or split-data method
Restore valid coverage
Reanalysed benchmarks changed substantially, but conservatism matters
The researchers reproduced six prior audit configurations covering Fashion-MNIST, CIFAR-10 and a housing dataset; multi-run, one-run, white-box and black-box settings; and up to 5,000 runs or canaries in selected experiments. Corrected lower bounds were 5% to 93% smaller than the originally selected values, although one already-valid holdout method improved by up to 40% with K-fold reuse. The authors stress that valid methods can be conservative: a smaller certified bound is not proof that a system is more private. It is a more defensible statement about what the audit can establish.[1]