Can more data fix measurement errors in AI?
No, not in these simulations. Across five model families, noisy or misassigned input features reduced predictive performance and distorted feature-importance rankings; larger samples narrowed variation but did not remove the bias.
The short version
- What happened
- The researchers ran regression and classification simulations with sample sizes of 2,000, 3,000 and 4,000, six signal structures, two non-zero error levels and five model families.
- Who it affects
- People may receive confident but distorted decisions when an influential feature is recorded inaccurately.
- What remains unproven
- All data were simulated; no clinical, employment, lending or public-service decision was evaluated.
Keep in mind
All data were simulated; no clinical, employment, lending or public-service decision was evaluated.
The three-feature designs are much simpler than operational datasets with correlation, missingness and selection effects.
Only non-differential feature error was studied, so error linked to the outcome or a protected group remains untested.
Permutation importance was the interpretation method; the results cannot automatically quantify distortion in every explanation technique.
Increasing sample size reduced simulation variability but the study did not test every possible error-correction strategy.
Source analysis · 1 linked source. Primary analysed source date: 11 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: more records made the estimates steadier, not the bad measurements truthful
Across the controlled experiments, substituting an error-burdened feature for its error-free counterpart usually lowered regression R-squared, increased mean squared error and reduced classification accuracy or F1. The direction held across multiple linear or logistic regression, support-vector machines, random forests, XGBoost and multilayer perceptrons. Increasing the simulated sample from 2,000 to 4,000 observations mostly narrowed the spread between runs; it did not eliminate the bias introduced by the faulty input.
That distinction matters for organisations accumulating large operational datasets. A large file can make an estimate look precise while the underlying measurement remains systematically wrong. If a hospital records a proxy instead of a patient's true exposure, a lender relies on a misclassified category, or an employer assigns a group average to an individual, model tuning cannot reconstruct information that was never measured correctly. The paper therefore supports data-quality audits before treating scale, architecture or hyperparameter search as a remedy.[1]
What the simulations actually tested
The authors created synthetic regression and binary-classification problems with three features. Two features generated the outcome and one was deliberately non-predictive. They then replaced one predictive feature with an error-affected version while retaining an error-free benchmark. The study examined classical additive error, binary misclassification, Berkson error—where actual values vary around assigned values—and group-summary assignment, where five group means stand in for individual values.
Sample sizes were 2,000, 3,000 and 4,000. Six coefficient settings changed whether the error-prone feature was stronger than, equal to or weaker than the other predictive feature. Classical and Berkson error used standard-deviation levels of 0.5 and 0.75; binary misclassification used probabilities of 0.1 and 0.3. Each factorial cell was repeated ten times. Data were split 80:20 for training and testing, with stratification for classification, and hyperparameters were tuned separately for each model and measurement condition using 20 Optuna trials.[1]
Why feature importance can become misleading
The researchers calculated permutation feature importance with ten permutations for every fitted model. As classical error increased, the measured importance of the affected predictor generally fell, while the clean predictive feature stayed stable and the deliberately irrelevant feature remained near zero. In practical terms, a genuinely influential factor can appear less important merely because it is measured less reliably. Rankings can then reward the easiest variable to record rather than the variable most connected to the outcome.
This is not only a technical concern. Feature rankings are often shown to clinicians, auditors, regulators and managers as explanations for model behaviour. If measurement quality differs across demographic groups, sites or service settings, an explanation can also shift without any true change in the underlying relationship. The paper evaluates permutation importance rather than every explanation method, but its core warning travels: explanation tools describe the fitted data-generating pipeline, including its errors. They do not independently verify that the inputs represent reality.[1]
The model family did not provide a universal escape
Model architecture affected the size of the degradation, and interactions between model, error level and signal structure were statistically significant in many analyses. But no model family was consistently immune across all four error mechanisms and both task types. The largest losses tended to occur when the damaged feature carried more of the outcome signal. When the clean predictor was stronger, performance declined less because the model had another informative route.
That is more useful than a simple ranking of algorithms. It suggests the correct response depends on how error enters the workflow: laboratory imprecision differs from a miscoded category, and assigning neighbourhood averages to individuals differs from both. Practitioners need repeat measurements, validation subsets, sensitivity analysis or measurement-error models suited to the mechanism. Swapping a linear model for a neural network, or vice versa, may change robustness at the margin but does not make an error-prone feature accurate.[1]
What this changes for people who build and use models
Data teams should document how each consequential feature was obtained, what unit it represents, how often it is missing or recoded, and whether a value belongs to the individual or to a group. A model card that lists only predictive performance is incomplete when the input process can distort both decisions and explanations. Validation should include plausible error ranges and should examine whether performance and feature rankings change across sites, devices and subgroups.
For clinicians, teachers, workers, borrowers and other affected people, the practical issue is contestability. A decision should not become harder to challenge because a dashboard presents a precise feature-importance chart. Organisations need a route to correct an inaccurate source record and rerun the decision. The study does not quantify harm in any of these domains, but it shows why governance must reach upstream to measurement and record creation rather than beginning only after a model has been trained.[1]
Limits and what would change the assessment
The evidence is entirely simulated. The data contain three features, normal or Bernoulli distributions, specified additive structures and a deliberately simple non-predictive variable. Real datasets have correlated predictors, missingness, selection bias, repeated observations, nonlinear causal structures and error that may depend on the outcome or on protected characteristics. Ten replications per cell supported the paper's factorial analysis, but the reported post-hoc power estimate is based on observed effects and does not replace broader replication.
The study also evaluates permutation importance rather than SHAP, counterfactual explanations or causal attribution. Reproducible code was available on request rather than linked as a public repository in the article. Confidence would rise if the patterns were reproduced on realistic benchmark datasets with known measurement processes, under correlated and differential error, and across explanation methods. Prospective studies showing that targeted remeasurement or correction improves decisions would turn a strong simulation warning into direct operational evidence.[1]