A stock model scored 63.84%—a calendar rule did better
A peer-reviewed audit found that forward-filled non-trading days created an easy target in S&P 500 data. SHAP then explained the artefact faithfully, while performance on genuine next-day moves fell below a majority baseline.
The short version
- What happened
- The public file contained 1,825 calendar-day rows from November 2015 to November 2020; 567 rows, or 31.1%, were exact forward-filled copies of the previous trading session.
- Who it affects
- Workers and savers can be affected when an apparently accurate model informs trading, risk limits or product marketing. Independent data audits and simple baselines are safeguards against allocating money on a shortcut that disappears in live markets.
- What remains unproven
- One public S&P 500 file; 2015–2020 only; six walk-forward folds; no other asset, geography, learner or post-2020 market; historical backtest rather than prospective deployment.
Keep in mind
One public S&P 500 file; 2015–2020 only; six walk-forward folds; no other asset, geography, learner or post-2020 market; historical backtest rather than prospective deployment.
Source analysis · 1 linked source. Primary analysed source date: 17 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A copied-price artefact created a predictable label
Researchers audited a public S&P 500 daily-price file used to train an XGBoost model for next-day market direction. The raw file covered 1,825 calendar days from 23 November 2015 to 20 November 2020. It included weekends and market holidays, with 567 rows—31.1%—created by copying the previous trading session's prices and volume. Because the target was defined as whether the next close was strictly higher, those repeated rows automatically became 'Down'. A calendar-only rule therefore reached 71.19% test accuracy, beating the model's reported 63.84% and the 59.32% majority baseline.[1]
The score disappeared when the question became a real trading-day forecast
After feature construction, the original experiment used 1,766 rows and 32 engineered features, split sequentially into 1,412 training and 354 test observations. Among the test observations, 108 had a zero prior return created by the copied rows. On the remaining 246 genuine next-day moves, accuracy was 49.19%, below the 58.54% majority baseline. Removing non-trading rows left 1,199 genuine sessions. A six-fold walk-forward audit then produced pooled AUC of 0.476 with expanding training windows and 0.460 with rolling windows—near or below chance—and accuracy below the corresponding majority baselines.
The paper also tested a simple long-or-cash strategy with ten-basis-point trading costs. Its model strategy had an annualised return of minus 11.59%, a Sharpe ratio of minus 0.241 and maximum drawdown of minus 34.7%, while buy-and-hold returned 15.50% annualised with a 0.588 Sharpe ratio. That backtest is historical rather than investable evidence, but it reinforces the central finding: apparent label accuracy did not translate into a useful trading signal.[1]
The distinction that matters
How a calendar artefact became an explainable market signal
The model learned a real pattern in the file, but the pattern was created by data preparation rather than price discovery.
Input
Forward-fill weekends and holidays
567 copied rows; 31.1% of the raw file
Target
Label only strictly higher next closes as Up
Copied prices become automatically Down
Audit
Remove copied rows and walk forward
AUC 0.476 expanding; 0.460 rolling
An explanation can be faithful to the wrong problem
SHAP highlighted the zero-return feature that encoded copied calendar rows, so the explanation was faithful to what XGBoost had learned. The audit also found strongly correlated volatility features, with correlations up to 0.947, and showed that adding individual SHAP values across that cluster overstated the group's contribution. Dropping 14 of 32 features did not reduce predictive performance. Explainability therefore helped reveal the shortcut, but a plausible attribution chart could not repair a flawed target or duplicated observations.
The study is a forensic re-analysis of one US index file from 2015–2020, not evidence that every financial model leaks or that markets are inherently unpredictable. It did not test other assets, regions, learners or the 2021–2025 market. The practical lesson for analysts, vendors and investment committees is narrower: inspect timestamps and duplicate rows before modelling; compare against calendar and majority rules; use chronological holdouts; and report economic performance after costs. The assessment would change if a locked pipeline retained value across later periods, different markets and realistic execution assumptions.[1]