Can seizure-warning AI survive a clean patient split?
Five EEG pipelines fell close to the 0.5 AUC reference when tested on unseen patients. Randomly splitting overlapping windows made one baseline look much stronger, showing why a promising benchmark is not yet a usable seizure alarm.
The short version
- What happened
- Five fixed pipelines were evaluated on 17,424 CHB-MIT EEG windows from 22 independent pediatric subject groups using 20 train-validation-test partitions that held out whole people.
- Who it affects
- A seizure warning must catch enough events without exhausting a person with false alarms. Inflated window-level scores can move a model towards trials or procurement before anyone knows whether its thresholds transfer to a new patient or remain useful over time.
- What remains unproven
- Unreviewed, non-preregistered retrospective analysis of public datasets.
Keep in mind
Unreviewed, non-preregistered retrospective analysis of public datasets.
The chronological control was post hoc and evaluated only PSD+LDA, not the four deep models.
CHB-MIT contained 22 independent pediatric groups; external testing covered 13 Siena participants and only the spectral baseline.
The study measured selected EEG windows, not continuous alarms, seizure-level sensitivity, warning time or patient outcomes.
AUC 0.5 is a reference point, not proof that a model is statistically equivalent to chance in every population.
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
Holding out whole people changed the headline result
A New Zealand preprint found that five seizure-prediction pipelines that can look promising under familiar evaluation schemes performed close to the 0.5 ROC reference when tested on people they had never seen. The researchers used 17,424 twenty-second EEG windows from the CHB-MIT database. Its 23 case folders represented 22 independent pediatric subject groups; 69 of 182 annotated seizures contributed preictal windows. For each of 20 fixed partitions, 14 groups trained the models, four set checkpoints and thresholds, and four were held out for testing.
The five pipelines ranged from a 90-feature spectral classifier to an EEG transformer with more than 850,000 parameters. Mean pooled test AUC ranged only from 0.527 to 0.553. Giving each of the 22 subject groups equal weight moved the range to 0.502–0.523. A single pipeline's standard deviation across partitions was 0.06–0.08—larger than the 0.026 spread between model means. The study therefore does not support ranking one architecture above another for cold-start use on an unseen patient.[1]
Random windows shared more information than a future alarm would have
The team traced every window back to its recording and seizure. In all ten cases eligible for the original index-split audit, at least one seizure contributed windows to more than one partition and raw EEG samples overlapped across a boundary. A stricter control kept complete preictal blocks together and ordered them in time. Across 11 eligible cases and 11,175 windows, the fixed PSD-plus-LDA baseline averaged AUC 0.572 under chronological blocks, compared with 0.815 when the same windows and class counts were randomly allocated. Random allocation scored higher in every case.
Threshold transfer exposed another practical gap. A validation rule designed to label at most 1.67% of interictal windows positive produced mean test false-positive proportions of 1.81–3.13% on new people and exceeded its target in six to ten of 20 partitions, depending on the model. At those strict thresholds, only 2.33–5.56% of preictal windows were detected. A zero-shot external check of the spectral baseline on 5,657 windows from 13 Siena participants produced pooled AUC 0.480. The four deep models were not tested externally.[1]
The distinction that matters
The same model, three different questions
Changing the test boundary changes what a reported score means, even when the pipeline stays fixed.
Unseen people
Hold out four of 22 subject groups
Five pipelines: mean pooled AUC 0.527–0.553
Within one person
Keep seizure blocks chronological
PSD+LDA mean AUC 0.572 across 11 cases
Random windows
Shuffle the same windows and class counts
PSD+LDA mean AUC 0.815, with related time information mixed
Window classification is still several steps from a warning device
The paper is a retrospective computational audit, not a prospective test of continuous alarms. Sampled-window false positives are not false alarms per hour, and the analysis did not measure seizure-level sensitivity, time in warning, alarm persistence or how a patient would act. The chronological control was added after the original results were inspected, was not preregistered and used only the simple spectral baseline. CHB-MIT is a pediatric dataset collected at one centre, while Siena is a separate small public dataset; neither represents the full range of people living with epilepsy.
For patients, the result is a warning about evidence quality rather than proof that seizure forecasting is impossible. Developers should keep people, seizure episodes and overlapping raw samples on one side of the train-test boundary, freeze thresholds before external testing and report continuous-alarm outcomes. Regulators and hospitals should ask what information crossed each split before treating an AUC as clinically meaningful. Auckland University of Technology supported the work; the authors declared no competing interests and released code, partitions, window manifests and predictions for comparison.[1]