Evaluating the Generalizability of EEG-Based AI Models in Alzheimer’s and Dementia Diagnosis

INTRODUCTION We thoroughly investigated the generalizability of deep learning models trained on electroencephalography (EEG) data to detect Alzheimer’s disease and dementia at the individual subject level. Although average model performance appears strong, it may obscure large inter-individual variability, raising concerns for clinical deployment.

We trained a Hopfield-enhanced deep neural network on a publicly available EEG dataset consisting of 88 participants, including individuals diagnosed with Alzheimer’s disease (AD), frontotemporal dementia (FTD), and cognitively normal controls (CN). Resting-state EEG recordings were segmented and used to train the model in a leave-onesubject-out (LOSO) cross-validation setup across multiple detection tasks: AD vs. CN, AD vs. FTD, FTD vs. CN, and AD vs. FTD vs. CN.

While the model demonstrated high average performance (e.g., up to 83% accuracy), subject-level results revealed inconsistencies. Some individuals achieved perfect prediction even at the first training epoch, suggesting spurious memorization, while others predicted falsely throughout, with performance below chance. These patterns persisted despite consistent training conditions and no data leakage.

Our findings highlight that strong group-level performance may be misleading in clinical settings, where decisions are made at the individual level. The models should be generalizable across individuals and be evaluated per individual before being considered for diagnostic use. Hopfield networks show promise in capturing patterns in EEG data, but patient-level validation and transparent reporting are essential to avoid premature clinical translation. Highlights Deep learning models trained on EEG can achieve high average performance in detecting Alzheimer’s and dementia disease. The subject-level evaluation revealed significant variability, including below-chance performance for some indi- viduals despite overall strong results. Group-level metrics alone may be misleading; General- izable models and individual-level validation are critical before the clinical adoption of EEG-based AI models. Competing Interest Statement The authors have declared no competing interest. Funding Statement Promobilia Foundation Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The datasets are already public on the OpenNeuro platform, and the dataset authors have already published a paper on that dataset; we have cited it. https://openneuro.org/datasets/ds004504/versions/1.0.8 I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes contributing authors. Funding information The Promobilia Foundation, Grant/Award Number: A23110 Data Availability https://openneuro.org/datasets/ds004504/versions/1.0.7 Andreas Miltiadous, Katerina D. Tzimourta, Theodora Afrantou, Panagiotis Ioannidis, Nikolaos Grigoriadis, Dimitrios G. Tsalikakis, Pantelis Angelidis, Markos G. Tsipouras, Evripidis Glavas, Nikolaos Giannakeas, and Alexandros T. Tzallas (2024). A dataset of EEG recordings from: Alzheimer's disease, Frontotemporal dementia and Healthy subjects. OpenNeuro. [Dataset] doi: doi:10.18112/openneuro.ds004504.v1.0.8 Abbreviations - EEG - Electroencephalography - LOSO - Leave-one-subject-out - AD - Alzheimer’s - CN - Healthy controls - FTD - Frontotemporal dementia - AI - Artificial Intelligence - ML - Machine Learning.