Can artificial intelligence diagnose seizures based on patients’ descriptions? A study of GPT-4

Introduction Generalist large language models (LLMs) have shown diagnostic potential in various medical contexts. However, there has been little work on this topic in relation to epilepsy. This paper aims to test the performance of an LLM (OpenAI’s GPT-4) on the differential diagnosis of epileptic and functional/dissociative seizures (FDS) based on patients’ descriptions.

GPT-4 was asked to diagnose 41 cases of epilepsy (n=16) or FDS (n=25) based on transcripts of patients describing their symptoms. It was first asked to perform this task without being given any additional training examples (‘zero-shot’) before being asked to perform it having been given one, two, and three examples of each condition (one-, two, and three-shot). As a benchmark, three experienced neurologists were also asked to perform this task without access to any additional clinical information.

In the zero-shot condition, GPT-4’s average balanced accuracy was 57% (κ: .15). Balanced accuracy improved in the one-shot condition (64%, κ: .27), though did not improve any further in the two-shot (62%, κ: .24) or three-shot (62%, κ: .23) conditions. Performance in all four conditions was worse than the average balanced accuracy of the experienced neurologists (71%, κ: .41). Significance Although its ‘raw’ performance was poor, GPT-4 showed noticeable improvement having been given just one example of a patient describing epilepsy and FDS. Giving two and three examples did not further improve performance, but more elaborate approaches (e.g. more refined prompt engineering, fine-tuning, or retrieval augmented generation) could unlock the full diagnostic potential of LLMs. Competing Interest Statement The authors have declared no competing interest. Funding Statement This research project was funded by Epilepsy Research UK (grant number 160296-1). Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The Leicester South Ethics Committee reviewed and granted ethical permission for this research (REC reference: 20/EM/0106). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability Data are not available for sharing.