Automating intended target identification for paraphasias in discourse using a large language model

Purpose To date there are no automated tools for the identification and fine-grained classification of paraphasias within discourse, the production of which is the hallmark characteristic of most people with aphasia (PWA). In this work we fine-tune a large language model (LLM) to automatically predict paraphasia targets in Cinderella story retellings. Method Data consisted of 353 Cinderella story retellings containing 2,489 paraphasias from PWA, for which research assistants identified their intended targets. We supplemented this training data with 256 sessions from control participants, to which we added 2,427 synthetic paraphasias. We conducted four experiments using different training data configurations to fine-tune the LLM to automatically “fill in the blank” of the paraphasia with a predicted target, given the context of the rest of the story retelling. We tested the experiments’ predictions against our human-identified targets and stratified our results by ambiguity of the targets and clinical factors. Results The model trained on controls and PWA achieved 46.8% accuracy at exactly matching the human-identified target. Fine-tuning on PWA data, with or without controls, led to comparable performance. The model performed better on targets with less human ambiguity, and on paraphasias from participants with less severe or fluent aphasia. Conclusion We were able to automatically identify the intended target of paraphasias in discourse using just the surrounding language about half of the time. These findings take us a step closer to automatic aphasic discourse analysis. In future work, we will incorporate phonological information from the paraphasia to further improve predictive utility.