Automated Detection Of Clinical High Risk Population Of Schizophrenia: Assessing The Generalizability Of NLP And LLM-Based Methods

Automated Detection Of Clinical High Risk Population Of Schizophrenia: Assessing The Generalizability Of NLP And LLM-Based Methods | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automated Detection Of Clinical High Risk Population Of Schizophrenia: Assessing The Generalizability Of NLP And LLM-Based Methods Jiaee Cheong, Cheryl M. Corcoran, Kathryn E. Lewandowski, Ofer Pasternak, and 26 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8777643/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract Background and Hypothesis : Research has indicated that linguistic features can be used for the early detection of schizophrenia. Given that traditional clinician-based assessment can be labor intensive and time-consuming, more research has turned towards the usage of automated means to extract and analyze the linguistic features of schizophrenia. However, most of these existing studies have chiefly focused on deploying the LLMs with no comparison against more well-established NLP-based methods. As a result, there is less insight on the utility of using LLMs and whether the benefits of using of LLMs for analysis outweighs the costs. Moreover, given LLM’s prompt sensitivity, there is also a lack of research investigating how different prompt engineering methods affect the different model’s output across different settings. Another longstanding open question within the field pertains to how best to objectively assess prodromal psychotic symptoms and how best to analyze the different transcripts. In this study, we systematically assess the efficacy of large language models (LLMs) and natural-language processing (NLP) methods to perform automated linguistic analysis of clinical high risk (CHR) psychotic symptoms. We seek to understand the reliability of using LLMs to analyze patient transcripts for the early identification of CHR individuals in comparison against more established NLP-based methods. Study Design : We trained models using a large international dataset of 374 patients, of which 331 are clinically high risk (CHR) and 43 are community controls (CC). Two types of interviews were conducted: an open-ended and a semi-structured interview based on the Positive SYmptoms and Diagnostic Criteria for the CAARMS [73] Harmonized with the SIPS [74] (PSYCHS) protocol [32]. Trained research assistants carried out these interviews which were audio and video-recorded across different sites prior to October 13 2024. We used two different feature extraction methods, the principal component analysis (PCA) and feature selection (FS), and conducted experiments using four different machine learning (ML) models and two large language models (LLMs), namely Llama and Qwen. For each of the LLM, we used three different prompting strategies: a neutral prompt, an NLP based prompt and a PSYCHS interview-based prompt to better understand each LLM performance under different reasoning setting. Results : Across both the open and PSYCHS-based transcripts, the NLP combined with ML-based methods, which relies on objective quantifiable metrics, demonstrated fairly consistent results within a range of 0.60 − 0.90. This is in contrast to LLM-based methods, which provided highly variable results depending on the interview format and prompt used, with the lowest being 0.320 and the highest being 0.880 across all experimental settings. In general, both categories of methods seem to produce more accurate results using the PSYCHS-based transcripts. Llama generally performs better than text-based methods, which require semantic reasoning (e.g. the PSYCHS based prompt), and yielded the highest accuracy and F1 of 0.880 and 0.930 when used on the PSYCHS-based interview transcripts. On the other hand, Qwen generally performed better than numerical-reasoning based tasks (e.g. the NLP-based prompt) and performed the best across the PSYCHS-based interview transcripts with an accuracy and F1 of 0.880 and 0.930. Conclusions : Overall, we find that NLP-based methods are more reliable and consistent. LLM-based methods are highly variable and do not demonstrate sufficient reliability. Their output differs greatly depending on the input transcript and prompt type provided. We suggest that more emphasis should be placed on developing interpretable and clinically grounded methods to automate linguistic analysis of schizophrenia. Further experiments need to be conducted before deploying such models for high-stakes use cases and for identifying more precise and automated methods to understand how clinical features of schizophrenia are expressed linguistically. automated linguistic analysis clinical high risk natural language processing large language models Figures Figure 1 1. INTRODUCTION Abnormalities in language and speech have been established as one of the early characteristics in individuals with subclinical psychotic symptoms [ 1 , 2 , 3 , 4 ]. Affected individuals may display an atypical linguistic or speech-based profile such as reduced word count, poverty of content [ 3 , 5 , 6 ] and reduced sentiment valence-arousal [ 7 ] that are indicative of core symptoms in schizophrenia (SZ), such as alogia and flat affect [ 8 , 9 ]. Several studies further suggest that language features could be used as biomarkers for early detection of schizophrenia [ 10 , 11 , 12 ] which would allow timely intervention and management [ 13 ]. Earlier efforts in linguistic analysis of SZ symptoms typically relied on manual clinician-based assessment which is considered less scalable and more labor-intensive [ 14 , 15 ]. As a result, there has been a growing body of research using automated methods such as natural language processing (NLP) to extract and analyze relevant features [ 16 , 17 , 18 , 19 , 15 ]. Past studies focusing on automated methods have typically leveraged NLP-based methods for automated linguistic feature extraction and data analysis [ 4 , 16 , 17 ]. This line of work has proven fruitful. For instance, NLP-based methods have successfully differentiated between patients with schizophrenia and a community control group by utilizing semantic and syntactic features with accuracy exceeding 80% [ 16 , 17 ]. Moreover, NLP-derived linguistic measures have also shown potential for identifying prodromal psychotic symptoms and for predicting later transition to psychosis among of clinical high-risk (CHR) individuals [ 18 , 19 , 15 ]. In more recent years, large language models (LLMs) have been increasingly used across a range of healthcare settings such as radiology [ 20 ], ophthalmology [ 21 ] and psychiatry [ 22 ]. Recent studies have also demonstrated their efficacy across symptom prediction [ 23 ], crisis detection [ 24 ] and health coaching [ 25 ]. Within the context of biomarker analysis, studies have shown that the use of LLMs have provided new methods to non-invasively analyze language disturbances and the classification of diagnoses [ 26 , 27 , 28 , 29 ]. Within the context of neuropsychiatric disorders, research has indicated that contemporary generative LLMs can be used to differentiate among the three major primary progressive aphasias (PPA) variants from a speech sample at a level consistent with that of expert clinicians [ 27 ]. Further, Liu et al. have investigated its utility in evaluating negative symptoms in schizophrenia [ 28 ]. However, most of these existing studies have chiefly focused on deploying the LLMs with no comparison against more well-established NLP-based methods (Gap 1). As a result, there is less insight on the utility of using LLMs and whether the benefits of using LLMs for analysis outweighs the costs. Moreover, given LLM’s prompt sensitivity [ 30 ], there is also a lack of research investigating how different prompt engineering methods affect the different model’s output across different settings (Gap 2). Another long-standing open question within the field pertains to how best to objectively assess prodromal psychotic symptoms and how best to analyze different types of transcripts (Gap 3) [ 31 , 32 , 33 ]. A range of transcript types have been used within the context of automated linguistic analysis in schizophrenia, ranging from open-ended interviews with conversational opening questions [ 34 , 35 ], standardized or structured clinical screening interviews [ 32 , 36 ], as well as task-based cognitive assessment transcripts [ 37 , 38 , 39 ]. Given the proliferation of using LLMs to assess and predict mental health [ 40 , 41 ], we build on a growing body of work investigating how LLMs can be used to automatically assess SZ symptoms [ 28 ] and attempt to address the research gaps outlined above. Our key contributions and the main research questions (RQs) that we attempt to address are outlined as follow: • RQ 1 : How do LLMs compare against well-establish NLP methods across CHR symptom analysis and identification? RQ 2 : How does the ML performance differ across the different feature extraction methods and how does the LLM performance differ across different prompt-engineering methods ? RQ 3 : How generalizable are the findings across different interview formats i.e. the open-ended interviews vs. the PSYCHS-based semi-structured clinical screening interviews ? Due to the ethical concerns of existing AI research for mental health and growing calls to work with open-source models [ 42 , 43 , 44 ], we worked with open-source models and sought to compare LLMs’ performance against more established NLP-based methods. Moreover, we also contribute to a growing body of work assessing the generalizability and clinical utility of using LLMs for automated assessment and diagnosis [ 45 , 46 ]. By addressing the aforementioned RQs, we assist clinicians and researchers in deciding how best to leverage NLP or LLM-based tools for better symptomatology understanding, diagnosis and analysis. Figure 1. Flowchart for the entire data processing and analyzes pipeline . Across Experiment Setting 1: NLP + ML Experiments, first, we extract NLP features across both the open and PSYCHS-based interview transcripts (A. Input Transcript) using Stanza (B. NLP Feature Extraction). Subsequently, we perform data pre-processing via principal component analysis (PCA) and feature selection (FS) (Stage C. Data Pre-processing). We then perform ML classification (Stage D. ML methods) across both sets of features. Across Experiment Setting 2: LLM Experiments, we feed the open and PSYCHS-based transcripts directly to Prompt 1: Neutral and Prompt 3: PSYCHS-based (Stage E. Prompt Types). We use the NLP features selected via feature selection as the input to Prompt 2: NLP-based. We then perform LLM analysis and classification (Stage F. LLM used) across all 3 sets of prompt engineering outputs. 2. METHODS 2.1. Participants We used language samples collected via the Accelerating Medicines Partnership for Schizophrenia (AMP-SCZ) project [ 47 ]. We used version 3’s release where 2192 samples were provided as part of this release in total. Although three kinds of speech samples were collected: open-ended language samples, PSYCHS language samples and daily audio diaries, we only used the open-ended language samples and PSYCHS language samples within this analysis. Given that not all participants provided interview transcripts, we only worked with a subset of participants who provided at least one interview transcript. This resulted in 843 samples. Table 1 provides the demographic breakdown of the analyzed samples. Out of the total 843 samples, 513 were female and 330 were males. We see that the subsets of CHR vs. CC participants are broadly comparable across sociodemographic features. CHR participants are those that meet the diagnostic criteria as determined using the Positive Symptoms and Diagnostic Criteria for the CAARMS [ 73 ] and Harmonized with the SIPS [ 74 ] (PSYCHS) protocol [ 32 ]. Community control (CC) participants were recruited from the community. CHR participants completed screening, baseline assessments and a series of follow-up assessments over a period of 24 months. CC participants completed screening and baseline assessments and a subset (5 per site) completed month 2, 12 and 24 visits. All samples were collected in separate sessions. Inclusion and exclusion criteria were aligned across sites to support comparability. Further details of the protocol are included in the supplementary material. 2.2. Speech Task and Speech Task Validity Figure 1 presents an overview of the entire data processing and analysis pipeline. Both the semi-structured clinical PSYCHS interview transcripts and the open-ended qualitative interview transcripts were made available. 2.2.1. Open-ended Language Samples The open-ended interviews were collected at baseline and month 2 and were collected remotely and on-site using the Zoom communications platform or a digital recorder. Instead of conducting a semi-structured interview, interviewers instead described that “In this interview, in particular, I would really like to get to know you better and learn what your life is like”. The content of the open-ended language samples was chiefly directed by the interviewees. The interviews typically lasted 10 to 30 minutes and the audiofiles were transcribed using the Transcription service TranscribeMe! 2.2.2. Semi-structured Clinical Interviews The semi-structured clinical interview is based on the Positive SYmptoms and Diagnostic Criteria for the CAARMS [ 73 ] Harmonized with the SIPS [ 74 ] (PSYCHS) [ 32 ] interview. The PSYCHS is a measure to evaluate attenuated positive symptoms in psychosis risk based on a harmonized Comprehensive Assessment of At-Risk Mental States (CAARMS) [ 73 ] and Structured Interview for Psychosis-Risk Syndromes [ 74 ]. The PSYCHS language samples were collected at screening, baseline and then monthly. The PSYCHS semi-structured interview lasted 30 minutes or longer; but only the first 30 minutes were transcribed. Across the two, semi-structured interviews are generally considered more consistent [ 48 ] and codable [ 49 ] whereas open-ended interviews may elicit more fine-grained details and analysis [ 50 ]. The use of two different types of interviews, a semi-structured clinical-screening interview vs. an open-ended interview, allows us to gain different linguistic samples and NLP features which will facilitate comparison across the different methods and prompting techniques. 2.3. Speech and Linguistic Features We reviewed previous studies that applied automated natural language processing methods to analyze speech. With reference to Fig. 1, subsequently, in alignment with existing works [ 51 , 27 ], we used Python and a commonly used NLP package, Stanza [ 52 ], to extract a wide range of syntactic features ranging from part-of-speech (POS) tagging, named entity recognition (NER) to overall lexical characteristics such as word frequency and sentiment. Subsequently, two different data pre-processing methods were adopted as outlined below. 2.4. Data Pre-processing Next, with reference to Fig. 1, we adopted two data pre-processing methods to facilitate comparison. 2.4.1. Approach 1: Principal Component Analysis (PCA) In alignment with the work done by Tang et al. [ 51 ], in order to represent the variance from linguistic features without an assumption about the underlying latent constructs, we normalized and performed a principal component analysis (PCA) on the data. Another benefit of applying feature reduction is to prevent model over-fitting. This resulted in a total of 6 six features representing a total of 82% of the total variance. 2.4.2. Approach 2: Feature Selection (FS) The second pre-processing method entailed selecting the speech variables or features that were statistically significantly different between the CHR and CC groups, as done in [ 19 ]. As summarized in Tables 2 and 3 , this resulted in 14 features used for the OPEN transcripts and 21 features used for the PSYCHS transcripts respectively. We used Cohen’s d to evaluate the effect size calculation. In general, a Cohen’s d value of around 0.2 is considered a small effect, 0.5 a medium effect and 0.8 and above is considered a large effect. 2.5. Machine Learning Models We performed data balancing via Synthetic Minority Over-sampling Technique (SMOTE) and conducted experiments across four different classifiers: (i) Support Vector Machine (SVM), (ii) multi-layer perceptron (MLP), (iii) logistic regression (LR) and (iv) Gradient Boosted Decision Trees (GBDT). The models were selected in alignment with existing works [ 5 , 19 ]. Support vector machine is a widely used algorithm for binary classification tasks and performs well in scenarios with limited training data and a large feature set. Gradient boosted decision trees (GBDT) is a powerful ensemble technique that builds decision trees sequentially, with each tree correcting the residual errors of the previous ones. We implemented GBDT using CatBoost, a fast, scalable, and high performance GBDT library [ 53 ]. We selected these models for 2 primary reasons. First, SVM and decision trees are 2 widely employed classifiers and this allows comparability with previous studies. Second, they differ in complexity and overfitting risk. SVMs include explicit regularization and tend to generalize well when the number of features exceeds the number of samples. CatBoost, in contrast, can capture more complex interactions but is more prone to overfitting under such conditions [ 54 ]. Comparing their performance allows us to assess whether model complexity impacts generalization [ 5 ] across the different data pre-processing method and interview transcripts. Further details on the training parameters are available within the Supplement. 2.6. Model Performance Evaluation In alignment with existing work [ 28 , 55 , 56 ], we evaluate model performance using the (i) root mean squared error (RMSE), (ii) mean absolute error (MAE), and (iii) Pearson’s correlation coefficient (PCC) between model-estimated scores and the ratings provided by clinicians. 2.7. LLMs used As illustrated in Fig. 1, large language models (LLMs) are employed to analyze text to detect linguistic features related to psychotic symptoms. Given the increasing scrutiny of the ethics and privacy concerns of using closed-source LLMs [ 42 , 43 ] and the growing calls to focus on open-source models [ 44 ], within the context work, we chose to work with open-source models in order to facilitate transparent research. Moreover, as yet, no LLM is approved by the FDA [ 57 ] and the use of public or proprietary LLMs like ChatGPT risks invoking Health Insurance Portability and Accountability Act (HIPAA) non-compliance. We experimented with two LLMs from two different families outlined below. All experiments are conducted locally using a 4xNVIDIA DGX A100 GPU instance. 2.7.1. Llama 2 We used the highly popular open-source LLM by Meta, Llama 2, as the first benchmark [ 58 ]. It has been adopted and deployed for mental health assessment [ 59 ] and intervention [ 60 ]. Llama 2 has been pre-trained on 2 trillion tokens and fine-tuned on over on 1 million new human annotations using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). We chose the 7B version in alignment with existing work [ 61 , 62 ] and specifically worked with the Llama 2-chat version which is optimized for dialogue use cases. 2.7.2. Qwen 2.5 The second model we used is from the Qwen family [ 63 ], specifically the Qwen 2.5-Instruct. It is a quantized model capable of performing faster computation with less memory storage. Qwen 2 has been evaluated across various settings such as mental health crisis detection [ 24 ] and suicidal ideation [ 46 ]. 2.8. Prompting Methodology In alignment with existing work [ 23 , 64 , 28 ] we employed zero-shot prompting, with an English prompt that includes (i) task description, (ii) evaluation criteria, and (iii) conversation content. The prompt structure is outlined as follow: Prompt ZS = InputTranscripts + Prompt Part 1− S + Prompt Part 2− Q + Output (1) where InputTranscripts are the original transcripts gathered through the two different interview protocols (i.e. open-ended and the semi-structured interview). Prompt Part 1− S provides specific guidelines for the assessment or evaluation criteria to adopt. Prompt Part 2− Q poses the question that we want the LLM to answer. Output determines the type of model output providing a brief rationale of their decision. The goal of the prompt is to provide the LLM with adequate information to make an informed decision on the transcripts provided. Subsequently, we designed three different prompting strategies for Prompt Part 1− S as illustrated below. 2.8.1. Prompt 1: Neutral The CHR syndrome for psychosis provides a well-established paradigm for the early detection and intervention of psychosis [ 65 , 66 ]. The neutral prompt thus gives a broad context of CHR psychosis risk symptoms, such as unusual perceptual experiences, disorganized communication or social isolation, which were selected based upon clinical evidence and existing research [ 67 , 68 , 69 , 70 ]. The specific prompt then becomes: “please analyze the following interview transcript and determine whether the individual demonstrates characteristics consistent with Clinical High Risk (CHR) for psychosis status. Consider the following areas in your analysis: Unusual perceptual experiences. Unusual thought content or beliefs. Disorganized communication. Decreased functioning or motivation. Social withdrawal or isolation.” 2.8.2. Prompt 2: NLP-based The NLP-based prompt requires the LLM to conduct their assessment based on evidence derived from the extracted NLP features summarized in Tables 2 and 3 for the open and PSYCHS-based interviews respectively. This entails providing the LLMs with the features selected via the procedure outlined in Section 2.4.2. The features in Table 2 were provided to the LLMs when running the analysis across the open-interview transcripts. Correspondingly, the features in Table 3 were provided to the LLMs when running the NLP-based prompt across the PSYCHS-interview transcripts. Specifically, Prompt Part 1− S therefore becomes: “Using the computational linguistic features extracted from speech data, evaluate Clinical High Risk (CHR) status. Research indicates that individuals at CHR may show: Reduced semantic density. Increased speech graph disconnectivity. Altered syntactic complexity. Changes in affective language patterns. Decreased coherence in narrative structure.” 2.8.3. Prompt 3: PSYCHS-based The PSYCHS-based prompt is structured exactly according to how a PSYCHS-based clinical-screening assessment is conducted. Specifically, PromptPart 1 − S therefore becomes: “Using the Psychological Screening for Clinical High Risk (PSYCHS) interview protocol, evaluate this individual’s CHR status. The PSYCHS assessment focuses on: Domain 1: Positive Symptoms (Attenuated) P1: Unusual thought content/delusional ideas P2: Suspiciousness/persecutory ideas P3: Grandiose ideas P4: Perceptual abnormalities/hallucinations P5: Disorganized communication Domain 2: Negative Symptoms N1: Social anhedonia N2: Avolition N3: Expression of emotion N4: Experience of emotion and self N5: Ideational richness N6: Occupational functioning Domain 3: Disorganization D1: Odd behavior/appearance D2: Bizarre thinking D3: Trouble with focus and attention D4: Personal hygiene.” 3. RESULTS 3.1. NLP Feature Analysis Results With reference to Tables 2 and 3 , we see that the speech features that were statistically significantly different between the CHR and CC groups differ between the OPEN and PSYCHS interview transcripts. This suggests that interview structure has a significant impact on various linguistic features such as speech content, syntax and lexical complexity and thus should be used judiciously when trying to perform automated linguistic analysis of schizophrenia. 3.2. NLP Results With reference to Table 4 , across the open-interview transcripts , the NLP-feature selection method (approach 2) in general produced better predictive outcomes across the different ML methods. For PCA, CatBoost seems to be the best method overall as it achieves the highest accuracy (0.727) and F1-score (0.760). Across feature selection, MLP is the best method across accuracy (0.887) and F1 (0.940) but logistic regression may be a better method according to AUROC (0.772). Across the PSYCHS -interview transcripts, there is no observable difference between the PCA-based and FS-based methods. In general, the ML-based methods largely provide accuracy results within a range of 0.60 − 0.90 and does not fluctuate too much across different models, settings and interview transcripts. With reference to Table 4 , we see that across all measures, the standard deviation of the averaged values all fall within a range of 0.05 − 0.20. 3.3. LLM Results Across the LLM results in Table 5 , we see an interesting trend where Llama in general performs much better than Qwen across text-based prompting, i.e. the P1: neutral-based prompt and P3: PSYCHS-based prompt whereas Qwen produces more accurate prediction outcomes across the NLP-based prompting (i.e. P2: NLP-based prompting). For instance, across the open transcripts, Llama P3 produce the best accuracy of 0.700 and the best AUROC of 0.658 across all the results from both models. The second best result is given by Qwen P2 which gives an accuracy of 0.600 and an AUROC of 0.500. Across the PSYCHS transcripts, we see a similar trend where Llama produced better prediction results across P1 (accuracy = 0.780, F1 = 0.857, AUROC = 0.812) and P3 (accuracy = 0.880, F1 = 0.930, AUROC = 0.751) and Qwen produced better prediction results across P2 (accuracy = 0.860, F1 = 0.925). Overall, we see that LLMs perform better across the PSYCHS-based interview transcripts. We further discuss this phenomenon within the discussion section below. 3.4. NLP vs. LLM Results Looking at the results in Tables 4 and 5 , we see a distinct difference between the variability in results across the two different automated classification methods. First, the results using NLP and ML-based methods are largely more consistent and fluctuate less across the different ML methodologies. For instance, with reference to Table 4 , most of the ML methods provide an accuracy greater than 0.80. The best performing results hover around 0.899 whereas the worst accuracy is around 0.613. We see that across all measures, the standard deviation of the averaged values all fall within a range of 0.05 − 0.20 with the most consistent being precision with a standard deviation of 0.056 and the highest being AUROC with a standard deviation of 0.193. In contrast, with reference to Table 5 , LLM-based methods exhibited greater variability in outcome. To illustrate, the best performing setting using Llama-P3: PSYCHS-based prompting achieved an impressive outcome at an accuracy of 0.880 and F1-score of 0.930. However, the setting using Qwen-P3: PSYCHS-based prompting yielded an extremely low accuracy of 0.320 and F1 of 0.346. Across the averaged measures, we see that the standard deviations of the averaged values fall within a range of 0.121 − 0.379 with the most consistent being AUROC with a standard deviation of 0.121 and the most variable being recall with a standard deviation of 0.379. 4. DISCUSSION How do LLMs compare against well-established NLP methods? We see that across prediction performance, LLMs do not necessarily perform better than NLP-based methods. For instance, across the open transcripts, the use of any of the feature preprocessing option (PCA or FS) combined with any ML methodology is likely to outperform an LLM like Qwen. Across the open transcripts, the best NLP-based result is the feature selection combined with MLP which gives an accuracy and F1 of 0.887 and 0.901 respectively. On the other hand, the best LLM performance given by Llama using a PSYCHS-based prompt yielded an accuracy of 0.700 and F1 of 0.776. Moreover, the NLP and ML-based methods are also likely to be more interpretable (i.e. easier to understand and interpret) [ 71 ]. This is because we are able to understand the entire classification pipeline from when and how the features are selected (e.g. PCA, FS) and how the ML models perform the classification. As a result, such methods are often deemed more trustworthy than black-box approaches as researchers and clinicians are able to verify whether the outcomes were arrived at via a reasonable process or procedure [ 72 ]. Within the context of this experiment setting, given that NLP and ML-based methods have greater diagnostic accuracy and are inherently more interpretable, it would seem that using NLP combined with ML-based methods would be a better option than adopting LLM-based methods. Our experiment also cautions against the blind adoption of LLMs without sufficient evaluation and understanding. Performance difference across different feature extraction methods and different prompt-engineering techniques Research efforts that rely on automated analysis and assessment of language and speech using NLP can vary in terms of their technical approaches [ 19 , 6 ]. For instance, in the context of psychosis, some works have focused on using latent semantic analysis (LSA) and word-embeddings extracted using neural network models to assess and evaluate semantic coherence whereas other works focused on analyzing the syntactic structure and changes in sentence patterns using part-of-speech (POS) tagging [ 19 ]. Our experimental results seem to indicate that even if there is variation in terms of the predictive performance of the ML methods trained using different feature selection methods, the results will still largely fall within a reasonable range and remain largely consistent across different ML methods employed. However, this is not true for LLM-based assessment outcomes. We witness an interesting trend where each model type consistently performs better across certain prompt-engineering methods. For instance, Qwen consistently produces more accurate assessment across the NLP-based prompting method. whereas Llama is often better across the neutral and PSYCHS-based prompting. One potential reason for this is that Llama is trained for dialogue use and is optimized to work with text-based input and textual reasoning [ 58 ] whereas Qwen is optimized to work across numerical reasoning. This has significant implications for the clinical community. Despite the proliferation of using LLMs to assess and predict mental health [ 40 , 41 ], insufficient research has been devoted towards understanding which LLM to use for what input and for what use cases. Thus, clinicians and researchers may need to devote greater effort to investigate which model may work best given their input setting. For instance, it may be better to choose Llama to analyze a text-based transcript and it may be better to choose Qwen to analyze digital phenotyping data. How generalizable are the findings across different interview formats:? Another noteworthy aspect is that the efficacy of NLP combined with ML-based methods seems to be more consistent and generalizable across the different transcript types (i.e. open vs. PSYCHS-based interview). This suggest that the NLP-based methods are picking up on the more reliable and consistent signals when attempting to distinguish between the community controls and CHR individuals. We see that despite the difference in diagnostic accuracy, all the ML-based methods largely provide accuracy results within a range of 0.60 − 0.90 across all models and interview formats (i.e. open vs. PSYCHS-based). Overall, compared to NLP-based methods, LLMs are less consistent and display greater variation depending on the input transcript type and LLM used (i.e. Llama vs. Qwen) and prompting strategy (i.e. neutral prompt, NLP-based prompt and PSYCHS-based prompting). We hypothesize that this could be due to how different interview methods may naturally lead to different text-based output. Given how NLP-based methods typically rely on quantifiable metrics such as text-to-token ratio etc. to assess the presence (or absence) of quantifiable linguistic features associated with CHR symptoms, it may be more consistent and robust against changes in semantic content of the input transcript. On the other hand, LLMs typically conduct their analysis based on the semantic content of the input transcript and will thus be more susceptible to performance changes when presented with a transcript that is generic (e.g. the open-ended language samples) vs. a language sample that specifically discusses semantic material related to CHR symptoms. In other words, LLMs pick up on the verbalized symptoms presented within the transcript rather than rely on objective linguistic measures like NLP-based methods. This is supported by our results which showed that LLMs generally perform much better across the PSYCHS-based interview transcripts rather than the open-ended language samples. This finding has several implications. First, this suggests that NLP-based methods may be more reliable and objective method than LLMs. Second, the usage of LLM will be suitable if the transcript content provides adequate semantic content for the LLM to perform contextualized reasoning. Third, even if LLMs are utilized within such settings, there is still a need to investigate and evaluate which model and prompting technique are best suited for the task in order to ensure reliable and trustworthy outputs. 4.1. Limitation We did not run experiments using other LLMs. Another significant limitation is that we did not have the chance to explore many important limitations of leveraging LLMs in mental health settings such as problem of bias and interpretability. We encourage future work to investigate such issues. We wish to caution that technical results do not translate to real-world application and deployability. Moreover, we have only conducted experiments within a zero-shot context. Future work can replicate the above experiments using other prompt engineering techniques and deployment methodologies such as few-shot prompting. Conclusion Across different input interview types, NLP-based methods seem more reliable and consistent across different feature extraction methods compared with LLM-based methods across different prompt engineering techniques. LLMs are better at analyzing text which provides semantic-based content or signal for automated analysis. Even though the usage of LLMs seem promising, there is still a need to investigate which model and which prompting methodology to utilize in order to ensure consistent and reliable results. References Cheryl Mary Corcoran and, Guillermo A, Cecchi (2020) Using language processing and speech analysis for the identification of psychosis and other disorders. Biol Psychiatry: Cogn Neurosci Neuroimaging 5(8):770–779 Murray Alpert RJ, Shaw ER, Pouget, Kelvin O, Lim (2002) A comparison of clinical ratings with vocal acoustic measures of flat affect and alogia. J Psychiatr Res 36(5):347–353 Alex S, Cohen KR, Mitchell, Elvevag B (2014) What do we really know about blunted vocal affect and alogia? a˚ meta-analysis of objective assessments. Schizophr Res 159:2–3 Cheryl M, Corcoran VA, Mittal CE, Bearden RE, Gur K, Hitczenko Z, Bilgrami A, Savic GA, Cecchi, Wolff P (2020) Language as a biomarker for psychosis: a natural language processing approach. Schizophr Res 226:158–166 Rybner A, Jessen ET, Damsgaard Mortensen M, Nyhus Larsen S, Zhou Y, Koelkebeck K, Bliksted V, Fusaroli R (2023) Vocal markers of schizophrenia: assessing the generalizability of machine learning models and their clinical applicability. Eur Psychiatry 66(S1):S186–S186 Parola A, Simonsen A, Lin JM, Zhou Y, Wang H, Ubukata S, Koelkebeck K, Bliksted V, Fusaroli R (2023) Voice patterns as markers of schizophrenia: building a cumulative generalizable approach via a cross-linguistic and meta-analysis based investigation, Schizophrenia bulletin , vol. 49, no. Supplement 2, pp. S125–S141 Katiah Llerena GP, Strauss, Alex S, Cohen (2012) Looking at the other side of the coin: a meta-analysis of selfreported emotional arousal in people with schizophrenia. Schizophr Res 142:1–3 Alex S, Cohen T, Cowan TP, Le, Elana K, Schwartz B, Kirkpatrick IM, Raugh HC, Chapman, Strauss GP (2020) Ambulatory digital phenotyping of blunted affect and alogia using objective facial and vocal analysis: proof of concept. Schizophr Res 220:141–146 Alex S, Cohen CR, Cox, Thanh P, Le T, Cowan MD, Masucci GP, Strauss, Kirkpatrick B (2020) Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia. npj Schizophrenia 6(1):26 Alex S, Cohen KR, Mitchell NM, Docherty, Horan WP (2016) Vocal expression in schizophrenia: Less than meets the ear. J Abnorm Psychol 125(2):299 Asher Cohen D, Joshi A, Bondre PK, Chand N, Chaturvedi S, Choudhary S, Dutt A, Khan C, Langholm M, Kumar et al (2024) Digital phenotyping correlates of mobile cognitive measures in schizophrenia: A multisite global mental health feasibility trial. PLOS Digit Health 3(6):e0000526 Nicholas Cummins M, Arribas T, Gibbs-Dean E, Molina D, Sethi M, Kempton S, Morgan T, Spencer K, Diederen, Olah J (2023) Towards a scalable approach to assess speech organization across the psychosis-spectrum-online assessment in conjunction with automated transcription and extraction of speech measures Asher Cohen JA, Naslund S, Chang S, Nagendra A, Bhan A, Rozatkar J, Thirthalli A, Bondre D, Tugnawat PV, Reddy et al (2023) Relapse prediction in schizophrenia with smartphone digital phenotyping during covid-19: a prospective, three-site, two-country, longitudinal study, Schizophrenia , vol. 9, no. 1, p. 6 Eric YH, Chen, Linda CW, Lam CS, Kan CSKY, Chan CL, Kwok DGH, Nguyen, Chen RYL (1996) Language disorganisation in schizophrenia: validation and assessment with a new clinical rating instrument. East Asian Archives Psychiatry 6(1):4 Neguine Rezaii E, Walker, Wolff P (2019) A machine learning approach to predicting psychosis using semantic density and latent content analysis. NPJ schizophrenia 5(1):9 Aguila AF-BDD, Cerda M, Gaspar PA, Terissi LD (2022) Manuel Duran, and Camila´ Valderrama, Automatic language analysis identifies and predicts schizophrenia in first-episode of psychosis. Schizophrenia 8(1):53 Alban E, Voppel, Janna N, de Boer SG, Brederoo HG, Schnack, Iris EC, Sommer (2021) Quantified language connectedness in schizophrenia-spectrum disorders. Psychiatry Res 304:114130 Gillinder Bedi F, Carrillo GA, Cecchi DF, Slezak M, Sigman Nat´aliaB, Mota Sidarta´, Ribeiro DC, Javitt M, Copelli, Cheryl M, Corcoran (2015) Automated analysis of free speech predicts psychosis onset in high-risk youths. npj Schizophrenia 1(1):1–7 Cheryl M, Corcoran F, Carrillo D, Fernandez-Slezak G, Bedi C, Klim DC, Javitt, Carrie E´ Bearden, and, Guillermo A, Cecchi (2018) Prediction of psychosis across protocols and risk cohorts using automated language analysis, World Psychiatry , vol. 17, no. 1, pp. 67–75 Takeshi Nakaura R, Ito D, Ueda T, Nozaki Y, Fushimi Y, Matsui M, Yanagawa A, Yamada T, Tsuboyama N, Fujima et al (2024) The impact of large language models on radiology: a guide for radiologists on the latest innovations in ai. Japanese J Radiol 42(7):685–696 Betzler BK, Chen H, Cheng C-Y, Cecilia S, Lee G, Ning SJ, Song AY, Lee R, Kawasaki P, van Wijngaarden A, Grzybowski et al (2023) Large language models and their impact in ophthalmology. Lancet Digit Health 5(12):e917–e924 Volkmer S, Meyer-Lindenberg A, Schwarz E (2024) Large language models in psychiatry: Opportunities and challenges. Psychiatry Res 339:116026 Xu X, Yao B, Dong Y, Gabriel S, Yu H, Hendler J, Ghassemi M, Dey AK, Wang D (2024) Mental-llm: Leveraging large language models for mental health prediction via online text data, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , vol. 8, no. 1, pp. 1–32 Guifeng Deng S, Rao T, Lin A, Dai P, Wang J, Xie H, Song K, Zhao D, Xu Z, Cheng et al (2025) Evaluating large language models in crisis detection: A real-world benchmark from psychological support hotlines, arXiv preprint arXiv:2506.01329 Justin Khasentino A, Belyaeva X, Liu Z, Yang NA, Furlotte C, Lee E, Schenck Y, Patel et al (2025) Jian Cui, Logan Douglas Schneider,., A personal health large language model for sleep and fitness coaching, Nature Medicine , pp. 1–10 Tingyu Mo, Jacqueline CK, Lam VOK, Li, Cheung LYL (2024) Leveraging large language models for identifying interpretable linguistic markers and enhancing alzheimer’s disease diagnostics. medRxiv, pp. 2024–2008 Neguine Rezaii D, Hochberg M, Quimby B, Wong M, Brickhouse A, Touroutoglou BC, Dickerson, Wolff P (2024) Artificial intelligence classifies primary progressive aphasia from connected speech. Brain 147(9):3070–3082 Liu C-M, Chan Y-H, Ho M-Y, Liu C-C, Lu M-H, Liao Y-A (2025) Ming-Hsien Hsieh, and Yufeng Jane Tseng, Analyzing generative ai and machine learning in auto-assessing schizophrenia’s negative symptoms. Schizophr Bull, p. sbaf102 Li R, Cao M, Fu D, Wei W, Wang D, Yuan Z, Hu R, Deng W (2024) Deciphering language disturbances in schizophrenia: A study using fine-tuned language models. Schizophr Res 271:120–128 Arabzadeh N, Clarke CLA (2025) A human-ai comparative analysis of prompt sensitivity in llm-based relevance judgment, in Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 2784–2788 Donald A, Young Z, Campbell KK, Zakzanis, Weinstein E (2003) A comparison between an interview and a self-report method of insight assessment in chronic schizophrenia. Schizophr Res 63:1–2 Scott W, Woods S, Parker MJ, Kerr, Barbara C, Walsh SA, Wijtenburg N, Prunier AR, Nunez et al (2024) Kate Buccilli, Catalina Mourgues-Codern, Kali Brummitt,., Development of the psychs: Positive symptoms and diagnostic criteria for the caarms harmonized with the sips, Early Intervention in Psychiatry , vol. 18, no. 4, pp. 255–272 Maayan KS-WN, Bergman H, Davenport C, Amanda J, Kirkham S, Grabowski, Clive E, Adams (2015) First rank symptoms for schizophrenia. Cochrane Database Syst Reviews,, 1 Tashrif Billah KIK, Cho O, Borders Y, Chung M, Ennis GR, Jacobs E, Liebenthal DH, Mathalon D, Mohandass, Spero C, Nicholas et al (2025) Enabling fair data stewardship in complex international multi-site studies: Data operations for the accelerating medicines partnership® schizophrenia program. Schizophrenia 11(1):55 Mlay JP, Naidu T, Ramlall S, Patankar K, Israel K, Esquivel L, Curran L, Mhlungu SS, Zondi M, Lessells R et al (2025) Strategies for relapse prevention among people with schizophrenia in kwazulu-natal province, south africa: HealtCCare providers’ perspectives. PLoS ONE 20(3):e0316313 Thomas S, Christy (2025) Screening for prodromal symptoms of schizophrenia in primary care settings. Lynchburg J Med Sci 1(1):8 Kho JJ, Song S, Tan SMX, Fitriyah NH, Lokadjaja MC, Yee JY, Yang Z, Chen EYH, Lee J, Wen W, Goh B (2025) Leveraging computational linguistics and machine learning for detection of ultra-high risk of mental health disorders in youths. Schizophrenia 11(1):98 Gumus M, Koo M, Studzinski CM, Bhan A, Robin J, Black SE (2024) Linguistic changes in neurodegenerative diseases relate to clinical symptoms. Front Neurol 15:1373341 Marion C, Leaman, Lisa A, Edmonds (2023) Analyzing language in the picnic scene picture and in conversation: The type of discourse sample we choose influences findings in people with aphasia. Am J Speech-Language Pathol 32(4):1413–1430 Ali M, Lucasius C, Patel TP, Aitken M, Vorstman J, Szatmari P, Battaglia M, Kundur D (2025) Speech as a multimodal digital phenotype for multi-task llm-based mental health prediction, arXiv preprint arXiv:2505.23822 Jared Moore D, Grabb W, Agnew K, Klyman S, Chancellor DC, Ong, Haber N Expressing stigma and inappropriate responses prevents llms from safely replacing mental health providers., in Proceedings of the (2025) ACM Conference on Fairness, Accountability, and Transparency , 2025, pp. 599–627 Tala Mirzaei L, Amini, Esmaeilzadeh P (2024) Clinician voices on ethics of llm integration in healtCCare: a thematic analysis of ethical concerns and implications. BMC Med Inf Decis Mak 24(1):250 Esmaeilzadeh P (2025) Ethical implications of using general-purpose llms in clinical settings: a comparative analysis of prompt engineering strategies and their impact on patient safety. BMC Med Inf Decis Mak 25(1):342 Fabio Dennstadt J, Hastings PM, Putora M, Schmerder, Cihoric N (2025) Implementing large language¨ models in healtCCare while balancing control, collaboration, costs and security. NPJ Digit Med 8(1):143 Zhang T, Teng S, Jia H (2024) and Simon D’Alfonso, Leveraging llms to predict affective states via smartphone sensor features, in Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing , pp. 709–716 Joao Pedro C, Azevedo, Adonias C, de Oliveira F, Silva L, Coutinho, Teles AS Harnessing˜ generative llms to detect and explain suicidal ideation in brazilian portuguese texts, in (2025) IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS) . IEEE, 2025, pp. 15–20 Cassandra MJ, Wannan B, Nelson J, Addington K, Allott A, Anticevic C, Arango, Justin T, Baker CE, Bearden T, Billah S, Bouix et al (2024) Accelerating medicines partnership® schizophrenia (amp® scz): rationale and study design of the largest global prospective cohort study of clinical high risk for psychosis. Schizophr Bull 50(3):496–512 William C, Adams (2015) Conducting semi-structured interviews, Handbook of practical program evaluation , pp. 492–505 Saglam Y (2024) Which data gathering method is superior: An open-ended questionnaire or a semi-structured interview? Int J Stud Educ (IJonSE), 6, 3 Rahul S, Chauhan (2022) Unstructured interviews: are they really all that bad? Hum Resource Dev Int 25(4):474–487 Sunny X, Tang MJ, Spilka M, John ML, Birnbaum E, Saito SA, Berretta LM, Behbehani, Mark Y, Liberman, Anil K, Malhotra W, Simpson et al (2025) Automated speech and language markers of longitudinal changes in psychosis symptoms. NPP—Digital Psychiatry Neurosci 3(1):13 Qi P, Zhang Y, Zhang Y, Bolton J, Manning CD Stanza: A python natural language processing toolkit for many human languages, arXiv preprint arXiv :(2003). 07082 , 2020 Liudmila Prokhorenkova G, Gusev A, Vorobev AV, Dorogush, Gulin A (2018) Catboost: unbiased boosting with categorical features. Adv Neural Inf Process Syst, 31 Visar Berisha C, Krantsevich PR, Hahn S, Hahn G, Dasarathy P, Turaga, Liss J (2021) Digital medicine and the curse of dimensionality. NPJ Digit Med 4(1):153 Huang Y-J, Lin Y-T, Liu C-C, Lee L-E, Hung S-H, Lo J-K, Fu L-C (2022) Assessing schizophrenia patients through linguistic and acoustic features using deep learning techniques. IEEE Trans Neural Syst Rehabil Eng 30:947–956 Mina Bishay P, Palasek S, Priebe, Patras I (2021) Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Trans Affect Comput 12(4):949–961 Gary E, Weissman T, Mankowitz, Genevieve P, Kanter (2025) Unregulated large language models produce medical device-like output. NPJ Digit Med 8(1):148 Hugo Touvron L, Martin K, Stone P, Albert A, Almahairi Y, Babaei N, Bashlykov S, Batra P, Bhargava S, Bhosale et al (2023) Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 Zijing Yang E, Alveyn M, Dey N, Arumalla MD, Russell S, Norton, Galloway JB (2024) Impact of visualising healtCCare quality performance: a systematic review. BMJ open 14(11):e083620 Gopika S, Kalaiselvi K, Jacob M, Vignesh K (2024) Empowering mental health resilience: Llama 2-based ai model for all stages. in Revitalizing Health Through Humanities. Routledge, pp 402–409 Sarah Sandmann S, Riepenhausen L, Plagwitz, Varghese J (2050) Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks, Nature communications , vol. 15, no. 1, pp. 2024 Nivethitha V, Verma N, Bokadia R, Jaju R, Fine-tuning llama-2-7b model for conversation summarization, in (2024) 15th International Conference on Computing Communication and Networking Technologies (ICCCNT) . IEEE, 2024, pp. 1–7 Binyuan Hui J, Yang Z, Cui J, Yang D, Liu L, Zhang T, Liu J, Zhang B, Yu K, Lu et al (2024) Qwen2. 5-coder technical report, arXiv preprint arXiv:2409.12186 Flathers M, Xia W, Hau C, Nelson BW, Cheong J, Burns J, Torous J (2025) Interpreting psychiatric digital phenotyping data with large language models: a preliminary analysis. BMJ Mental Health, 28, 1 De Pablo PF-PGS, Christoph U, Correll A, Meyer-Lindenberg MJ, Millan S, Borgwardt S, Galderisi A, Bechdolf A, Pfennig (2020) Prevention of psychosis: advances in detection, prognosis, and intervention. JAMA psychiatry 77(7):755–765Lars Vedel Kessing Ana Catalan GS, de Pablo JV, Serrano P, Mosillo H, Baldwin U, Correll I, Bonoldi et al (2021) Annual research review: Prevention of psychosis in adolescents–systematic review and meta-analysis of advances in detection, prognosis and intervention, Journal of Child Psychology and Psychiatry , vol. 62, no. 5, pp. 657–673 Hussein Hijazi, Moukaddam N (2025) Schizophrenia & psychosis. prodromal symptoms of schizophrenia: Understanding and addressing challenges. Psychiatric Times, 42, 6 Chen Y-F, Yeh T-P, Hsu H-C, Lane H-Y, Lu S-H, Ma W-F (2025) Clinical symptoms of the prodromal stage preceding serious mental illness in taiwanese young adults: a qualitative study. BMC Psychiatry 25(1):187 Tandy J, Miller TH, McGlashan SW, Woods K, Stein N, Driesen CM, Corcoran R, Hoffman, Davidson L (1999) Symptom assessment in schizophrenic prodromal states. Psychiatr Q 70(4):273–287 Klosterkotter J, Hellmich M, Steinmeyer EM, Schultze-Lutter F (2001) Diagnosing schizophrenia¨ in the initial prodromal phase. Arch Gen Psychiatry 58(2):158–164 Dan W, Joyce A, Kormilitzin KA, Smith, Cipriani A (2023) Explainable artificial intelligence for mental health through transparency and interpretability for understandability. npj Digit Med 6(1):6 Dmytro Savchuk, Doroshenko A (2025) Explainable ai methods to increase trustworthiness in healtCCare. i n Responsible and Explainable Artificial Intelligence in HealtCCare . Elsevier, pp 55–89 Alison R, Yung AR, Yung HP, Yuen PD, Mcgorry, Lisa J, Phillips D, Kelly M, Dell’Olio SM, Francey EM, Cosgrave E, Killackey et al (2005) Mapping the onset of psychosis: the comprehensive assessment of at-risk mental states. Aust N Z J Psychiatry 39:11–12 Thomas H, McGlashan BC, Walsh, Scott W, Woods J, Addington K, Cadenhead T, Cannon, Walker E (2001) Structured interview for psychosis-risk syndromes. Yale School of Medicine, New Haven, CT Cheong J, Bangar A, Kalkan S, Gunes H (2025) U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection. In: Proceedings of the 4th Machine Learning for Health Symposium (Proceedings of Machine Learning Research). Vol. 259. PMLR, 203–218 Cheong J, Kalkan S, Gunes H. Causal structure learning of bias for fair affect recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 340–349., Cheong J, Kalkan S (2023a) and H. Gunes. 2023b. Counterfactual fairness for facial expression recognition. In: ECCV 2022 Workshops. Springer, 245–261 Cheong J, Kalkan S, Gunes H (2024) FairReFuse: referee-guided fusion for multimodal causal fairness in depression detection. In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 7224–7232 Cheong J, Kalkan S, Gunes H (2021) The Hitchhiker’s Guide to Bias and Fairness in Facial Affective Signal Processing: Overview and techniques. IEEE Signal Process Mag 38(6):39–49. 10.1109/MSP.2021.3106619 Cheong J, Kuzucu S, Kalkan S, Gunes H (2023) Towards gender fairness for mental health prediction. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 5932–5940 Cheong J, Spitale M, Gunes H (2025) Small but Fair! Fairness for Multimodal Human-Human and Robot-Human Mental Wellbeing Coaching. IEEE Transactions on Affective Computing. IEEE Spitale M, Cheong J, Gunes H (2024) Underneath the numbers: Quantitative and qualitative gender fairness in llms for depression prediction. arXiv preprint arXiv:2406.08183 Cheong J, Spitale M, Gunes H (2023), September It’s not Fair!–Fairness for a Small Dataset of Multi-modal Dyadic Mental Well-being Coaching. In 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 1–8). IEEE Cheong J, Mogharabin A, Liang P, Gunes H, Kalkan S (2025b) Fairwell: Fair multimodal self-supervised learning for wellbeing prediction. arXiv preprint arXiv:2508.16748 Kwok AMH, Cheong J, Kalkan S, Gunes H (2025) Machine learning fairness for depression detection using eeg data. In 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE J. Castillo, J. Cheong, S. Choudhary, A. Bondre, A. R. Rozatkar, … J. & Torous (2025).Mobile cognitive remote assessment of schizophrenia: a global multi-site pilot study. Schizophrenia , 11 (1), 144 Chen K, Torous J, Cheong J (2025) The Current State/Trends in Digital Phenotyping for Mental Health Research and Care. Psychiatric Clinics Kuzucu S, Cheong J, Gunes H, Kalkan S (2024) Uncertainty as a fairness measure. Journal of Artificial Intelligence Research , 81 , 307–335. Tables Tables are available in the Supplementary Files section. Additional Declarations The authors declare no competing interests. Supplementary Files SUPPLEMENT.docx Tables.docx Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8777643","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":585531066,"identity":"fd6130bb-5b5e-411f-8d84-eb5e7e41014d","order_by":0,"name":"Jiaee Cheong","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAxklEQVRIiWNgGAWjYBACxgbmgw8SKmzgAgZEaGFLNvhwJo0ELQwMPGaSM9sOk6CFuf2AgTTPmfPy/NMOMH74wXDYmLDDehISjHkqbhvOuJ3ALNnDcNiMsJYZDAeSec7cTmAAImkGhsM2RGhhbDjM23YuQR5oy28itTAzNs5sO5BgcDuBDWQLEQ7rSWNm+HAm2XDj7cQ2yx6DdMLeN2w///1HQoWdvNzt5MM3flRYGzYQ1IJQwdhAXETKE6FmFIyCUTAKRjoAAOg9PSj0zGzTAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-5964-2284","institution":"Harvard Medical School","correspondingAuthor":true,"prefix":"","firstName":"Jiaee","middleName":"","lastName":"Cheong","suffix":""},{"id":585603349,"identity":"68d276b0-8b9d-476c-be1c-6dd299acf87a","order_by":1,"name":"Cheryl M. Corcoran","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai, New York, NY, USA","correspondingAuthor":false,"prefix":"","firstName":"Cheryl","middleName":"M.","lastName":"Corcoran","suffix":""},{"id":585605388,"identity":"9468a8f9-c8e2-4378-8e73-85625f63a722","order_by":2,"name":"Kathryn E. Lewandowski","email":"","orcid":"","institution":"McLean Hospital","correspondingAuthor":false,"prefix":"","firstName":"Kathryn","middleName":"E.","lastName":"Lewandowski","suffix":""},{"id":585605389,"identity":"e2427864-471c-4f52-844d-eb10b8a877e1","order_by":3,"name":"Ofer Pasternak","email":"","orcid":"","institution":"McLean Hospital","correspondingAuthor":false,"prefix":"","firstName":"Ofer","middleName":"","lastName":"Pasternak","suffix":""},{"id":585605390,"identity":"e2ced9a1-9f0a-4592-8f34-91b9bc27ce1c","order_by":4,"name":"Sinead Kelly","email":"","orcid":"","institution":"Brigham and Women's Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Sinead","middleName":"","lastName":"Kelly","suffix":""},{"id":585605391,"identity":"4b21c619-037d-403c-9155-70618d52d486","order_by":5,"name":"Sylvain Bouix","email":"","orcid":"","institution":"École de technologie supérieure","correspondingAuthor":false,"prefix":"","firstName":"Sylvain","middleName":"","lastName":"Bouix","suffix":""},{"id":585605392,"identity":"eefd1630-1148-449d-a8ed-3643f0a29383","order_by":6,"name":"Abraham Reichenberg","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Abraham","middleName":"","lastName":"Reichenberg","suffix":""},{"id":585605393,"identity":"e85e903b-a538-4be7-ab56-254552a79eed","order_by":7,"name":"Carrie E. Bearden","email":"","orcid":"","institution":"Semel Institute for Neuroscience and Human Behavior","correspondingAuthor":false,"prefix":"","firstName":"Carrie","middleName":"E.","lastName":"Bearden","suffix":""},{"id":585605394,"identity":"86f9dc51-b5f2-4a74-8a6e-a6ed1fb730d9","order_by":8,"name":"Guillermo Cecchi","email":"","orcid":"","institution":"IBM Research","correspondingAuthor":false,"prefix":"","firstName":"Guillermo","middleName":"","lastName":"Cecchi","suffix":""},{"id":585605395,"identity":"a76dd3fa-35d8-44a6-b525-bb7b93dc1f89","order_by":9,"name":"Justin T. Baker","email":"","orcid":"","institution":"Mclean Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Justin","middleName":"T.","lastName":"Baker","suffix":""},{"id":585605396,"identity":"8cca02d8-540e-4023-a645-870aec105d46","order_by":10,"name":"Marek Kubicki","email":"","orcid":"","institution":"Massachusetts General Hospital, Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Marek","middleName":"","lastName":"Kubicki","suffix":""},{"id":585605397,"identity":"dc92ed48-63c8-4ee8-bc9a-b5a6848f8183","order_by":11,"name":"Tina Kapur","email":"","orcid":"","institution":"Brigham and Women's Hospital, Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Tina","middleName":"","lastName":"Kapur","suffix":""},{"id":585605398,"identity":"bf53f6d8-a16e-4f8c-b4f4-550a402fe510","order_by":12,"name":"Daniel H. Mathalon","email":"","orcid":"","institution":"University of California","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"H.","lastName":"Mathalon","suffix":""},{"id":585605399,"identity":"7551f60a-daf8-43aa-956d-3ac02803891f","order_by":13,"name":"Kang-Ik K. Cho","email":"","orcid":"","institution":"Brigham and Women's Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Kang-Ik","middleName":"K.","lastName":"Cho","suffix":""},{"id":585605400,"identity":"f224a244-2d9f-4e6a-bed2-586034c253f3","order_by":14,"name":"Inge Winter-van Rossum","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Inge","middleName":"Winter-van","lastName":"Rossum","suffix":""},{"id":585605401,"identity":"feb7b5d0-c5ad-4751-abfd-8a33ed8390c0","order_by":15,"name":"Michael J. Coleman","email":"","orcid":"","institution":"Brigham and Women's Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"J.","lastName":"Coleman","suffix":""},{"id":585605402,"identity":"659ce59a-cc81-4d37-b86e-9a0ebb4fe122","order_by":16,"name":"Tashrif Billah","email":"","orcid":"","institution":"Brigham and Women's Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Tashrif","middleName":"","lastName":"Billah","suffix":""},{"id":585605403,"identity":"ac1a8c78-a32c-4237-af57-2cc2df62a0b1","order_by":17,"name":"Dheshan Mohandass","email":"","orcid":"","institution":"Brigham and Women's Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Dheshan","middleName":"","lastName":"Mohandass","suffix":""},{"id":585605404,"identity":"0b55adb7-5648-4756-8b15-1bfe50dd0407","order_by":18,"name":"Yoonho Chung","email":"","orcid":"","institution":"Mclean Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Yoonho","middleName":"","lastName":"Chung","suffix":""},{"id":585605405,"identity":"8d264a19-904f-4d6e-aa89-63d0c5488cf9","order_by":19,"name":"Habiballah Rahimi Eichi","email":"","orcid":"","institution":"Mclean Hospital and Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Habiballah","middleName":"Rahimi","lastName":"Eichi","suffix":""},{"id":585605406,"identity":"f925a0c1-68b4-4194-825b-148c9b5df484","order_by":20,"name":"Youngsun T. Cho","email":"","orcid":"","institution":"Yale University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Youngsun","middleName":"T.","lastName":"Cho","suffix":""},{"id":585605407,"identity":"b8b92224-d8e6-4e43-aaa0-79cd90205b76","order_by":21,"name":"Zailyn Tamayo","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Zailyn","middleName":"","lastName":"Tamayo","suffix":""},{"id":585605408,"identity":"f7efc0f5-11de-4468-9983-e4bfe40562b5","order_by":22,"name":"Jessica Hartmann","email":"","orcid":"","institution":"Orygen, Parkville","correspondingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"","lastName":"Hartmann","suffix":""},{"id":585605409,"identity":"944adf32-4bfc-4e56-8647-96e7e1a969d5","order_by":23,"name":"Patrick D. McGorry","email":"","orcid":"","institution":"Orygen, Parkville","correspondingAuthor":false,"prefix":"","firstName":"Patrick","middleName":"D.","lastName":"McGorry","suffix":""},{"id":585605410,"identity":"6b5c351c-ca92-4ea4-9bcc-85e1263ebcbf","order_by":24,"name":"Rene S. Kahn","email":"","orcid":"","institution":"University of California","correspondingAuthor":false,"prefix":"","firstName":"Rene","middleName":"S.","lastName":"Kahn","suffix":""},{"id":585605411,"identity":"2299912f-4e2d-4276-9aa5-1bfa7dfd6051","order_by":25,"name":"John M. Kane","email":"","orcid":"","institution":"Feinstein Institute for Medical Research","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"M.","lastName":"Kane","suffix":""},{"id":585605412,"identity":"20881809-d080-487a-b493-62c4cba09b5e","order_by":26,"name":"Scott W. Woods","email":"","orcid":"","institution":"Yale University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Scott","middleName":"W.","lastName":"Woods","suffix":""},{"id":585605413,"identity":"e5ab2e10-670b-42fb-84d5-5094bcdf6504","order_by":27,"name":"Martha E. Shenton","email":"","orcid":"","institution":"Massachusetts General Hospital, Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Martha","middleName":"E.","lastName":"Shenton","suffix":""},{"id":585605414,"identity":"b4c1f1fe-2f27-418d-982f-dfd931a59f31","order_by":28,"name":"Barnaby Nelson","email":"","orcid":"","institution":"The University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Barnaby","middleName":"","lastName":"Nelson","suffix":""},{"id":585605415,"identity":"83772b8f-b7d8-421b-9720-f7c5714bab71","order_by":29,"name":"John Torous","email":"","orcid":"","institution":"Beth Israel Deaconess Medical Center, Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Torous","suffix":""}],"badges":[],"createdAt":"2026-02-03 15:14:29","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-8777643/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-8777643/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":101943060,"identity":"64d1efaa-a573-4c69-a0ff-9c8d5f8abf7c","added_by":"auto","created_at":"2026-02-05 09:40:06","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":583882,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFlowchart for the entire data processing and analyzes pipeline\u003c/strong\u003e. Across Experiment Setting 1: NLP + ML Experiments, first, we extract NLP features across both the open and PSYCHS-based interview transcripts (A. Input Transcript) using Stanza (B. NLP Feature Extraction). Subsequently, we perform data pre-processing via principal component analysis (PCA) and feature selection (FS) (Stage C. Data Pre-processing). We then perform ML classification (Stage D. ML methods) across both sets of features. Across Experiment Setting 2: LLM Experiments, we feed the open and PSYCHS-based transcripts directly to Prompt 1: Neutral and Prompt 3: PSYCHS-based (Stage E. Prompt Types). We use the NLP features selected via feature selection as the input to Prompt 2: NLP-based. We then perform LLM analysis and classification (Stage F. LLM used) across all 3 sets of prompt engineering outputs.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8777643/v2/aa100258103aff25b6eed488.png"},{"id":101945532,"identity":"ca63b657-4674-496a-9fc5-82c140813b09","added_by":"auto","created_at":"2026-02-05 09:58:00","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1664420,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8777643/v2/14b3035a-2074-464c-b42c-6bb122566150.pdf"},{"id":101899695,"identity":"dca934fb-42fb-452e-a5c4-e1a8e32776b3","added_by":"auto","created_at":"2026-02-04 18:42:22","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":23283,"visible":true,"origin":"","legend":"","description":"","filename":"SUPPLEMENT.docx","url":"https://assets-eu.researchsquare.com/files/rs-8777643/v2/7cc4bf27fa963f1e7577670e.docx"},{"id":101899697,"identity":"3d1ab26b-0f31-43c8-8229-5d50c3ba79c8","added_by":"auto","created_at":"2026-02-04 18:42:22","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":896406,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-8777643/v2/074610681fa3dcbd58cc2465.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"Automated Detection Of Clinical High Risk Population Of Schizophrenia: Assessing The Generalizability Of NLP And LLM-Based Methods","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eAbnormalities in language and speech have been established as one of the early characteristics in individuals with subclinical psychotic symptoms [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Affected individuals may display an atypical linguistic or speech-based profile such as reduced word count, poverty of content [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] and reduced sentiment valence-arousal [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] that are indicative of core symptoms in schizophrenia (SZ), such as alogia and flat affect [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Several studies further suggest that language features could be used as biomarkers for early detection of schizophrenia [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] which would allow timely intervention and management [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Earlier efforts in linguistic analysis of SZ symptoms typically relied on manual clinician-based assessment which is considered less scalable and more labor-intensive [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. As a result, there has been a growing body of research using automated methods such as natural language processing (NLP) to extract and analyze relevant features [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003ePast studies focusing on automated methods have typically leveraged NLP-based methods for automated linguistic feature extraction and data analysis [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. This line of work has proven fruitful. For instance, NLP-based methods have successfully differentiated between patients with schizophrenia and a community control group by utilizing semantic and syntactic features with accuracy exceeding 80% [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Moreover, NLP-derived linguistic measures have also shown potential for identifying prodromal psychotic symptoms and for predicting later transition to psychosis among of clinical high-risk (CHR) individuals [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn more recent years, large language models (LLMs) have been increasingly used across a range of healthcare settings such as radiology [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], ophthalmology [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] and psychiatry [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Recent studies have also demonstrated their efficacy across symptom prediction [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], crisis detection [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] and health coaching [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Within the context of biomarker analysis, studies have shown that the use of LLMs have provided new methods to non-invasively analyze language disturbances and the classification of diagnoses [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWithin the context of neuropsychiatric disorders, research has indicated that contemporary generative LLMs can be used to differentiate among the three major primary progressive aphasias (PPA) variants from a speech sample at a level consistent with that of expert clinicians [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Further, Liu \u003cem\u003eet al.\u003c/em\u003e have investigated its utility in evaluating negative symptoms in schizophrenia [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. However, most of these existing studies have chiefly focused on deploying the LLMs with no comparison against more well-established NLP-based methods (Gap 1). As a result, there is less insight on the utility of using LLMs and whether the benefits of using LLMs for analysis outweighs the costs. Moreover, given LLM\u0026rsquo;s prompt sensitivity [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], there is also a lack of research investigating how different prompt engineering methods affect the different model\u0026rsquo;s output across different settings (Gap 2). Another long-standing open question within the field pertains to how best to objectively assess prodromal psychotic symptoms and how best to analyze different types of transcripts (Gap 3) [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. A range of transcript types have been used within the context of automated linguistic analysis in schizophrenia, ranging from open-ended interviews with conversational opening questions [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e], standardized or structured clinical screening interviews [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e], as well as task-based cognitive assessment transcripts [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e, \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eGiven the proliferation of using LLMs to assess and predict mental health [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e, \u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e], we build on a growing body of work investigating how LLMs can be used to automatically assess SZ symptoms [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] and attempt to address the research gaps outlined above. Our key contributions and the main research questions (RQs) that we attempt to address are outlined as follow:\u003c/p\u003e \u003cp\u003e\u0026bull; \u003cb\u003eRQ 1\u003c/b\u003e: How do LLMs compare against well-establish NLP methods across CHR symptom analysis and identification?\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eRQ 2\u003c/b\u003e: How does the ML performance differ across the different feature extraction methods and how does the LLM performance differ across different prompt-engineering methods ?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eRQ 3\u003c/b\u003e: How generalizable are the findings across different interview formats i.e. the open-ended interviews vs. the PSYCHS-based semi-structured clinical screening interviews ?\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eDue to the ethical concerns of existing AI research for mental health and growing calls to work with open-source models [\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e, \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e], we worked with open-source models and sought to compare LLMs\u0026rsquo; performance against more established NLP-based methods. Moreover, we also contribute to a growing body of work assessing the generalizability and clinical utility of using LLMs for automated assessment and diagnosis [\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e, \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e]. By addressing the aforementioned RQs, we assist clinicians and researchers in deciding how best to leverage NLP or LLM-based tools for better symptomatology understanding, diagnosis and analysis.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eFigure\u0026nbsp;1. Flowchart for the entire data processing and analyzes pipeline\u003c/b\u003e. Across Experiment Setting 1: NLP\u0026thinsp;+\u0026thinsp;ML Experiments, first, we extract NLP features across both the open and PSYCHS-based interview transcripts (A. Input Transcript) using Stanza (B. NLP Feature Extraction). Subsequently, we perform data pre-processing via principal component analysis (PCA) and feature selection (FS) (Stage C. Data Pre-processing). We then perform ML classification (Stage D. ML methods) across both sets of features. Across Experiment Setting 2: LLM Experiments, we feed the open and PSYCHS-based transcripts directly to Prompt 1: Neutral and Prompt 3: PSYCHS-based (Stage E. Prompt Types). We use the NLP features selected via feature selection as the input to Prompt 2: NLP-based. We then perform LLM analysis and classification (Stage F. LLM used) across all 3 sets of prompt engineering outputs.\u003c/p\u003e"},{"header":"2. METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\n \u003ch2\u003e2.1. Participants\u003c/h2\u003e\n \u003cp\u003eWe used language samples collected via the Accelerating Medicines Partnership for Schizophrenia (AMP-SCZ) project [\u003cspan class=\"CitationRef\"\u003e47\u003c/span\u003e]. We used version 3\u0026rsquo;s release where 2192 samples were provided as part of this release in total. Although three kinds of speech samples were collected: open-ended language samples, PSYCHS language samples and daily audio diaries, we only used the open-ended language samples and PSYCHS language samples within this analysis. Given that not all participants provided interview transcripts, we only worked with a subset of participants who provided at least one interview transcript. This resulted in 843 samples. Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e provides the demographic breakdown of the analyzed samples. Out of the total 843 samples, 513 were female and 330 were males. We see that the subsets of CHR vs. CC participants are broadly comparable across sociodemographic features. CHR participants are those that meet the diagnostic criteria as determined using the Positive Symptoms and Diagnostic Criteria for the CAARMS [\u003cspan class=\"CitationRef\"\u003e73\u003c/span\u003e] and Harmonized with the SIPS [\u003cspan class=\"CitationRef\"\u003e74\u003c/span\u003e] (PSYCHS) protocol [\u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e]. Community control (CC) participants were recruited from the community. CHR participants completed screening, baseline assessments and a series of follow-up assessments over a period of 24 months. CC participants completed screening and baseline assessments and a subset (5 per site) completed month 2, 12 and 24 visits. All samples were collected in separate sessions. Inclusion and exclusion criteria were aligned across sites to support comparability. Further details of the protocol are included in the supplementary material.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\n \u003ch2\u003e2.2. Speech Task and Speech Task Validity\u003c/h2\u003e\n \u003cp\u003eFigure 1 presents an overview of the entire data processing and analysis pipeline. Both the semi-structured clinical PSYCHS interview transcripts and the open-ended qualitative interview transcripts were made available.\u003c/p\u003e\n \u003cdiv id=\"Sec5\" class=\"Section3\"\u003e\n \u003ch2\u003e2.2.1. Open-ended Language Samples\u003c/h2\u003e\n \u003cp\u003eThe open-ended interviews were collected at baseline and month 2 and were collected remotely and on-site using the Zoom communications platform or a digital recorder. Instead of conducting a semi-structured interview, interviewers instead described that \u0026ldquo;In this interview, in particular, I would really like to get to know you better and learn what your life is like\u0026rdquo;. The content of the open-ended language samples was chiefly directed by the interviewees. The interviews typically lasted 10 to 30 minutes and the audiofiles were transcribed using the Transcription service TranscribeMe!\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e\n \u003ch2\u003e2.2.2. Semi-structured Clinical Interviews\u003c/h2\u003e\n \u003cp\u003eThe semi-structured clinical interview is based on the Positive SYmptoms and Diagnostic Criteria for the CAARMS [\u003cspan class=\"CitationRef\"\u003e73\u003c/span\u003e] Harmonized with the SIPS [\u003cspan class=\"CitationRef\"\u003e74\u003c/span\u003e] (PSYCHS) [\u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e] interview. The PSYCHS is a measure to evaluate attenuated positive symptoms in psychosis risk based on a harmonized Comprehensive Assessment of At-Risk Mental States (CAARMS) [\u003cspan class=\"CitationRef\"\u003e73\u003c/span\u003e] and Structured Interview for Psychosis-Risk Syndromes [\u003cspan class=\"CitationRef\"\u003e74\u003c/span\u003e].\u003c/p\u003e\n \u003ctable id=\"Tab1\" border=\"1\"\u003e\u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003eThe PSYCHS language samples were collected at screening, baseline and then monthly. The PSYCHS semi-structured interview lasted 30 minutes or longer; but only the first 30 minutes were transcribed.\u003c/p\u003e\n \u003cp\u003eAcross the two, semi-structured interviews are generally considered more consistent [\u003cspan class=\"CitationRef\"\u003e48\u003c/span\u003e] and codable [\u003cspan class=\"CitationRef\"\u003e49\u003c/span\u003e] whereas open-ended interviews may elicit more fine-grained details and analysis [\u003cspan class=\"CitationRef\"\u003e50\u003c/span\u003e]. The use of two different types of interviews, a semi-structured clinical-screening interview vs. an open-ended interview, allows us to gain different linguistic samples and NLP features which will facilitate comparison across the different methods and prompting techniques.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n \u003ch2\u003e2.3. Speech and Linguistic Features\u003c/h2\u003e\n \u003cp\u003eWe reviewed previous studies that applied automated natural language processing methods to analyze speech. With reference to Fig.\u0026nbsp;1, subsequently, in alignment with existing works [\u003cspan class=\"CitationRef\"\u003e51\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e], we used Python and a commonly used NLP package, Stanza [\u003cspan class=\"CitationRef\"\u003e52\u003c/span\u003e], to extract a wide range of syntactic features ranging from part-of-speech (POS) tagging, named entity recognition (NER) to overall lexical characteristics such as word frequency and sentiment. Subsequently, two different data pre-processing methods were adopted as outlined below.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\n \u003ch2\u003e2.4. Data Pre-processing\u003c/h2\u003e\n \u003cp\u003eNext, with reference to Fig.\u0026nbsp;1, we adopted two data pre-processing methods to facilitate comparison.\u003c/p\u003e\n \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e\n \u003ch2\u003e2.4.1. Approach 1: Principal Component Analysis (PCA)\u003c/h2\u003e\n \u003cp\u003eIn alignment with the work done by Tang \u003cem\u003eet al.\u003c/em\u003e [\u003cspan class=\"CitationRef\"\u003e51\u003c/span\u003e], in order to represent the variance from linguistic features without an assumption about the underlying latent constructs, we normalized and performed a principal component analysis (PCA) on the data. Another benefit of applying feature reduction is to prevent model over-fitting. This resulted in a total of 6 six features representing a total of 82% of the total variance.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec10\" class=\"Section3\"\u003e\n \u003ch2\u003e2.4.2. Approach 2: Feature Selection (FS)\u003c/h2\u003e\n \u003cp\u003eThe second pre-processing method entailed selecting the speech variables or features that were statistically significantly different between the CHR and CC groups, as done in [\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e]. As summarized in Tables \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, this resulted in 14 features used for the OPEN transcripts and 21 features used for the PSYCHS transcripts respectively. We used Cohen\u0026rsquo;s d to evaluate the effect size calculation. In general, a Cohen\u0026rsquo;s d value of around 0.2 is considered a small effect, 0.5 a medium effect and 0.8 and above is considered a large effect.\u003c/p\u003e\n \u003ctable id=\"Tab2\" border=\"1\"\u003e\u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\n \u003ctable id=\"Tab3\" border=\"1\"\u003e\u003c/table\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n \u003ch2\u003e2.5. Machine Learning Models\u003c/h2\u003e\n \u003cp\u003eWe performed data balancing via Synthetic Minority Over-sampling Technique (SMOTE) and conducted experiments across\u003c/p\u003e\n \u003cp\u003efour different classifiers: (i) Support Vector Machine (SVM), (ii) multi-layer perceptron (MLP), (iii) logistic regression (LR) and (iv) Gradient Boosted Decision Trees (GBDT). The models were selected in alignment with existing works [\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e]. Support vector machine is a widely used algorithm for binary classification tasks and performs well in scenarios with limited training data and a large feature set. Gradient boosted decision trees (GBDT) is a powerful ensemble technique that builds decision trees sequentially, with each tree correcting the residual errors of the previous ones. We implemented GBDT using CatBoost, a fast, scalable, and high performance GBDT library [\u003cspan class=\"CitationRef\"\u003e53\u003c/span\u003e]. We selected these models for 2 primary reasons. First, SVM and decision trees are 2 widely employed classifiers and this allows comparability with previous studies. Second, they differ in complexity and overfitting risk. SVMs include explicit regularization and tend to generalize well when the number of features exceeds the number of samples. CatBoost, in contrast, can capture more complex interactions but is more prone to overfitting under such conditions [\u003cspan class=\"CitationRef\"\u003e54\u003c/span\u003e]. Comparing their performance allows us to assess whether model complexity impacts generalization [\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e] across the different data pre-processing method and interview transcripts. Further details on the training parameters are available within the Supplement.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\n \u003ch2\u003e2.6. Model Performance Evaluation\u003c/h2\u003e\n \u003cp\u003eIn alignment with existing work [\u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e55\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e56\u003c/span\u003e], we evaluate model performance using the (i) root mean squared error (RMSE), (ii) mean absolute error (MAE), and (iii) Pearson\u0026rsquo;s correlation coefficient (PCC) between model-estimated scores and the ratings provided by clinicians.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\n \u003ch2\u003e2.7. LLMs used\u003c/h2\u003e\n \u003cp\u003eAs illustrated in Fig.\u0026nbsp;1, large language models (LLMs) are employed to analyze text to detect linguistic features related to psychotic symptoms. Given the increasing scrutiny of the ethics and privacy concerns of using closed-source LLMs [\u003cspan class=\"CitationRef\"\u003e42\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e43\u003c/span\u003e] and the growing calls to focus on open-source models [\u003cspan class=\"CitationRef\"\u003e44\u003c/span\u003e], within the context work, we chose to work with open-source models in order to facilitate transparent research. Moreover, as yet, no LLM is approved by the FDA [\u003cspan class=\"CitationRef\"\u003e57\u003c/span\u003e] and the use of public or proprietary LLMs like ChatGPT risks invoking Health Insurance Portability and Accountability Act (HIPAA) non-compliance. We experimented with two LLMs from two different families outlined below. All experiments are conducted locally using a 4xNVIDIA DGX A100 GPU instance.\u003c/p\u003e\n \u003cdiv id=\"Sec14\" class=\"Section3\"\u003e\n \u003ch2\u003e2.7.1. Llama 2\u003c/h2\u003e\n \u003cp\u003eWe used the highly popular open-source LLM by Meta, Llama 2, as the first benchmark [\u003cspan class=\"CitationRef\"\u003e58\u003c/span\u003e]. It has been adopted and deployed for mental health assessment [\u003cspan class=\"CitationRef\"\u003e59\u003c/span\u003e] and intervention [\u003cspan class=\"CitationRef\"\u003e60\u003c/span\u003e]. Llama 2 has been pre-trained on 2 trillion tokens and fine-tuned on over on 1\u0026nbsp;million new human annotations using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). We chose the 7B version in alignment with existing work [\u003cspan class=\"CitationRef\"\u003e61\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e62\u003c/span\u003e] and specifically worked with the Llama 2-chat version which is optimized for dialogue use cases.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec15\" class=\"Section3\"\u003e\n \u003ch2\u003e2.7.2. Qwen 2.5\u003c/h2\u003e\n \u003cp\u003eThe second model we used is from the Qwen family [\u003cspan class=\"CitationRef\"\u003e63\u003c/span\u003e], specifically the Qwen 2.5-Instruct. It is a quantized model capable of performing faster computation with less memory storage. Qwen 2 has been evaluated across various settings such as mental health crisis detection [\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e] and suicidal ideation [\u003cspan class=\"CitationRef\"\u003e46\u003c/span\u003e].\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e\n \u003ch2\u003e2.8. Prompting Methodology\u003c/h2\u003e\n \u003cp\u003eIn alignment with existing work [\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e64\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e] we employed zero-shot prompting, with an English prompt that includes (i) task description, (ii) evaluation criteria, and (iii) conversation content. The prompt structure is outlined as follow:\u003c/p\u003e\n \u003cp\u003e\u003cem\u003ePrompt\u003c/em\u003e \u003csub\u003e\u0026nbsp;\u003cem\u003eZS\u003c/em\u003e\u0026nbsp;\u003c/sub\u003e = \u003cem\u003eInputTranscripts\u003c/em\u003e\u0026thinsp;+\u0026thinsp;\u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e1\u0026minus;\u003cem\u003eS\u003c/em\u003e\u003c/sub\u003e + \u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e2\u0026minus;\u003cem\u003eQ\u003c/em\u003e\u003c/sub\u003e + \u003cem\u003eOutput\u003c/em\u003e (1)\u003c/p\u003e\n \u003cp\u003ewhere \u003cem\u003eInputTranscripts\u003c/em\u003e are the original transcripts gathered through the two different interview protocols (i.e. open-ended and the semi-structured interview). \u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e1\u0026minus;\u003cem\u003eS\u003c/em\u003e\u003c/sub\u003e provides specific guidelines for the assessment or evaluation criteria to adopt. \u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e2\u0026minus;\u003cem\u003eQ\u003c/em\u003e\u003c/sub\u003e poses the question that we want the LLM to answer. \u003cem\u003eOutput\u003c/em\u003e determines the type of model output providing a brief rationale of their decision. The goal of the prompt is to provide the LLM with adequate information to make an \u003cem\u003einformed decision\u003c/em\u003e on the transcripts provided. Subsequently, we designed three different prompting strategies for \u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e1\u0026minus;\u003cem\u003eS\u003c/em\u003e\u003c/sub\u003e as illustrated below.\u003c/p\u003e\n \u003cdiv id=\"Sec17\" class=\"Section3\"\u003e\n \u003ch2\u003e2.8.1. Prompt 1: Neutral\u003c/h2\u003e\n \u003cp\u003eThe CHR syndrome for psychosis provides a well-established paradigm for the early detection and intervention of psychosis [\u003cspan class=\"CitationRef\"\u003e65\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e66\u003c/span\u003e]. The neutral prompt thus gives a broad context of CHR psychosis risk symptoms, such as unusual perceptual experiences, disorganized communication or social isolation, which were selected based upon clinical evidence and existing research [\u003cspan class=\"CitationRef\"\u003e67\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e68\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e69\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e70\u003c/span\u003e]. The specific prompt then becomes: \u0026ldquo;please analyze the following interview transcript and determine whether the individual demonstrates characteristics consistent with Clinical High Risk (CHR) for psychosis status. Consider the following areas in your analysis: Unusual perceptual experiences. Unusual thought content or beliefs. Disorganized communication. Decreased functioning or motivation. Social withdrawal or isolation.\u0026rdquo;\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec18\" class=\"Section3\"\u003e\n \u003ch2\u003e2.8.2. Prompt 2: NLP-based\u003c/h2\u003e\n \u003cp\u003eThe NLP-based prompt requires the LLM to conduct their assessment based on evidence derived from the extracted NLP features summarized in Tables \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e for the open and PSYCHS-based interviews respectively. This entails providing the LLMs with the features selected via the procedure outlined in Section 2.4.2. The features in Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e were provided to the LLMs when running the analysis across the open-interview transcripts. Correspondingly, the features in Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e were provided to the LLMs when running the NLP-based prompt across the PSYCHS-interview transcripts.\u003c/p\u003e\n \u003cp\u003eSpecifically, \u003cem\u003ePrompt\u003c/em\u003e\u003csub\u003e\u003cem\u003ePart\u003c/em\u003e1\u0026minus;\u003cem\u003eS\u003c/em\u003e\u003c/sub\u003e therefore becomes: \u0026ldquo;Using the computational linguistic features extracted from speech data, evaluate Clinical High Risk (CHR) status. Research indicates that individuals at CHR may show: Reduced semantic density.\u003c/p\u003e\n \u003cp\u003eIncreased speech graph disconnectivity. Altered syntactic complexity. Changes in affective language patterns. Decreased coherence in narrative structure.\u0026rdquo;\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec19\" class=\"Section3\"\u003e\n \u003ch2\u003e2.8.3. Prompt 3: PSYCHS-based\u003c/h2\u003e\n \u003cp\u003eThe PSYCHS-based prompt is structured exactly according to how a PSYCHS-based clinical-screening assessment is conducted. Specifically, \u003cem\u003ePromptPart\u003c/em\u003e1\u0026thinsp;\u0026minus;\u0026thinsp;\u003cem\u003eS\u003c/em\u003e therefore becomes: \u0026ldquo;Using the Psychological Screening for Clinical High Risk (PSYCHS) interview protocol, evaluate this individual\u0026rsquo;s CHR status. The PSYCHS assessment focuses on: Domain 1: Positive Symptoms (Attenuated) P1: Unusual thought content/delusional ideas P2: Suspiciousness/persecutory ideas P3: Grandiose ideas P4: Perceptual abnormalities/hallucinations P5: Disorganized communication Domain 2: Negative Symptoms N1: Social anhedonia N2: Avolition N3: Expression of emotion N4: Experience of emotion and self N5: Ideational richness N6: Occupational functioning Domain 3: Disorganization D1: Odd behavior/appearance D2: Bizarre thinking D3: Trouble with focus and attention D4: Personal hygiene.\u0026rdquo;\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"3. RESULTS","content":"\u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e3.1. NLP Feature Analysis Results\u003c/h2\u003e \u003cp\u003eWith reference to Tables\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, we see that the speech features that were statistically significantly different between the CHR and CC groups differ between the OPEN and PSYCHS interview transcripts. This suggests that interview structure has a significant impact on various linguistic features such as speech content, syntax and lexical complexity and thus should be used judiciously when trying to perform automated linguistic analysis of schizophrenia.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003e3.2. NLP Results\u003c/h2\u003e \u003cp\u003eWith reference to Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, across the \u003cem\u003eopen-interview transcripts\u003c/em\u003e, the NLP-feature selection method (approach 2) in general produced better predictive outcomes across the different ML methods. For PCA, CatBoost seems to be the best method overall as it achieves the highest accuracy (0.727) and F1-score (0.760). Across feature selection, MLP is the best method across accuracy (0.887) and F1 (0.940) but logistic regression may be a better method according to AUROC (0.772). Across the \u003cem\u003ePSYCHS\u003c/em\u003e-interview transcripts, there is no observable difference between the PCA-based and FS-based methods. In general, the ML-based methods largely provide accuracy results within a range of 0.60\u0026thinsp;\u0026minus;\u0026thinsp;0.90 and does not fluctuate too much across different models, settings and interview transcripts. With reference to Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, we see that across all measures, the standard deviation of the averaged values all fall within a range of 0.05\u0026thinsp;\u0026minus;\u0026thinsp;0.20.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e3.3. LLM Results\u003c/h2\u003e \u003cp\u003eAcross the LLM results in Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, we see an interesting trend where Llama in general performs much better than Qwen across text-based prompting, i.e. the P1: neutral-based prompt and P3: PSYCHS-based prompt whereas Qwen produces more accurate prediction outcomes across the NLP-based prompting (i.e. P2: NLP-based prompting). For instance, across the open transcripts, Llama P3 produce the best accuracy of 0.700 and the best AUROC of 0.658 across all the results from both models. The second best result is given by Qwen P2 which gives an accuracy of 0.600 and an AUROC of 0.500. Across the PSYCHS transcripts, we see a similar trend where Llama produced better prediction results across P1 (accuracy\u0026thinsp;=\u0026thinsp;0.780, F1\u0026thinsp;=\u0026thinsp;0.857, AUROC\u0026thinsp;=\u0026thinsp;0.812) and P3 (accuracy\u0026thinsp;=\u0026thinsp;0.880, F1\u0026thinsp;=\u0026thinsp;0.930, AUROC\u0026thinsp;=\u0026thinsp;0.751) and Qwen produced better prediction results across P2 (accuracy\u0026thinsp;=\u0026thinsp;0.860, F1\u0026thinsp;=\u0026thinsp;0.925). Overall, we see that LLMs perform better across the PSYCHS-based interview transcripts. We further discuss this phenomenon within the discussion section below.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section2\"\u003e \u003ch2\u003e3.4. NLP vs. LLM Results\u003c/h2\u003e \u003cp\u003eLooking at the results in Tables\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e and \u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, we see a distinct difference between the variability in results across the two different automated classification methods. First, the results using NLP and ML-based methods are largely more consistent and fluctuate less across the different ML methodologies. For instance, with reference to Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, most of the ML methods provide an accuracy greater than 0.80. The best performing results hover around 0.899 whereas the worst accuracy is around 0.613. We see that across all measures, the standard deviation of the averaged values all fall within a range of 0.05\u0026thinsp;\u0026minus;\u0026thinsp;0.20 with the most consistent being \u003cem\u003eprecision\u003c/em\u003e with a standard deviation of 0.056 and the highest being AUROC with a standard deviation of 0.193.\u003c/p\u003e \u003cp\u003eIn contrast, with reference to Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, LLM-based methods exhibited greater variability in outcome. To illustrate, the best performing setting using Llama-P3: PSYCHS-based prompting achieved an impressive outcome at an accuracy of 0.880 and F1-score of 0.930. However, the setting using Qwen-P3: PSYCHS-based prompting yielded an extremely low accuracy of 0.320 and F1 of 0.346. Across the averaged measures, we see that the standard deviations of the averaged values fall within a range of 0.121\u0026thinsp;\u0026minus;\u0026thinsp;0.379 with the most consistent being \u003cem\u003eAUROC\u003c/em\u003e with a standard deviation of 0.121 and the most variable being recall with a standard deviation of 0.379.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. DISCUSSION","content":"\u003cp\u003e\u003cstrong\u003eHow do LLMs compare against well-established NLP methods?\u003c/strong\u003e We see that across prediction performance, LLMs do not necessarily perform better than NLP-based methods. For instance, across the open transcripts, the use of any of the feature preprocessing option (PCA or FS) combined with any ML methodology is likely to outperform an LLM like Qwen. Across the open transcripts, the best NLP-based result is the feature selection combined with MLP which gives an accuracy and F1 of 0.887 and 0.901 respectively. On the other hand, the best LLM performance given by Llama using a PSYCHS-based prompt yielded an accuracy of 0.700 and F1 of 0.776.\u003c/p\u003e\n\u003ctable id=\"Tab4\" border=\"1\"\u003e\u003c/table\u003e\n\u003cp\u003e\u003c/p\u003e\n\u003cp\u003eMoreover, the NLP and ML-based methods are also likely to be more interpretable (i.e. easier to understand and interpret) [\u003cspan class=\"CitationRef\"\u003e71\u003c/span\u003e]. This is because we are able to understand the entire classification pipeline from when and how the features are selected (e.g. PCA, FS) and how the ML models perform the classification. As a result, such methods are often deemed more trustworthy than black-box approaches as researchers and clinicians are able to verify whether the outcomes were arrived at via a reasonable process or procedure [\u003cspan class=\"CitationRef\"\u003e72\u003c/span\u003e]. Within the context of this experiment setting, given that NLP and ML-based methods have greater diagnostic accuracy and are inherently more interpretable, it would seem that using NLP combined with ML-based methods would be a better option than adopting LLM-based methods. Our experiment also cautions against the blind adoption of LLMs without sufficient evaluation and understanding.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePerformance difference across different feature extraction methods and different prompt-engineering techniques\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eResearch efforts that rely on automated analysis and assessment of language and speech using NLP can vary in terms of their technical approaches [\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e6\u003c/span\u003e]. For instance, in the context of psychosis, some works have focused on using latent semantic analysis (LSA) and word-embeddings extracted using neural network models to assess and evaluate \u003cem\u003esemantic coherence\u003c/em\u003e whereas other works focused on analyzing the syntactic structure and changes in sentence patterns using part-of-speech (POS) tagging [\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e]. Our experimental results seem to indicate that even if there is variation in terms of the predictive performance of the ML methods trained using different feature selection methods, the results will still largely fall within a reasonable range and remain largely consistent across different ML methods employed.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eHowever, this is not true for LLM-based assessment outcomes. We witness an interesting trend where each model type consistently performs better across certain prompt-engineering methods. For instance, Qwen consistently produces more accurate assessment across the NLP-based prompting method. whereas Llama is often better across the neutral and PSYCHS-based prompting. One potential reason for this is that Llama is trained for dialogue use and is optimized to work with text-based input and textual reasoning [\u003cspan class=\"CitationRef\"\u003e58\u003c/span\u003e] whereas Qwen is optimized to work across numerical reasoning. This has significant implications for the clinical community. Despite the proliferation of using LLMs to assess and predict mental health [\u003cspan class=\"CitationRef\"\u003e40\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e41\u003c/span\u003e], insufficient research has been devoted towards understanding which LLM to use for what input and for what use cases. Thus, clinicians and researchers may need to devote greater effort to investigate which model may work best given their input setting. For instance, it may be better to choose Llama to analyze a text-based transcript and it may be better to choose Qwen to analyze digital phenotyping data.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHow generalizable are the findings across different interview formats:?\u003c/strong\u003e Another noteworthy aspect is that the efficacy of NLP combined with ML-based methods seems to be more consistent and generalizable across the different transcript types (i.e. open vs. PSYCHS-based interview). This suggest that the NLP-based methods are picking up on the more reliable and consistent signals when attempting to distinguish between the community controls and CHR individuals. We see that despite the difference in diagnostic accuracy, all the ML-based methods largely provide accuracy results within a range of 0.60 − 0.90 across all models and interview formats (i.e. open vs. PSYCHS-based). Overall, compared to NLP-based methods, LLMs are less consistent and display greater variation depending on the input transcript type and LLM used (i.e. Llama vs. Qwen) and prompting strategy (i.e. neutral prompt, NLP-based prompt and PSYCHS-based prompting).\u003c/p\u003e\n\u003cp\u003eWe hypothesize that this could be due to how different interview methods may naturally lead to different text-based output. Given how NLP-based methods typically rely on quantifiable metrics such as text-to-token ratio etc. to assess the presence (or absence) of quantifiable linguistic features associated with CHR symptoms, it may be more consistent and robust against changes in semantic content of the input transcript. On the other hand, LLMs typically conduct their analysis based on the semantic content of the input transcript and will thus be more susceptible to performance changes when presented with a transcript that is generic (e.g. the open-ended language samples) vs. a language sample that specifically discusses semantic material related to CHR symptoms. In other words, LLMs pick up on the verbalized symptoms presented within the transcript rather than rely on objective linguistic measures like NLP-based methods. This is supported by our results which showed that LLMs generally perform much better across the PSYCHS-based interview transcripts rather than the open-ended language samples.\u003c/p\u003e\n\u003cp\u003eThis finding has several implications. First, this suggests that NLP-based methods may be more reliable and objective method than LLMs. Second, the usage of LLM will be suitable if the transcript content provides adequate semantic content for the LLM to perform contextualized reasoning. Third, even if LLMs are utilized within such settings, there is still a need to investigate and evaluate which model and prompting technique are best suited for the task in order to ensure reliable and trustworthy outputs.\u003c/p\u003e\n\u003cdiv id=\"Sec26\" class=\"Section2\"\u003e\n \u003ch2\u003e4.1. Limitation\u003c/h2\u003e\n \u003cp\u003eWe did not run experiments using other LLMs. Another significant limitation is that we did not have the chance to explore many important limitations of leveraging LLMs in mental health settings such as problem of bias and interpretability. We encourage future work to investigate such issues. We wish to caution that technical results do not translate to real-world application and deployability. Moreover, we have only conducted experiments within a zero-shot context. Future work can replicate the above experiments using other prompt engineering techniques and deployment methodologies such as few-shot prompting.\u003c/p\u003e\n\u003c/div\u003e\n"},{"header":"Conclusion","content":"\u003cp\u003eAcross different input interview types, NLP-based methods seem more reliable and consistent across different feature extraction methods compared with LLM-based methods across different prompt engineering techniques. LLMs are better at analyzing text which provides semantic-based content or signal for automated analysis. Even though the usage of LLMs seem promising, there is still a need to investigate which model and which prompting methodology to utilize in order to ensure consistent and reliable results.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eCheryl Mary Corcoran and, Guillermo A, Cecchi (2020) Using language processing and speech analysis for the identification of psychosis and other disorders. Biol Psychiatry: Cogn Neurosci Neuroimaging 5(8):770\u0026ndash;779\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMurray Alpert RJ, Shaw ER, Pouget, Kelvin O, Lim (2002) A comparison of clinical ratings with vocal acoustic measures of flat affect and alogia. J Psychiatr Res 36(5):347\u0026ndash;353\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlex S, Cohen KR, Mitchell, Elvevag B (2014) What do we really know about blunted vocal affect and alogia? a˚ meta-analysis of objective assessments. Schizophr Res 159:2\u0026ndash;3\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheryl M, Corcoran VA, Mittal CE, Bearden RE, Gur K, Hitczenko Z, Bilgrami A, Savic GA, Cecchi, Wolff P (2020) Language as a biomarker for psychosis: a natural language processing approach. Schizophr Res 226:158\u0026ndash;166\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRybner A, Jessen ET, Damsgaard Mortensen M, Nyhus Larsen S, Zhou Y, Koelkebeck K, Bliksted V, Fusaroli R (2023) Vocal markers of schizophrenia: assessing the generalizability of machine learning models and their clinical applicability. Eur Psychiatry 66(S1):S186\u0026ndash;S186\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eParola A, Simonsen A, Lin JM, Zhou Y, Wang H, Ubukata S, Koelkebeck K, Bliksted V, Fusaroli R (2023) Voice patterns as markers of schizophrenia: building a cumulative generalizable approach via a cross-linguistic and meta-analysis based investigation, \u003cem\u003eSchizophrenia bulletin\u003c/em\u003e, vol. 49, no. Supplement 2, pp. S125\u0026ndash;S141\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatiah Llerena GP, Strauss, Alex S, Cohen (2012) Looking at the other side of the coin: a meta-analysis of selfreported emotional arousal in people with schizophrenia. Schizophr Res 142:1\u0026ndash;3\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlex S, Cohen T, Cowan TP, Le, Elana K, Schwartz B, Kirkpatrick IM, Raugh HC, Chapman, Strauss GP (2020) Ambulatory digital phenotyping of blunted affect and alogia using objective facial and vocal analysis: proof of concept. Schizophr Res 220:141\u0026ndash;146\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlex S, Cohen CR, Cox, Thanh P, Le T, Cowan MD, Masucci GP, Strauss, Kirkpatrick B (2020) Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia. npj Schizophrenia 6(1):26\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlex S, Cohen KR, Mitchell NM, Docherty, Horan WP (2016) Vocal expression in schizophrenia: Less than meets the ear. J Abnorm Psychol 125(2):299\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsher Cohen D, Joshi A, Bondre PK, Chand N, Chaturvedi S, Choudhary S, Dutt A, Khan C, Langholm M, Kumar et al (2024) Digital phenotyping correlates of mobile cognitive measures in schizophrenia: A multisite global mental health feasibility trial. PLOS Digit Health 3(6):e0000526\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNicholas Cummins M, Arribas T, Gibbs-Dean E, Molina D, Sethi M, Kempton S, Morgan T, Spencer K, Diederen, Olah J (2023) Towards a scalable approach to assess speech organization across the psychosis-spectrum-online assessment in conjunction with automated transcription and extraction of speech measures\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsher Cohen JA, Naslund S, Chang S, Nagendra A, Bhan A, Rozatkar J, Thirthalli A, Bondre D, Tugnawat PV, Reddy et al (2023) Relapse prediction in schizophrenia with smartphone digital phenotyping during covid-19: a prospective, three-site, two-country, longitudinal study, \u003cem\u003eSchizophrenia\u003c/em\u003e, vol. 9, no. 1, p. 6\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEric YH, Chen, Linda CW, Lam CS, Kan CSKY, Chan CL, Kwok DGH, Nguyen, Chen RYL (1996) Language disorganisation in schizophrenia: validation and assessment with a new clinical rating instrument. East Asian Archives Psychiatry 6(1):4\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNeguine Rezaii E, Walker, Wolff P (2019) A machine learning approach to predicting psychosis using semantic density and latent content analysis. NPJ schizophrenia 5(1):9\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAguila AF-BDD, Cerda M, Gaspar PA, Terissi LD (2022) Manuel Duran, and Camila\u0026acute; Valderrama, Automatic language analysis identifies and predicts schizophrenia in first-episode of psychosis. Schizophrenia 8(1):53\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlban E, Voppel, Janna N, de Boer SG, Brederoo HG, Schnack, Iris EC, Sommer (2021) Quantified language connectedness in schizophrenia-spectrum disorders. Psychiatry Res 304:114130\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGillinder Bedi F, Carrillo GA, Cecchi DF, Slezak M, Sigman Nat\u0026acute;aliaB, Mota Sidarta\u0026acute;, Ribeiro DC, Javitt M, Copelli, Cheryl M, Corcoran (2015) Automated analysis of free speech predicts psychosis onset in high-risk youths. npj Schizophrenia 1(1):1\u0026ndash;7\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheryl M, Corcoran F, Carrillo D, Fernandez-Slezak G, Bedi C, Klim DC, Javitt, Carrie E\u0026acute; Bearden, and, Guillermo A, Cecchi (2018) Prediction of psychosis across protocols and risk cohorts using automated language analysis, \u003cem\u003eWorld Psychiatry\u003c/em\u003e, vol. 17, no. 1, pp. 67\u0026ndash;75\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTakeshi Nakaura R, Ito D, Ueda T, Nozaki Y, Fushimi Y, Matsui M, Yanagawa A, Yamada T, Tsuboyama N, Fujima et al (2024) The impact of large language models on radiology: a guide for radiologists on the latest innovations in ai. Japanese J Radiol 42(7):685\u0026ndash;696\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBetzler BK, Chen H, Cheng C-Y, Cecilia S, Lee G, Ning SJ, Song AY, Lee R, Kawasaki P, van Wijngaarden A, Grzybowski et al (2023) Large language models and their impact in ophthalmology. Lancet Digit Health 5(12):e917\u0026ndash;e924\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVolkmer S, Meyer-Lindenberg A, Schwarz E (2024) Large language models in psychiatry: Opportunities and challenges. Psychiatry Res 339:116026\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu X, Yao B, Dong Y, Gabriel S, Yu H, Hendler J, Ghassemi M, Dey AK, Wang D (2024) Mental-llm: Leveraging large language models for mental health prediction via online text data, \u003cem\u003eProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies\u003c/em\u003e, vol. 8, no. 1, pp. 1\u0026ndash;32\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuifeng Deng S, Rao T, Lin A, Dai P, Wang J, Xie H, Song K, Zhao D, Xu Z, Cheng et al (2025) Evaluating large language models in crisis detection: A real-world benchmark from psychological support hotlines, \u003cem\u003earXiv preprint arXiv:2506.01329\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJustin Khasentino A, Belyaeva X, Liu Z, Yang NA, Furlotte C, Lee E, Schenck Y, Patel et al (2025) Jian Cui, Logan Douglas Schneider,., A personal health large language model for sleep and fitness coaching, \u003cem\u003eNature Medicine\u003c/em\u003e, pp. 1\u0026ndash;10\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTingyu Mo, Jacqueline CK, Lam VOK, Li, Cheung LYL (2024) Leveraging large language models for identifying interpretable linguistic markers and enhancing alzheimer\u0026rsquo;s disease diagnostics. medRxiv, pp. 2024\u0026ndash;2008\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNeguine Rezaii D, Hochberg M, Quimby B, Wong M, Brickhouse A, Touroutoglou BC, Dickerson, Wolff P (2024) Artificial intelligence classifies primary progressive aphasia from connected speech. Brain 147(9):3070\u0026ndash;3082\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu C-M, Chan Y-H, Ho M-Y, Liu C-C, Lu M-H, Liao Y-A (2025) Ming-Hsien Hsieh, and Yufeng Jane Tseng, Analyzing generative ai and machine learning in auto-assessing schizophrenia\u0026rsquo;s negative symptoms. Schizophr Bull, p. sbaf102\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi R, Cao M, Fu D, Wei W, Wang D, Yuan Z, Hu R, Deng W (2024) Deciphering language disturbances in schizophrenia: A study using fine-tuned language models. Schizophr Res 271:120\u0026ndash;128\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArabzadeh N, Clarke CLA (2025) A human-ai comparative analysis of prompt sensitivity in llm-based relevance judgment, in \u003cem\u003eProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval\u003c/em\u003e, pp. 2784\u0026ndash;2788\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDonald A, Young Z, Campbell KK, Zakzanis, Weinstein E (2003) A comparison between an interview and a self-report method of insight assessment in chronic schizophrenia. Schizophr Res 63:1\u0026ndash;2\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eScott W, Woods S, Parker MJ, Kerr, Barbara C, Walsh SA, Wijtenburg N, Prunier AR, Nunez et al (2024) Kate Buccilli, Catalina Mourgues-Codern, Kali Brummitt,., Development of the psychs: Positive symptoms and diagnostic criteria for the caarms harmonized with the sips, \u003cem\u003eEarly Intervention in Psychiatry\u003c/em\u003e, vol. 18, no. 4, pp. 255\u0026ndash;272\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMaayan KS-WN, Bergman H, Davenport C, Amanda J, Kirkham S, Grabowski, Clive E, Adams (2015) First rank symptoms for schizophrenia. Cochrane Database Syst Reviews,, 1\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTashrif Billah KIK, Cho O, Borders Y, Chung M, Ennis GR, Jacobs E, Liebenthal DH, Mathalon D, Mohandass, Spero C, Nicholas et al (2025) Enabling fair data stewardship in complex international multi-site studies: Data operations for the accelerating medicines partnership\u0026reg; schizophrenia program. Schizophrenia 11(1):55\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMlay JP, Naidu T, Ramlall S, Patankar K, Israel K, Esquivel L, Curran L, Mhlungu SS, Zondi M, Lessells R et al (2025) Strategies for relapse prevention among people with schizophrenia in kwazulu-natal province, south africa: HealtCCare providers\u0026rsquo; perspectives. PLoS ONE 20(3):e0316313\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThomas S, Christy (2025) Screening for prodromal symptoms of schizophrenia in primary care settings. Lynchburg J Med Sci 1(1):8\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKho JJ, Song S, Tan SMX, Fitriyah NH, Lokadjaja MC, Yee JY, Yang Z, Chen EYH, Lee J, Wen W, Goh B (2025) Leveraging computational linguistics and machine learning for detection of ultra-high risk of mental health disorders in youths. Schizophrenia 11(1):98\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGumus M, Koo M, Studzinski CM, Bhan A, Robin J, Black SE (2024) Linguistic changes in neurodegenerative diseases relate to clinical symptoms. Front Neurol 15:1373341\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarion C, Leaman, Lisa A, Edmonds (2023) Analyzing language in the picnic scene picture and in conversation: The type of discourse sample we choose influences findings in people with aphasia. Am J Speech-Language Pathol 32(4):1413\u0026ndash;1430\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAli M, Lucasius C, Patel TP, Aitken M, Vorstman J, Szatmari P, Battaglia M, Kundur D (2025) Speech as a multimodal digital phenotype for multi-task llm-based mental health prediction, \u003cem\u003earXiv preprint arXiv:2505.23822\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJared Moore D, Grabb W, Agnew K, Klyman S, Chancellor DC, Ong, Haber N Expressing stigma and inappropriate responses prevents llms from safely replacing mental health providers., in \u003cem\u003eProceedings of the\u003c/em\u003e (2025) \u003cem\u003eACM Conference on Fairness, Accountability, and Transparency\u003c/em\u003e, 2025, pp. 599\u0026ndash;627\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTala Mirzaei L, Amini, Esmaeilzadeh P (2024) Clinician voices on ethics of llm integration in healtCCare: a thematic analysis of ethical concerns and implications. BMC Med Inf Decis Mak 24(1):250\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEsmaeilzadeh P (2025) Ethical implications of using general-purpose llms in clinical settings: a comparative analysis of prompt engineering strategies and their impact on patient safety. BMC Med Inf Decis Mak 25(1):342\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFabio Dennstadt J, Hastings PM, Putora M, Schmerder, Cihoric N (2025) Implementing large language\u0026uml; models in healtCCare while balancing control, collaboration, costs and security. NPJ Digit Med 8(1):143\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang T, Teng S, Jia H (2024) and Simon D\u0026rsquo;Alfonso, Leveraging llms to predict affective states via smartphone sensor features, in \u003cem\u003eCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing\u003c/em\u003e, pp. 709\u0026ndash;716\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJoao Pedro C, Azevedo, Adonias C, de Oliveira F, Silva L, Coutinho, Teles AS Harnessing˜ generative llms to detect and explain suicidal ideation in brazilian portuguese texts, in (2025) \u003cem\u003eIEEE 38th International Symposium on Computer-Based Medical Systems (CBMS)\u003c/em\u003e. IEEE, 2025, pp. 15\u0026ndash;20\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCassandra MJ, Wannan B, Nelson J, Addington K, Allott A, Anticevic C, Arango, Justin T, Baker CE, Bearden T, Billah S, Bouix et al (2024) Accelerating medicines partnership\u0026reg; schizophrenia (amp\u0026reg; scz): rationale and study design of the largest global prospective cohort study of clinical high risk for psychosis. Schizophr Bull 50(3):496\u0026ndash;512\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilliam C, Adams (2015) Conducting semi-structured interviews, \u003cem\u003eHandbook of practical program evaluation\u003c/em\u003e, pp. 492\u0026ndash;505\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaglam Y (2024) Which data gathering method is superior: An open-ended questionnaire or a semi-structured interview? Int J Stud Educ (IJonSE), 6, 3\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRahul S, Chauhan (2022) Unstructured interviews: are they really all that bad? Hum Resource Dev Int 25(4):474\u0026ndash;487\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSunny X, Tang MJ, Spilka M, John ML, Birnbaum E, Saito SA, Berretta LM, Behbehani, Mark Y, Liberman, Anil K, Malhotra W, Simpson et al (2025) Automated speech and language markers of longitudinal changes in psychosis symptoms. NPP\u0026mdash;Digital Psychiatry Neurosci 3(1):13\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQi P, Zhang Y, Zhang Y, Bolton J, Manning CD Stanza: A python natural language processing toolkit for many human languages, \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:(2003). \u003cem\u003e07082\u003c/em\u003e, 2020\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiudmila Prokhorenkova G, Gusev A, Vorobev AV, Dorogush, Gulin A (2018) Catboost: unbiased boosting with categorical features. Adv Neural Inf Process Syst, 31\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVisar Berisha C, Krantsevich PR, Hahn S, Hahn G, Dasarathy P, Turaga, Liss J (2021) Digital medicine and the curse of dimensionality. NPJ Digit Med 4(1):153\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang Y-J, Lin Y-T, Liu C-C, Lee L-E, Hung S-H, Lo J-K, Fu L-C (2022) Assessing schizophrenia patients through linguistic and acoustic features using deep learning techniques. IEEE Trans Neural Syst Rehabil Eng 30:947\u0026ndash;956\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMina Bishay P, Palasek S, Priebe, Patras I (2021) Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Trans Affect Comput 12(4):949\u0026ndash;961\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGary E, Weissman T, Mankowitz, Genevieve P, Kanter (2025) Unregulated large language models produce medical device-like output. NPJ Digit Med 8(1):148\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHugo Touvron L, Martin K, Stone P, Albert A, Almahairi Y, Babaei N, Bashlykov S, Batra P, Bhargava S, Bhosale et al (2023) Llama 2: Open foundation and fine-tuned chat models, \u003cem\u003earXiv preprint arXiv:2307.09288\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZijing Yang E, Alveyn M, Dey N, Arumalla MD, Russell S, Norton, Galloway JB (2024) Impact of visualising healtCCare quality performance: a systematic review. BMJ open 14(11):e083620\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGopika S, Kalaiselvi K, Jacob M, Vignesh K (2024) Empowering mental health resilience: Llama 2-based ai model for all stages. in Revitalizing Health Through Humanities. Routledge, pp 402\u0026ndash;409\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSarah Sandmann S, Riepenhausen L, Plagwitz, Varghese J (2050) Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks, \u003cem\u003eNature communications\u003c/em\u003e, vol. 15, no. 1, pp. 2024\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNivethitha V, Verma N, Bokadia R, Jaju R, Fine-tuning llama-2-7b model for conversation summarization, in (2024) \u003cem\u003e15th International Conference on Computing Communication and Networking\u003c/em\u003e Technologies \u003cem\u003e(ICCCNT)\u003c/em\u003e. IEEE, 2024, pp. 1\u0026ndash;7\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBinyuan Hui J, Yang Z, Cui J, Yang D, Liu L, Zhang T, Liu J, Zhang B, Yu K, Lu et al (2024) Qwen2. 5-coder technical report, \u003cem\u003earXiv preprint arXiv:2409.12186\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFlathers M, Xia W, Hau C, Nelson BW, Cheong J, Burns J, Torous J (2025) Interpreting psychiatric digital phenotyping data with large language models: a preliminary analysis. BMJ Mental Health, 28, 1\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Pablo PF-PGS, Christoph U, Correll A, Meyer-Lindenberg MJ, Millan S, Borgwardt S, Galderisi A, Bechdolf A, Pfennig (2020) Prevention of psychosis: advances in detection, prognosis, and intervention. JAMA psychiatry 77(7):755\u0026ndash;765Lars Vedel Kessing\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAna Catalan GS, de Pablo JV, Serrano P, Mosillo H, Baldwin U, Correll I, Bonoldi et al (2021) Annual research review: Prevention of psychosis in adolescents\u0026ndash;systematic review and meta-analysis of advances in detection, prognosis and intervention, \u003cem\u003eJournal of Child Psychology and Psychiatry\u003c/em\u003e, vol. 62, no. 5, pp. 657\u0026ndash;673\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHussein Hijazi, Moukaddam N (2025) Schizophrenia \u0026amp; psychosis. prodromal symptoms of schizophrenia: Understanding and addressing challenges. Psychiatric Times, 42, 6\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen Y-F, Yeh T-P, Hsu H-C, Lane H-Y, Lu S-H, Ma W-F (2025) Clinical symptoms of the prodromal stage preceding serious mental illness in taiwanese young adults: a qualitative study. BMC Psychiatry 25(1):187\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTandy J, Miller TH, McGlashan SW, Woods K, Stein N, Driesen CM, Corcoran R, Hoffman, Davidson L (1999) Symptom assessment in schizophrenic prodromal states. Psychiatr Q 70(4):273\u0026ndash;287\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKlosterkotter J, Hellmich M, Steinmeyer EM, Schultze-Lutter F (2001) Diagnosing schizophrenia\u0026uml; in the initial prodromal phase. Arch Gen Psychiatry 58(2):158\u0026ndash;164\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDan W, Joyce A, Kormilitzin KA, Smith, Cipriani A (2023) Explainable artificial intelligence for mental health through transparency and interpretability for understandability. npj Digit Med 6(1):6\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDmytro Savchuk, Doroshenko A (2025) Explainable ai methods to increase trustworthiness in healtCCare. i\u003cem\u003en Responsible and Explainable Artificial Intelligence in HealtCCare\u003c/em\u003e. Elsevier, pp 55\u0026ndash;89\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlison R, Yung AR, Yung HP, Yuen PD, Mcgorry, Lisa J, Phillips D, Kelly M, Dell\u0026rsquo;Olio SM, Francey EM, Cosgrave E, Killackey et al (2005) Mapping the onset of psychosis: the comprehensive assessment of at-risk mental states. Aust N Z J Psychiatry 39:11\u0026ndash;12\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThomas H, McGlashan BC, Walsh, Scott W, Woods J, Addington K, Cadenhead T, Cannon, Walker E (2001) Structured interview for psychosis-risk syndromes. Yale School of Medicine, New Haven, CT\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Bangar A, Kalkan S, Gunes H (2025) U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection. In: Proceedings of the 4th Machine Learning for Health Symposium (Proceedings of Machine Learning Research). Vol. 259. PMLR, 203\u0026ndash;218\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Kalkan S, Gunes H. Causal structure learning of bias for fair affect recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 340\u0026ndash;349., Cheong J, Kalkan S (2023a) and H. Gunes. 2023b. Counterfactual fairness for facial expression recognition. In: ECCV 2022 Workshops. Springer, 245\u0026ndash;261\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Kalkan S, Gunes H (2024) FairReFuse: referee-guided fusion for multimodal causal fairness in depression detection. In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 7224\u0026ndash;7232\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Kalkan S, Gunes H (2021) The Hitchhiker\u0026rsquo;s Guide to Bias and Fairness in Facial Affective Signal Processing: Overview and techniques. IEEE Signal Process Mag 38(6):39\u0026ndash;49. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/MSP.2021.3106619\u003c/span\u003e\u003cspan address=\"10.1109/MSP.2021.3106619\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Kuzucu S, Kalkan S, Gunes H (2023) Towards gender fairness for mental health prediction. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 5932\u0026ndash;5940\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Spitale M, Gunes H (2025) Small but Fair! Fairness for Multimodal Human-Human and Robot-Human Mental Wellbeing Coaching. IEEE Transactions on Affective Computing. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpitale M, Cheong J, Gunes H (2024) Underneath the numbers: Quantitative and qualitative gender fairness in llms for depression prediction. arXiv preprint arXiv:2406.08183\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Spitale M, Gunes H (2023), September It\u0026rsquo;s not Fair!\u0026ndash;Fairness for a Small Dataset of Multi-modal Dyadic Mental Well-being Coaching. In \u003cem\u003e2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)\u003c/em\u003e (pp. 1\u0026ndash;8). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheong J, Mogharabin A, Liang P, Gunes H, Kalkan S (2025b) Fairwell: Fair multimodal self-supervised learning for wellbeing prediction. arXiv preprint arXiv:2508.16748\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKwok AMH, Cheong J, Kalkan S, Gunes H (2025) Machine learning fairness for depression detection using eeg data. In 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), pp. 1\u0026ndash;5. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Castillo, J. Cheong, S. Choudhary, A. Bondre, A. R. Rozatkar, \u0026hellip; J. \u0026amp; Torous (2025).Mobile cognitive remote assessment of schizophrenia: a global multi-site pilot study.\u003cem\u003eSchizophrenia\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 144\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen K, Torous J, Cheong J (2025) The Current State/Trends in Digital Phenotyping for Mental Health Research and Care. Psychiatric Clinics\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKuzucu S, Cheong J, Gunes H, Kalkan S (2024) Uncertainty as a fairness measure. \u003cem\u003eJournal of Artificial Intelligence Research\u003c/em\u003e, \u003cem\u003e81\u003c/em\u003e, 307\u0026ndash;335.\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":" \u003cp\u003eTables are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Harvard Medical School","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"automated linguistic analysis, clinical high risk natural language processing, large language models","lastPublishedDoi":"10.21203/rs.3.rs-8777643/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8777643/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground and Hypothesis\u003c/strong\u003e: Research has indicated that linguistic features can be used for the early detection of schizophrenia. Given that traditional clinician-based assessment can be labor intensive and time-consuming, more research has turned towards the usage of automated means to extract and analyze the linguistic features of schizophrenia. However, most of these existing studies have chiefly focused on deploying the LLMs with no comparison against more well-established NLP-based methods. As a result, there is less insight on the utility of using LLMs and whether the benefits of using of LLMs for analysis outweighs the costs. Moreover, given LLM’s prompt sensitivity, there is also a lack of research investigating how different prompt engineering methods affect the different model’s output across different settings. Another longstanding open question within the field pertains to how best to objectively assess prodromal psychotic symptoms and how best to analyze the different transcripts. In this study, we systematically assess the efficacy of large language models (LLMs) and natural-language processing (NLP) methods to perform automated linguistic analysis of clinical high risk (CHR) psychotic symptoms. We seek to understand the reliability of using LLMs to analyze patient transcripts for the early identification of CHR individuals in comparison against more established NLP-based methods.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStudy Design\u003c/strong\u003e: We trained models using a large international dataset of 374 patients, of which 331 are clinically high risk (CHR) and 43 are community controls (CC). Two types of interviews were conducted: an open-ended and a semi-structured interview based on the Positive SYmptoms and Diagnostic Criteria for the CAARMS [73] Harmonized with the SIPS [74] (PSYCHS) protocol [32]. Trained research assistants carried out these interviews which were audio and video-recorded across different sites prior to October 13 2024. We used two different feature extraction methods, the principal component analysis (PCA) and feature selection (FS), and conducted experiments using four different machine learning (ML) models and two large language models (LLMs), namely Llama and Qwen. For each of the LLM, we used three different prompting strategies: a neutral prompt, an NLP based prompt and a PSYCHS interview-based prompt to better understand each LLM performance under different reasoning setting.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Across both the open and PSYCHS-based transcripts, the NLP combined with ML-based methods, which relies on objective quantifiable metrics, demonstrated fairly consistent results within a range of 0.60 − 0.90. This is in contrast to LLM-based methods, which provided highly variable results depending on the interview format and prompt used, with the lowest being 0.320 and the highest being 0.880 across all experimental settings. In general, both categories of methods seem to produce more accurate results using the PSYCHS-based transcripts. Llama generally performs better than text-based methods, which require semantic reasoning (e.g. the PSYCHS based prompt), and yielded the highest accuracy and F1 of 0.880 and 0.930 when used on the PSYCHS-based interview transcripts. On the other hand, Qwen generally performed better than numerical-reasoning based tasks (e.g. the NLP-based prompt) and performed the best across the PSYCHS-based interview transcripts with an accuracy and F1 of 0.880 and 0.930.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e: Overall, we find that NLP-based methods are more reliable and consistent. LLM-based methods are highly variable and do not demonstrate sufficient reliability. Their output differs greatly depending on the input transcript and prompt type provided. We suggest that more emphasis should be placed on developing interpretable and clinically grounded methods to automate linguistic analysis of schizophrenia. Further experiments need to be conducted before deploying such models for high-stakes use cases and for identifying more precise and automated methods to understand how clinical features of schizophrenia are expressed linguistically.\u003c/p\u003e","manuscriptTitle":"Automated Detection Of Clinical High Risk Population Of Schizophrenia: Assessing The Generalizability Of NLP And LLM-Based Methods","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2026-02-04 18:42:14","doi":"10.21203/rs.3.rs-8777643/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2026-02-04 08:33:29","doi":"10.21203/rs.3.rs-8777643/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"645ec54d-793f-4ebc-b9ad-4c4e15439974","owner":[],"postedDate":"February 4th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-02-04T08:33:31+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-04 18:42:14","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-8777643","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8777643","identity":"rs-8777643","version":["v2"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}