Inter-rater Reliability of an LLM in Predicting Depression Among Indian Adults

Inter-rater Reliability of an LLM in Predicting Depression Among Indian Adults | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Inter-rater Reliability of an LLM in Predicting Depression Among Indian Adults Shivangi Verma, Dr. Ashwani Pundeer, Dr. Soniya Vats This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7431380/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In recent years, the developments in Artificial intelligence (AI) has reshaped several industries and professions. Given the rising prevalence of depression and the inability of existing health infrastructure to address this, the current study was undertaken to investigate the potential of AI in clinical settings. The first step is assessing how language can provide insights into psychological states of individuals. For the same purpose RoBERTa, a transformers based deep learning model was fine-tuned on DAIC-WOZ dataset to predict depression among Indian adults. Additionally, interviews of Indian adults were conducted and analyzed by both the model and a clinical psychologist to predict indicators of depression. The model achieved a macro F1-score of 0.82 on test split of DAIC-WOZ, indicating robust performance despite class imbalance. Cohen’s kappa of 0.628 indicated a substantial agreement was reached between both the model and the rater on the interviews. However, as revealed by the thematic analysis and attribution scores for interviews which were disagreed upon, the model’s tendency to generate false positives highlights the need for enhanced contextual analysis. These findings reveal that the language of depression is universal in its essence while emphasizing the necessity for culturally tailored datasets and multimodal approaches to improve predictions in resource constraints. Depression AI Deep learning Mental health Natural Language Processing Figures Figure 1 Figure 2 1. INTRODUCTION An alarming trend has been observed in the last decade where mental health disorders have emerged as a growing global crisis affecting individuals from all age groups and socioeconomic backgrounds. Major depressive disorder (MDD), with its global prevalence of 5–17% is a mood disorder that is characterized by persistent low mood, anhedonia, loss of energy, poor concentration, appetite change, sleep disturbances, psychomotor agitation or retardation, suicidal ideation and worthlessness or guilt [ 2 ]. The World Health Organization (WHO) has reported MDD is likely to be the primary disease burden cause globally by 2030 following its ranking as the third cause of the global disease burden in 2008 [ 3 ]. In low-income nations, over 75% of people with mental illness are unable to access treatment options. Even in high-income nations, just one-third of those with depression receive treatment. The proportion of individuals receiving minimally adequate treatment is highly variable, from around 23% in high-income nations to as low as 3% in low- and lower-middle-income nations. The origin of this lacuna does not just belong to infrastructure and medical facilities but to the general population's awareness, as well as that of affected populations. Several patients with depression first go for medical care to address bodily grievances instead of attending a mental health specialist. Patients in almost half of such situations do not say they feel depressed or experience low mood [ 3 ]. Consequently, screening for depression in primary care is important for early detection and treatment. Screening, assessment and diagnosis are separate but important processes in healthcare. Screening is performed to identify early signs of a disorder through chief complaints in the majority of cases, facilitating timely treatment and prevention. Evaluation is a more overall and thorough analysis of an individual's overall wellbeing, where some scales or tools utilized along with taking into account medical history, physical examinations and others. Diagnosis is the ultimate step, establishing a particular disorder or comorbidity of this disorder when the overall clinical picture is considered. Screening is particularly significant as it identifies problems early before they even reach their serious form, resulting in improved prognosis, treatment and reduced healthcare expense. Unfortunately, the existing instruments for assessment do not consider patient's entire history of presenting with psychological symptoms and thus frequently leading to errors and delay in diagnosis [ 8 ]. For instance, psychiatric wards have erroneously admitted patients suffering from Long COVID because the two conditions have some overlapping symptoms [ 13 ]. Standard clinical procedures such as administration of inventories and unstructured interviews fail to identify early or minimal signs of depression due to the existence of elements such as interviewer bias in asking symptoms, patient bias in describing symptoms, time constraints and over-reliance on subjective judgment. This is where deep and machine learning algorithms can come in to enable making correct and objective judgments while considering multiform sources of information unlike conventional instruments which use information from a few sources and forms. 1.2 Related works Both machine and deep learning algorithms are able to customize mental health assessment for individual use since they can address multiple variables simultaneously [ 12 ]. An exhaustive review by Iyortsuun et al (2020) [ 15 ] examined 33 studies and concluded that deep learning-based models proved to be more accurate in diagnosing schizophrenia, depression, anxiety, bipolar disorder and post-traumatic stress disorder than conventional approaches. Shatte et al (2019) [ 24 ] investigated deep learning’s capacity to evaluate intricate data sets, predict the prognosis and enable personalized interventions, highlighting its revolutionary influence on mental health outcome research. Together, these reviews illustrate how AI-based methods are improving diagnostic precision and making more effective, personalized treatment approaches in psychiatric treatment. Natural language processing (NLP) has proven itself to be a promising domain for psychological studies by drawing on textual data from various sources like social media messages, clinician notes and patient interviews in order to classify, identify and predict mental disorders. Malgaroli et al. (2023) [ 17 ] conducted a systematic review on the application of NLP in Mental Health Interventions (MHI) and consequently developed a research framework to address existing challenges. Results indicated a significant increase in NLP-based MHI studies since 2019 where large language models (LLMs) played a dominant role in data analysis. The review reported that text-based features contributed more to model accuracy than audio-based data and digital health platforms were the primary sources of MHI data. Torfi et al. (2020) [ 30 ] performed a survey on the development of natural language processing (NLP) fueled by deep learning with an emphasis on the manner in which computational advancements and access to large linguistic corpora have revolutionized the discipline. Their aim was to examine the role of deep learning models in NLP operations such as semantic analysis, part-of-speech tagging, named entity recognition and sentiment classification. Their approach involved discussing state-of-the-art deep learning methods such as recurrent neural networks (RNNs), convolutional neural networks (CNNs) and transformer-based models such as BERT, noting their application in automating and enhancing linguistic tasks. Results reported that deep learning greatly improves NLP applications, especially in task-dependent applications such as machine translation, text summarization and sentiment analysis. BERT and GPT transformer models showed better language understanding performance than previous models and were able to effectively capture text long-range dependencies. The work also highlighted grand challenges, e.g., access to large quantities of annotated training data, issues of interpretability and ethical risks of biases within AI-generated content. Teferra et al. (2024) [ 29 ] systemically searched from databases such as Semantic Scholar, PubMed and Google Scholar with research focus on screening depression using NLP techniques. The search term "depression screening," "depression detection," and "natural language processing" was applied and for inclusion studies were considered to be related if they were addressing the application of NLP techniques in screening or detecting depression. The review concluded that although there are possibilities to improve detection of depression using NLP, significant hurdles still exist in the form of ethical concerns and necessity for incorporating cross-cultural awareness. Uddin et al. (2021) [ 31 ] created a depressive symptom predictive system based on deep learning from text data with a Long Short-Term Memory (LSTM)-based Recurrent Neural Network (RNN) model to examine a huge collection of text submitted by youth on a Norwegian online forum. The model was trained using extraction of strong features representing depressive symptoms as defined by medical specialists such as psychiatrists. The explainable AI (XAI) method and Local Interpretable Model-Agnostic Explanations (LIME) was utilized to gain insights into model decisions. Their research focuses on how more improvement is needed in interpretability where the models did not just aim for correct predictions but also aimed to explain their decision-making processes to ensure better transparency and trustworthiness. Milintsevich et al. (2022) [ 19 ] employed a multi-target hierarchical regression model to predict the severity of depression symptoms from DAIC-WOZ interview transcripts. The researchers used S-RoBERTa to generate embeddings for dialog turns and a Bi-LSTM with attention for interviews, trained over 200 epochs. Results demonstrated a Mean Absolute Error (MAE) of 0.438–0.830 for symptom predictions, a macro-F1 score of 73.9 for binary classification and an MAE of 3.78 for total PHQ-8 scores which is competitive with state-of-the-art models. This symptom-focused NLP approach aligned with symptom network analysis offers significant potential for personalized prediction in clinical settings. 1.3 Supremacy of BERT in text classification Numerous studies have concluded that family of large language models such as Bidirectional Encoder Representations from Transformers (BERT) model [ 10 ] and its variants have established new standards in NLP tasks concerned with mental health analysis and outperform traditional models [21, 23 , 30 ]. The BERT model is superior to the traditional neural networks and long short-term memory (LSTM) models because it is based on Transformer architecture [ 32 ] which enables it to learn context from the left and right sides of a token/sentence at the same time. Unlike LSTMs, which read inputs sequentially and are plagued by long-range dependency constraints, BERT leverages multi head self-attention mechanisms to capture sentence-level semantics or in technical terms global dependencies more effectively. In addition, whereas typical neural networks and LSTMs usually need task-specific architectures, pre-trained representations of BERT can be fine-tuned with little adaptation on a wide range of NLP tasks with minimal feature engineering and computational costs. Empirical performance shows BERT's state of-the-art performance on several benchmarks which far outperforms prior models in question answering and language inference tasks. A BERT variant fine-tuned on a clinical dataset can overcome challenges in mental health infrastructure by examining text for patterns that might be missed by clinicians, providing a more comprehensive and impartial evaluation. Through the examination of clinical interview text, it can detect symptoms early on especially in regions with limited mental health professionals which enhances care accessibility in underserved areas and facilitates early intervention. This is in line with the increasing worldwide trend of using AI in healthcare systems to improve outcomes [ 1 , 7 ]. 1.4 The present study Despite the advances in areas of NLP, deep and machine learning and medicine, a key research gap identified is lack of model interpretability and no reports on generalization to real-world settings. Thus, most of the works have halted at performance metrics and token-level explanations without validating the model in actual contexts. The original studies and systematic reviews undertaken in these areas have suggested to increase interpretability, personalization, and the addressing of bias in order to better use NLP for the detection of depression [ 18 , 29 , 31 ]. Therefore, given the rising global prevalence of Major Depressive disorder, limitations in healthcare accessibility and infrastructure, suggestions and the gaps identified, the present study aims to (i) assess the efficacy of a BERT variant for predicting depression through text, (ii) compare the model’s predictions against a clinician’s to assess its generalizability and (iii) further analyze the model’s decision making both qualitatively and quantitatively. Thus, the current study adds to the expanding body of evidence for the potential of AI to serve as a useful resource for enhancing mental health care, especially in resource-scarce settings. 2. METHOD The study employed a mixed method based explanatory sequential research design which aims to assess to what extent an agreement exists between a fine-tuned RoBERTa model and a licensed clinician for predicting depression among Indian adult population. The interviews which were disagreed upon by the raters underwent a thematic analysis to better understand the model’s decision and potentially suggest steps to improve feature extraction and feature engineering for future studies. 2.1 Ethics The study was reviewed and approved by the School Ethics Review Board of School of Behavioral Forensics, NFSU, Gandhinagar between November 2024 to January 2025. The data collection proceeded from first week of February 2025 till 22nd April 2025. For clinical population, the permission of interviewing in-ward patients was obtained from nursing staff and informed consent was obtained from the patients with the interviews being conducted in the presence of the nursing staff only. For OPD based patients, the informed consent was obtained from the patient and/or informants and the interviews were conducted in presence of a clinical psychologist. Inorder to obtain consent from non-clinical population, google forms were circulated online. Confidentiality was maintained throughout the process where patients from clinical group were identified by codes such as C01, C02, etc. and non-clinical group was identified as NC01, NC02, etc. 2.2 Sample Convenient and purposive sampling were used in the current study where available population members in convenience who met the inclusion criteria were sampled. The population of research interest was Indian adults in both clinical and non-clinical settings. The sampling frame consists of 79 individuals with 22 patients from OPD and 26 patients were selected from wards of Hospital for Mental Health, Ahmedabad. The non-clinical population comprised of 31 Indian adults who based on personal judgement of the researcher and their self-report were selected due to no prior history of chief complaints and past diagnosis. The sample consisted of 37 females and 42 males within the age range 18 to 81 years. The clinical group was interviewed in person while the non-clinical group was interviewed either in-person or on video conferencing, based upon the preference given by the participants. 70 out of 79 interviews were conducted in participants’ mother tongue. Thus, the interviews underwent both forward translation into English and backward translation by SV and to ensure consistencies, reduce biases and errors in translation. Table 1 Diagnostic status of clinical group (N = 48) DIAGNOSIS No. of patients Alcohol use disorder (AUD) 3 Bipolar Mood disorder-I 5 Borderline Personality disorder 1 Cannabis and Tobacco use disorder 1 Intellectual disability (IDD) 1 Major depressive disorder (MDD) 2 MDD + psychotic features 2 Obsessive-compulsive disorder 2 Paranoid schizophrenia 1 Pending* 22 Schizoaffective 1 Schizophrenia 5 Schizophrenia + AUD 1 Schizophrenia + IDD 1 * used for OPD based patients who had not been formally diagnosed till the time of interview 2.3 Dataset The DAIC-WOZ Depression Database is a subset of the Distress Analysis Interview Corpus (DAIC), designed to aid in diagnosing psychological conditions such as depression and PTSD [ 14 ]. It includes approximately 50 hours of clinical interviews collected for developing a virtual interviewer named Ellie, controlled by a human operator, to identify verbal and nonverbal indicators of mental illness [ 9 ]. The dataset consists of 189 recorded sessions (with some exclusions and special cases), providing audio, video, transcriptions and various extracted features, including facial expressions, gaze and acoustic properties. Transcripts adhere to specific annotation guidelines and some audio sections have been scrubbed to protect participant privacy. The dataset is intended for use in AI-driven mental health assessments and computational analysis of psychological distress. The dataset was a part of AVEC 2017 challenge where a PHQ-8 score of 10 and above indicated depression. The PHQ-8 interview schedule in DAIC-WOZ serves as a supplement to the clinical interviews which are the gold standard for diagnosis [ 11 ]. The DAIC-WOZ dataset contains 189 interviews that have been split into training, validation and testing sets. Since 47 interviews in test set were spare with no PHQ binary, the current study used the 107 interviews for training and validating while the remaining 35 interviews for testing the model. 2.3.1 Data preprocessing A significant change from previous studies is not including prompts of the interviewer to train the model. Burdisso et al. (2024) [ 5 ] found that incorporating the interviewer’s prompts into depression detection models from the DAIC-WOZ dataset introduces a bias that significantly improves classification performance but not necessarily due to a better understanding of participants' distress. Models using interviewer’s prompts tended to focus on a specific section of interviews where questions about past mental health experiences were asked, effectively exploiting this bias to distinguish between depressed and control participants. By leveraging this shortcut, models achieved an F1 score of 0.90, the highest reported using only textual data. In the original dataset, the responses to the same question have been split into multiple samples for the same utterance due to differences in timestamps. At present, the timestamps were removed and responses for a single utterance were merged into one sample due to which the original number of samples were reduced even further. The dataset was converted to lowercase, cleaned to remove punctuations and unnecessary utterances (‘mhm’ or ‘hmm’), spaces, and contractions (don’t, won’t, etc.) were expanded. Instead of opting for automated lexicon- based approaches [ 6 , 16 , 22 ] for identifying depression markers from the corpus, we decided to opt for manual labelling to ensure complexity, nuances and subtlety of human language is not missed. An exhaustive process was undertaken to categorize responses in both training and test datasets. There are questions that are unable to differentiate between a depressed and non-depressed participants such as- “What's your dream job”, “Where are you from originally” and other fillers. Thus, the responses to these filler questions have been automatically categorized as non-depressed samples despite them coming from an individual exhibiting symptoms of depression. For questions which appeared to be clinically relevant SV went through the interviews to manually label responses as either 0 or 1 based on DSM-5’s criteria for Major Depressive Disorder. Additionally, social support was also considered as an important feature in line with existing literature [ 25 , 27 ]. AP verified the categorization by reviewing 10 interviews chosen from training split. After the suggested changes and agreement between both the authors on categorization of responses, a set of 2974 samples was used as training and validation set while another set of 1133 samples from 35 interviews was used as the test set. Inorder to affirm that the categorization process was indeed suitable for differentiating between the classes, average semantic similarity scores were computed for both train and test sets. Each statement was tokenized and encoded in batches with embeddings derived from the mean-pooled last hidden state of the model’s output whilst taking attention masks into account. Cosine similarity was then computed pairwise: (1) within each class to assess intra-group semantic similarity and (2) across the two classes to evaluate inter-group similarity. For each case, the mean of the similarity matrix (excluding diagonal self-similarity) was reported as the semantic similarity index to provide an extent to which both classes were distinct. Table 2 shows the obtained index values across three conditions for both sets. Table 2 Average similarity index for final train and test sets Comparison Mean Similarity (Train) Mean Similarity (Test) Within class 0 0.6250 0.6639 Within class 1 0.7656 0.7107 Between 0.3880 0.4547 From Table 2 , it can be inferred that the manual categorization was indeed effective especially when it came to discriminate between the two classes. The similarity within non-depressed class (class 0) was less in both sets than that of depressed class (class 1). This observation can be attributed to variability in responses to questions which addressed participants’ hobbies, living situations, aspirations, etc. While the stronger cohesion within depressed class exhibits similar usage of language to express common themes that align with symptomatic criteria of depression. 2.4 Measures RoBERTa (A Robustly Optimized BERT Pretraining Approach) is a transformer language model that extends BERT with enhanced training methods like longer training durations, elimination of the next sentence prediction task, and dynamic masking. It has 768 dimensions and has been trained on around 125 million parameters which also supports up to 512 tokens. The model was trained for five epochs using a linear warm-up and decay schedule where the peak learning rate was set to 3e-5 while applying a constant hidden-layer dropout of 0.2. The original set of 2794 samples consisted of 1972 samples from non-depressed class while 822 samples from depressed class. To address this class imbalance, we used random undersampling for majority class and brought both classes to 1:1 ratio. Finally, an early-stopping callback with a patience of one epoch was employed to stop training once validation loss failed to improve. The interview schedule used in current study was a modification of original DAIC-WOZ’s interview schedule which is based on PHQ-8. In context of the current research the interviews were to be conducted in clinical settings (OPD and wards) where time efficiency was must due to larger number of patients and resource restraints. This concern worked both ways since patients and/or their informants were aware that there was a problem that needed to be addressed immediately. Thus, the questions asked aimed to directly address the chief complaints and symptoms of patients within a shorter period of time. Hence, the filler questions that aim to establish rapport and enquire clinically irrelevant details have been removed. Dr. AP and Dr. SV who are clinical psychologists, provided their expert judgement on the interview schedule for its content validity. The experts determined if the original 18 items in the modified interview schedule were relevant in Indian context. They agreed on relevance of 16 items out of 18, whereupon after further discussion it was decided to modify the two items to match the appropriate context. Thus, the final interview schedule used in current study had 13 main questions with the last question having five sub-parts as provided in Appendix A. 2.5 Data analysis Performance metrics- sensitivity, precision, accuracy and macro-averaged F1 scores were reported on the test split. F1 score is of primary importance in the current work since it balances precision and recall, making it relevant in cases of class imbalance. The 79 interviews were examined by AP to determine whether the individuals exhibited indicators of depression and provided as an input to the fine-tuned RoBERTa model inorder to obtain its decision for indicators of depression. Jamovi, an open-source software for quantitative analysis was used to calculate the Cohen’s Kappa coefficient for inter-rater reliability between AP and the model. Additionally, the interviews upon which agreement couldn’t be reached between the two raters, underwent a thorough analysis both qualitatively and quantitatively. For the purpose of qualitative analysis, the interviews on which both the raters disagreed were analyzed following the blueprint of Braun & Clarke’s thematic analysis [ 4 ] to code and develop themes with an inductive approach. SV reviewed all the eight interviews multiple times to gain familiarity with the data and generate around 94 codes initially which after revision were reduced to 80 codes. SV identified the patterns prevalent across these codes to develop initial themes and sub-themes. After modifications finally a set of six themes emerged from the dataset. The iterative process of mapping the codes, categorizing them into groups to identify and revise the sub-themes and themes was done in an online open-sourced software Taguette by SV. The quantitative analysis was done by obtaining token attribution scores for one randomly selected interview out of the set of misclassified interviews. 3. RESULTS 3.1 Performance metrics Table 3 Classification report of fine-tuned Roberta-base model Class Precision Recall F1-Score Support 0 (Not depressed) 0.96 0.88 0.92 914 1 (Depressed) 0.63 0.83 0.71 219 Accuracy 0.87 Macro Avg 0.79 0.86 0.82 Weighted Avg 0.89 0.87 0.88 In Table 3 , it can be observed that the model has strong Precision of 0.96 and 0.88 recall for non-depressed class while it has a similar recall for depressed class, it has a moderate precision of 0.63 indicating the risk of generating false positives. The model has an accuracy of 87% with an F1- score with weighted average of 0.88 and a macro average of 0.82 reflecting an overall strong performance despite major class imbalance. 3.2 Inter-rater reliability estimates Table 4 Contingency table for rater and responses Model: Not Depressed Model: Depressed Total (Rater) AP: Not Depressed 61 8 69 AP: Depressed 0 10 10 Total (Model) 61 18 79 Out of 79 interviews 48 interviews belonged to clinical population while 31 interviews were taken with non-clinical population. In clinical population 22 interviews were conducted freshly with patients from OPD and hence there was no formal diagnosis available for them while the rest 28 interviews were conducted with patients from ward who were diagnosed formally on basis of a mutual decision given by psychiatrists and clinical psychologists with assessments and multiple sessions (Table 1 ). Out of these 28 patients there were four patients diagnosed with Major Depressive Disorder including comorbid features and 5 patients were diagnosed with Bipolar Mood Disorder-I without any comorbidities. As depicted in Table 4 , there were eight interviews where AP and the model had a disagreement. Out of these, one interview belonged to an inward patient of Alcohol use Disorder while the rest seven interviews belonged to OPD based patients. Thus, it is evident that the classification report obtained from test set of DAIC-WOZ is in alignment with interviews conducted on Indian population where the model is generating false positives while successfully identifying actual cases of depression. Table 5 Estimate of Cohen’s Kappa Statistic Value Number of subjects 79 Number of raters 2 Percent agreement 89% Cohen’s kappa (κ) 0.628 p -value < .001 From Table 5 , it can be concluded that for 79 interviews, the obtained Cohen’s Kappa coefficient was 0.628 indicating a substantial level of agreement between the two raters. For the observed agreement of 89% and the obtained coefficient, the p < 0.001 indicating the results are statistically significant and unlikely due to chance. 3.3 Analysis of false positives The eight interviews upon which the model and AP disagreed underwent a qualitative and quantitative analysis to gain a deeper understanding of the inconsistencies in depression prediction. This dual approach was employed to move beyond descriptives such as performance metrics and Cohen's kappa and instead explore the reasons behind the disagreements for further model refinement. By integrating qualitative insights such as thematic analysis with quantitative measures like token attribution scores, the analysis aimed to identify specific patterns and potential biases in the model’s predictions. The thematic analysis revealed that while many statements aligned with depression, they were not uniquely indicative of it and their cause did not lie solely in affective afflictions but other factors too (social, substance use, physical difficulties, etc.). Below, each theme is discussed in detail, highlighting its contribution to false positives and its interplay with other themes. 3.3.1 Discussion of themes 3.3.1.1 Psychological Symptoms The Psychological Symptoms theme includes emotional, behavioral, and cognitive disturbances that superficially resemble depression. Emotional Distress which is characterized by codes like "I have been feeling empty" and "After my engagement was broken, I have felt worthless sometimes," suggests significant affliction in mood. However, these afflictions may stem from situational stressors such as relationship breakdowns rather than a depressive disorder. For example, Participant C24's worthlessness following a broken engagement could reflect acute grief rather than chronic depression. Behavioral Disturbances, such as "I sit idle at one place all day" or "(informant) She throw things around the house," may indicate irritability associated with depression. Yet, these behaviors could also reflect frustration, anxiety, or personality traits, particularly when paired with interpersonal conflicts. Cognitive disturbances, including "I cannot concentrate on my work" and "I have been overthinking for a month now," are similar to cognitive impairment observed in depression but may also signal anxiety or stress-related rumination. The overlap of these symptoms with multiple conditions determines the risk of false positives when models prioritize symptom presence over context and cause. 3.3.1.2 Physical Symptoms Physical Symptoms are divided into subthemes of those indicative of depression (e.g., sleep disturbances, psychomotor retardation) and somatic complaints not indicative of depression (e.g., headaches, dizziness). Symptoms like "I have been slower than before" (C24) or "I do not eat" (C25) are classic depression markers but are not specific to it. For example, Participant C26's sleep issues ("I cannot sleep when people are always fighting outside") appear as result from environmental stressors rather than depression. Somatic complaints, such as "usually dizzy most of the day" (C21) or "I am old so I cannot have much energy" (C32), further complicate diagnosis, as they may reflect physical illnesses, aging, or substance use effects (C26). These findings align with existing literature indicating that somatic symptoms are prevalent in non-depressed populations, particularly among older adults or those with chronic conditions [ 26 ]. 3.3.1.3 Interpersonal Factors Interpersonal Factors including conflicts and familial relationships play a significant role in manifestation of symptoms. Codes like "I have frequent fights with my family especially my wife" (C47) and "I cook my own food and wash my own clothes" (C21) suggest social withdrawal and strained relationships which can intensify emotional distress. However, these interpersonal dynamics may be primary causes of distress rather than secondary to a depressive disorder. 3.3.1.4 Treatment and History The Treatment and History theme provides critical context for interpreting current symptoms. Codes like "He [Doctor] said I have a mental illness" (C32) indicate prior mental health engagement which may bias automated models to flag depression. Similarly, past symptoms such as "I was under stress" or "I cannot remember much but I used to drink a lot" (C26) suggest a history of past distress or substance use that complicates current symptom interpretation for the model. For example, Participant C26's alcohol use history may explain physical symptoms like vomiting which a model might falsely attribute to depression. This theme highlights the need for longitudinal data to differentiate chronic depressive symptoms from transient or substance-related issues, a challenge for automated systems relying on single-point assessments. 3.3.1.5 Response Style Contradictory Responses, observed in seven of eight participants, are a key contributor to false positives. Statements like "I am fine... I have been feeling empty" (C09) or "I get angry on small things these days... but mostly I am happy" (C25) reflect inconsistent self-reporting, potentially due to denial, fluctuating moods, or lack of insight. Such contradictions may exaggerate symptom scores in automated models as positive and negative statements are both counted leading to erroneous flagging. This aligns with research suggesting that patient denial or stigma can distort self-reports which can complicate depression diagnosis. 3.3.1.6 Symptoms of Other Disorders The presence of non-depressive symptoms, such as those related to Alcohol Use Disorder (e.g., "I vomit a lot due to drinking" in C26) or paranoia (e.g., "I feel as if they are against me" in C32), significantly complicates diagnosis. Paranoia, reported by Participants C21 and C32, may indicate conditions like anxiety, personality disorders, or psychosis rather than depression. Similarly, alcohol-related symptoms in C26 overlap with depressive physical complaints but have a distinct etiology. These findings echo studies highlighting the diagnostic challenge of distinguishing depression from other psychiatric conditions with overlapping symptoms. Automated models may fail to weight these non-depressive symptoms appropriately, leading to false positives. Table 6 Theme matrix ID contradictions Psychological complaints Physical complaints Interpersonal factors Past history Other symptoms C09 ✔ ✔ ✔ C21 ✔ ✔ ✔ ✔ ✔ C22 ✔ ✔ ✔ C24 ✔ ✔ ✔ ✔ C25 ✔ ✔ ✔ ✔ C26 ✔ ✔ ✔ ✔ ✔ C32 ✔ ✔ ✔ ✔ ✔ ✔ C47 ✔ ✔ ✔ ✔ ✔ The identified themes can lead to false positives in the model by emphasizing symptoms that overlap with depression but arise from alternative causes as discussed above. The code matrix illustrates this complexity, showing unique combinations of themes that might confuse a model relying on symptom presence alone. For instance, Participant C09 exhibits Emotional and behavioral disturbances which are classic depression markers. However, presence of contradictory responses suggest unreliable reporting, possibly due to situational distress or denial which the model might misinterpret as depression. Participant C21 exhibits complex cluster of symptoms ranging from Emotional Distress, Physical Complaints, Other Symptoms (e.g., suspiciousness) to Interpersonal Conflicts. In such cases the model might flag this as depression even in the presence of non-depressive symptoms like paranoia that indicate a different condition. Somatic complaints are prevalent across participants (e.g., C24, C25, C32) which might have been attributed to depression by the model but external factors (e.g., noisy environments, aging, or alcohol use, as in “I vomit a lot due to drinking” for C26) appear to be the root cause. Somatic symptoms of depression often overlap with physical illnesses due to which diagnosing depression becomes complicated (Mitchell et al., 2009). Similarly, Interpersonal Conflict in participants like C47 might amplify emotional distress, leading the model to flag for depression without considering relational context. In the present context the risk of false positives exists because the model fails to weigh the specificity of symptoms or their co-occurrence with non-depressive indicators (e.g., “I feel as if they are against me” in C32). Contradictory responses further complicate matters as inconsistent self-reports might inflate symptom scores erroneously. Thus, the model’s reliance on surface-level symptom matching without contextual nuance increases its risk for predicting false positives. 3.3.2 Attribution scores For the purpose of gaining quantitative insight into model’s decision, out of eight interviews, one was randomly chosen via fishbowl method to obtain the tokens with fifteen highest attribution scores. The results for C25’s interview depict "irritated" (+ 0.6) and "suspicious" (+ 0.55) are the most influential tokens driving the model’s prediction of depression, indicating a strong focus on affective symptoms and paranoia. Tokens such as “more” (+ 0.25), "less" (+ 0.15) and "decreased" (+ 0.15) are contributing factors to the model’s classification, though their influence is moderate compared to higher-scoring tokens. The tokens "less" and “decreased” are often associated with statements like reduced appetite or energy (e.g., "I eat less"), which directly signals a decline in typical functioning contributing to the class 1 prediction by emphasizing loss or reduction, a common depressive marker. Similarly, "more" (+ 0.25) indicates an increment in intensity of symptom, further supporting the class 1 prediction by suggesting a worsening state. Individually, these tokens contribute to the prediction by perhaps aligning with hallmark depression indicators like reduced functioning, declining states and periodic emotional struggles, though their moderate attribution scores suggest they are secondary drivers compared to more emotionally charged tokens. Tokens like “more”, "anger", and "become" also contribute, though to a lesser extent reflecting attention to affect and markers in change. This aligns with the thematic analysis, as “irritated” and "anger" correspond to Theme of Psychological Symptoms specifically emotional distress, highlighting the model’s tendency to overgeneralize affective language without considering situational factors like grief or stress. Similarly, the high attribution score of "suspicious" aligns with symptoms of other disorders confirming the challenge in isolating psychotic symptoms when depression indicators are present, despite its ability to differentiate them in their absence. 4. DISCUSSION The present study aims to assess how well a RoBERTa model, fine-tuned on DAIC WOZ dataset generalizes to Indian population by assessing inter-rater reliability between a clinical psychologist and the model for indicators of depression in 79 interviews (48 in clinical group and 31 in non-clinical). Further, to gain a deeper understanding behind model’s decision making, the interviews which were disagreed upon by both the raters underwent thematic analysis and one interview was randomly selected for obtaining token attribution scores. Despite the nature of scarce data and manual labelling which might have led to biases in both test and training sets, the RoBERTa model has achieved a macro F1 score of 0.82, performing in accordance with studies using text classification to predict depression [ 11 , 19 , 28 33 , 34 ]. The overall accuracy of the model is 0.87, implying that 87% of the total predictions across both classes are correct. While this seems high on its own, accuracy alone can be misleading especially in imbalanced datasets such as that in current study where non-depressed has significantly more instances than depressed. Therefore, it's important to also consider the macro-averaged F1-score of 0.82 (Table 3 ), which balances precision and recall. For the non-depressed class, the model demonstrates a strong precision of 0.96, which is that when it classifies an individual as non-depressed, it is correct a majority of the time. Its recall for the same class is also quite high at 0.88 which implies it gets 88% of all actual Class 0 instances correct. As compared, the depressed class has much less precision of only 0.63, meaning a higher level of false positives. Still, its recall is relatively stronger at 0.83, showing that it catches most actual cases of depression accurately. This trade-off between precision and recall for depressed class means that although the model is accurate at identifying depressed people when they exist, it is less accurate in validating those predictions because it has a greater number of false alarms. Thus, the model may benefit from further tuning or resampling techniques. Out of 79 interviews conducted the model and rater disagreed over eight interviews with a significant Cohen’s Kappa of 0.628, where all these interviews according to the clinician fell under non-depressed category but the model predicted the opposite class (Table 4 ). The results from fine tuning the model are reflected in its generalizability where it can be observed that the model’s ability to catch all cases of depression whilst producing false positives is a significant limitation to overcome. Qualitative (thematic analysis) and quantitative (attribution scores) analysis of these eight interviews reveals that symptoms mimicking depression often arise from situational stressors, substance use or other disorders, leading to false positives in automated models. The thematic analysis identified six key themes (Fig. 1 ): Psychological Symptoms, Physical Symptoms, Interpersonal Factors, Treatment and History, Response Style, and Symptoms of Other Disorders; revealing that symptoms resembling depression such as emotional distress (e.g., "I have been feeling empty"), somatic complaints (e.g., "I cannot sleep") and interpersonal conflicts (e.g., "I have frequent fights with my family"), often arise from situational stressors, substance use, or other psychological conditions like anxiety or paranoia rather than depression. For example, Participant C24’s reported worthlessness was linked to a recent broken engagement, suggesting acute grief rather than chronic depression, while Participant C26’s sleep disturbances were attributed to environmental disturbances rather than a depressive condition. The quantitative analysis of token attribution scores for Participant C25’s interview (Fig. 2 ) clarified the model’s decision-making process. Tokens such as "irritated" (+ 0.60), "suspicious" (+ 0.55), "less" (+ 0.15), and "decreased" (+ 0.15) were highly influential in driving the model’s depression prediction, reflecting its focus on affective and functional decline markers. However, the high attribution score for "suspicious" aligns with the findings of thematic analysis indicating the model’s challenge in distinguishing paranoia or anxiety-related symptoms from depression. Similarly, tokens like "less" and "decreased," often tied to reduced appetite or energy correspond to the somatic symptoms theme but lack contextual specificity leading to erroneous flagging when these symptoms stem from non-depressive causes like aging or substance use. Contradictory self-reports, prevalent in seven of the eight disagreed cases further complicate predictions, as the model may overemphasize both positive and negative statements. 4.1 Limitations and future implications In context of the present study, the fine-tuned RoBERTa model demonstrates strong potential for use of AI in screening depression among Indian adults within clinical settings while closely aligning with a clinicians' assessments. However, there are certain limitations that can be overcome to improve the results such as the lack of a clinical dataset specific to the Indian context, significant class imbalance in training and test sets and the ‘Lost in Translation’ effect i.e. reliance on manual categorization and transcription of interviews which might introduce biases and errors. The model's performance is limited by its inability to incorporate audio and visual cues and its decision criteria could be refined to identify specific areas of disturbance rather than solely detecting depression indicators based on symptoms. The results offer practical implications for AI in clinical environments, enabling rapid depression screening via short interviews aligned with DSM-5 criteria in high-demand settings. Future studies can leverage multilingual clinical BERT variants to mitigate translation errors and incorporate audio and visual cues to reduce manual input, potentially enhancing screening and diagnostic processes. The model operates efficiently on standard laptops and desktops, requiring no advanced hardware and henceforth, future iterations incorporating additional features must prioritize accessibility. Expanding rater diversity beyond clinicians, such as including psychiatrists and psychiatric social workers, could further validate the model’s reliability across different domains. 5. CONCLUSION The current study examined the potential of AI in clinical settings through a fine-tuned RoBERTa model for predicting depression among Indian adults, the decisions of which aligned closely with the decision of a clinical psychologist. Despite robust performance, false positives arose due to symptoms of depression overlapping with situational stressors and other disorders. This problem emerged due to inconsistent reporting by patients, situational stressors and other psychological conditions overlapping with classic symptoms of depression. These findings align closely with existing literature leveraging BERT variants and DAIC-WOZ dataset which have highlighted the significance of text-based detection of depression and other disorders. However, as discussed earlier, the study noted challenges such as risk of false positives which can be countered by an exhaustive analysis of model’s decision making to further improve its explainability, choice of datasets and feature extraction techniques. APPENDIX A: The interview schedule used in current study 1. How are you doing today 2. How close are you to your family 3. Do you have a close relationship with anyone else other than your family 4. Over the past two weeks have you been frequently experiencing anger or irritability that feels out of control 5. How easy is it for you to get a good night's sleep 6. Over the past two weeks has your interest or pleasure in doing things changed 7. How have you been feeling for the past two weeks 8. How is your energy level for the past two weeks 9. Over the past two weeks have you been overeating or do you not feel hungry at all 10. Within the past two weeks have you ever felt worthless or guilty 11. Over the past two weeks can you concentrate properly on the activities you usually do 12. Over the past two weeks have you noticed any change in your movement, are you too slow or have you been restless and fidgety 13. Have you previously seen a doctor regarding these problems. Follow up questions: a. What decision was given by the doctor b. How long ago was your disease identified c. What were your symptoms d. What got you to seek help e. Did you think you had a problem before you found out Declarations Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Author Contribution S.V. worked on Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Visualization, Writing – original draft and editing. Dr. A.P. supervised the work and assisted with Conceptualization, Methodology, Resources, Project administration, Formal Analysis, Validation and Writing – review. Dr. S.V. co-supervised and assisted with Conceptualization, Validation and Writing- review Data Availability Although the sets of interviews used to train, test and calculate inter-rater rater reliability are not available due to ethics surrounding patient confidentiality, the code used to generate results is open available in author's Git repository at https://github.com/sv6121096/RoBERTa-for-depression References Alhuwaydi AM (2024) Exploring the role of artificial intelligence in mental healthcare: current trends and future directions–a narrative review for a comprehensive insight. Risk Manage Healthc Policy 1339–1348. https://doi.org/10.2147/rmhp.s461562 American Psychiatric Association (2022) Diagnostic and statistical manual of mental disorders (5th ed., text rev.). https://doi.org/10.1176/appi.books.9780890425787 Bains N, Abdijadid S (2023), April 10 Major depressive disorder . In StatPearls. StatPearls Publishing. https://www.ncbi.nlm.nih.gov/books/NBK559078/ Braun V, Clarke V (2006) Using thematic analysis in psychology. Qualitative Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa Burdisso S, Reyes-Ramírez E, Villatoro-Tello E, Sánchez-Vega F, López-Monroy P, Motlicek P (2024) DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews. arXiv preprint arXiv:2404.14463 Cha J, Kim S, Park E (2022) A lexicon-based approach to examine depression detection in social media: the case of Twitter and university community. Humanit Soc Sci Commun 9:325. https://doi.org/10.1057/s41599-022-01313-2 Cruz-Gonzalez P, He AW, Lam EP, Ng IMC, Li MW, Hou R, Chan JN, Sahni Y, Guasch V, Miller N, Lau T (2025) Artificial intelligence in mental health care: a systematic review of diagnosis, monitoring, and intervention applications. Psychol Med 55:e18 & Sánchez Vidaña, D. I. https://doi.org/10.1017/S0033291724003295 DeSouza DD, Robin J, Gumus M, Yeung A (2021) Natural language processing as an emerging tool to detect late-life depression. Front Psychiatry 12:719125. https://doi.org/10.3389/fpsyt.2021.719125 DeVault D, Artstein R, Benn G, Dey T, Fast E, Gainer A, Georgila K, Gratch J, Hartholt A, Lhommet M, Lucas G, Marsella S, Morbini F, Nazarian A, Scherer S, Stratou G, Suri A, Traum D, Wood R, Xu Y, Rizzo A, Morency L-P (2014) SimSensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’14), Paris Devlin J, Chang MW, Lee K, Toutanova K (2019), June Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171–4186). https://doi.org/10.48550/arXiv.1810.04805 Dinkel H, Wu M, Yu K (2019) Text-based depression detection: What triggers an alert. arXiv preprint arXiv:1904.05154 Dubey A, Chahar R, Narang S (2024) A Comparative Analysis of Machine Learning Algorithms with Bagging and Boosting for Mental Health Assessment. Int J Electron Commun Eng 11:20–39. https://doi.org/10.14445/23488549/IJECE-V11I6P103 Ducharme J (2025), January 14 Dismissed and disbelieved, some Long COVID patients are pushed into psychiatric wards . Time . https://time.com/7206080/long-covid-psychiatric-wards/ Gratch J, Artstein R, Lucas GM, Stratou G, Scherer S, Nazarian A, Wood R, Boberg J, DeVault D, Marsella S, Traum DR The Distress Analysis Interview Corpus of human and computer interviews. InLREC 2014 May (pp. 3123–3128) Iyortsuun NK, Kim SH, Jhon M, Yang HJ, Pant S (2023) A Review of Machine Learning and Deep Learning Approaches on Mental Health Diagnosis. Healthc (Basel Switzerland) 11(3):285. https://doi.org/10.3390/healthcare11030285 Karmen C, Hsiung RC, Wetter T (2015) Screening internet forum participants for depression symptoms by assembling and enhancing multiple NLP methods. Comput Methods Programs Biomed 120(1):27–36 Malgaroli M, Hull TD, Zech JM, Althoff T (2023) Natural language processing for mental health interventions: a systematic review and research framework. Translational Psychiatry 13(1):309 Mao K, Wu Y, Chen J (2023) A systematic review on automated clinical depression diagnosis. npj Mental Health Res 2:20. https://doi.org/10.1038/s44184-023-00040-z Milintsevich K, Sirts K, Dias Gaël (2022) Towards Automatic Text-Based Estimation of Depression through Symptom Prediction. 10.21203/rs.3.rs-2096670/v1 Mitchell AJ, Vaze A, Rao S (2009) Clinical diagnosis of depression in primary care: a meta-analysis. Lancet (London England) 374(9690):609–619. https://doi.org/10.1016/S0140-6736(09)60879-5 Moradi M, Samwald M (2022) Deep learning, natural language processing, and explainable artificial intelligence in the biomedical domain. arXiv preprint arXiv:2202.12678 Nguyen T, Yates A, Zirikly A, Desmet B, Cohan A (2022) Improving the generalizability of depression detection by leveraging clinical interview schedules. arXiv preprint arXiv:2204.10432 Poświata R, Perełkiewicz M (2022) OPI@LT-EDI-ACL2022: Detecting signs of depression from social media text using RoBERTa pre-trained language models. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion (pp. 276–282). Association for Computational Linguistics. https://aclanthology.org/2022.ltedi-1.40/ Shatte ABR, Hutchinson DM, Teague SJ (2019) Machine learning in mental health: a scoping review of methods and applications. Psychol Med 49(9):1426–1448. https://doi.org/10.1017/S0033291719000151 Smith L, Hill N, Kokanovic R (2015) Experiences of depression, the role of social support and its impact on health outcomes. J Ment Health 24(6):342–346. https://doi.org/10.3109/09638237.2014.954693 Spitzer RL, Williams JB, Kroenke K, Linzer M, deGruy FV 3rd, Hahn SR, Brody D, Johnson JG (1994) Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA 272(22):1749–1756 Stewart RA, Patel TA, McDermott KA, Cougle JR (2022) Functional and structural social support in DSM-5 mood and anxiety disorders: A population-based study. J Affect Disord 308:528–534. https://doi.org/10.1016/j.jad.2022.04.026 Tadesse MM, Lin H, Xu B, Yang L (2019) Detection of depression-related posts in reddit social media forum. Ieee Access 7:44883–44893. https://doi.org/10.1109/ACCESS.2019.2909180 Teferra BG, Rueda A, Pang H, Valenzano R, Samavi R, Krishnan S, Bhat V (2024) Screening for Depression Using Natural Language Processing: Literature Review. Interact J Med Res 13:e55067. https://doi.org/10.2196/55067 Torfi A, Shirvani R, Keneshloo Y, Tavvaf, Nader, Fox, Edward (2020) Natural Language Processing Advancements By Deep Learning: A Survey. 10.48550/arXiv.2003.01200 Uddin MZ, Dysthe KK, Følstad A et al (2022) Deep learning for prediction of depressive symptoms in a large textual dataset. Neural Comput Applic 34:721–744. https://doi.org/10.1007/s00521-021-06426-4 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … Polosukhin,I. (2017). Attention is all you need. Advances in neural information processing systems, 30.https://doi.org/10.48550/arXiv.1706.03762 Zhang J, Guo Y (2024) Multilevel depression status detection based on fine-grained prompt learning. Pattern Recogn Lett 178:167–173. https://doi.org/10.1016/j.patrec.2024.01.005 Zhang X, Li C, Chen W et al (2025) Optimizing depression detection in clinical doctor-patient interviews using a multi-instance learning framework. Sci Rep 15:6637. https://doi.org/10.1038/s41598-025-90117-w Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7431380","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":505581243,"identity":"901ca834-e37a-4217-8783-b38c546c1761","order_by":0,"name":"Shivangi Verma","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6UlEQVRIie2RLwvCQBTATwRXDlZP/DPEL/DAYvDD3MrSBJNZEDQ4+8AvcU1sTw5cmTZhw2LSYhAshgV3Ftt2NsH7hQcP3o/3jxCD4SepIt6BtW1rkifQ11Fq7jYcDXr1AJXCdBTalfTuuSLkKtNQnNAnSEG6on49iueIEXu+4IUKJENEBrK3bvjjNMgHY/FeFCv0wBFAtjYr30torgAbFitOEANykBWRxl6a6SgkCgARvI5IrN1JqwtEM76dgDqyXzs1gdHSXZxpVT6yTL0yuqS3bNC258uSwT5QeEfdcoV1/qbaYDAY/ogXOMNPrNxvIVQAAAAASUVORK5CYII=","orcid":"","institution":"National forensic sciences university","correspondingAuthor":true,"prefix":"","firstName":"Shivangi","middleName":"","lastName":"Verma","suffix":""},{"id":505581244,"identity":"5de4050e-2bfa-4850-a4b0-c4e8ea312f41","order_by":1,"name":"Dr. Ashwani Pundeer","email":"","orcid":"","institution":"National forensic sciences university","correspondingAuthor":false,"prefix":"Dr.","firstName":"Ashwani","middleName":"","lastName":"Pundeer","suffix":""},{"id":505581245,"identity":"a1159356-3500-495b-8709-c169132aa787","order_by":2,"name":"Dr. Soniya Vats","email":"","orcid":"","institution":"National forensic sciences university","correspondingAuthor":false,"prefix":"Dr.","firstName":"Soniya","middleName":"","lastName":"Vats","suffix":""}],"badges":[],"createdAt":"2025-08-22 06:38:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7431380/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7431380/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95386809,"identity":"d9e9ceba-20c6-4b1d-a5ce-db6754b25009","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":219433,"visible":true,"origin":"","legend":"","description":"","filename":"FINALDRAFT.docx","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/8ec171c015ebbf65f23efed4.docx"},{"id":95386806,"identity":"5f5a6250-88d8-491a-b1a8-02109f15056e","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5334,"visible":true,"origin":"","legend":"","description":"","filename":"0a96564739854d399a8a21be6553590f.json","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/02647408716ed433cb44674f.json"},{"id":95386812,"identity":"89252343-7334-4a0e-93fe-8696b4542f91","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":122421,"visible":true,"origin":"","legend":"","description":"","filename":"0a96564739854d399a8a21be6553590f1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/1119e34d271c680be6593676.xml"},{"id":95526520,"identity":"468adb51-f4bb-4d8c-8071-264a62bd4d88","added_by":"auto","created_at":"2025-11-10 10:07:10","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":46913,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/8ecb6b70ddc0d9f1f9db4fdd.png"},{"id":95525584,"identity":"4a37c1c2-9c9c-49fe-9adb-f0a5fc1143f1","added_by":"auto","created_at":"2025-11-10 10:05:22","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9275,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/1548bd30908f21b1f9aa74b5.png"},{"id":95386813,"identity":"e836a25f-c0e5-4a9f-86cc-d09031006c3a","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"xml","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":118822,"visible":true,"origin":"","legend":"","description":"","filename":"0a96564739854d399a8a21be6553590f1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/d903087d549802c9c46889fe.xml"},{"id":95386811,"identity":"9d00710f-59ef-49ba-ba65-d543784941bd","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"html","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":128732,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/50f7718eaa7231489f8e5217.html"},{"id":95386805,"identity":"f7bf1d21-8f6b-41bd-8ae3-bd15f04c4571","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":105935,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eMapping sources of false positives in predicting depression\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/585071060ae985d88e67d7e3.jpeg"},{"id":95386807,"identity":"26d61fe4-25d9-4a50-a5c7-02377bb00a40","added_by":"auto","created_at":"2025-11-07 13:08:03","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":39003,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eToken attribution for C25's interview\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/12d87436a4fe6e4527b7b533.png"},{"id":100421798,"identity":"b438bb16-b7cb-4082-97e0-a648f2275f79","added_by":"auto","created_at":"2026-01-16 13:51:07","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1072700,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7431380/v1/e24aa7c5-0270-4d0b-b811-194f41a56b8c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Inter-rater Reliability of an LLM in Predicting Depression Among Indian Adults","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eAn alarming trend has been observed in the last decade where mental health disorders have emerged as a growing global crisis affecting individuals from all age groups and socioeconomic backgrounds. Major depressive disorder (MDD), with its global prevalence of 5\u0026ndash;17% is a mood disorder that is characterized by persistent low mood, anhedonia, loss of energy, poor concentration, appetite change, sleep disturbances, psychomotor agitation or retardation, suicidal ideation and worthlessness or guilt [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The World Health Organization (WHO) has reported MDD is likely to be the primary disease burden cause globally by 2030 following its ranking as the third cause of the global disease burden in 2008 [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. In low-income nations, over 75% of people with mental illness are unable to access treatment options. Even in high-income nations, just one-third of those with depression receive treatment. The proportion of individuals receiving minimally adequate treatment is highly variable, from around 23% in high-income nations to as low as 3% in low- and lower-middle-income nations. The origin of this lacuna does not just belong to infrastructure and medical facilities but to the general population's awareness, as well as that of affected populations. Several patients with depression first go for medical care to address bodily grievances instead of attending a mental health specialist. Patients in almost half of such situations do not say they feel depressed or experience low mood [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Consequently, screening for depression in primary care is important for early detection and treatment. Screening, assessment and diagnosis are separate but important processes in healthcare. Screening is performed to identify early signs of a disorder through chief complaints in the majority of cases, facilitating timely treatment and prevention. Evaluation is a more overall and thorough analysis of an individual's overall wellbeing, where some scales or tools utilized along with taking into account medical history, physical examinations and others. Diagnosis is the ultimate step, establishing a particular disorder or comorbidity of this disorder when the overall clinical picture is considered. Screening is particularly significant as it identifies problems early before they even reach their serious form, resulting in improved prognosis, treatment and reduced healthcare expense.\u003c/p\u003e\u003cp\u003eUnfortunately, the existing instruments for assessment do not consider patient's entire history of presenting with psychological symptoms and thus frequently leading to errors and delay in diagnosis [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. For instance, psychiatric wards have erroneously admitted patients suffering from Long COVID because the two conditions have some overlapping symptoms [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Standard clinical procedures such as administration of inventories and unstructured interviews fail to identify early or minimal signs of depression due to the existence of elements such as interviewer bias in asking symptoms, patient bias in describing symptoms, time constraints and over-reliance on subjective judgment. This is where deep and machine learning algorithms can come in to enable making correct and objective judgments while considering multiform sources of information unlike conventional instruments which use information from a few sources and forms.\u003c/p\u003e\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e\u003ch2\u003e1.2 Related works\u003c/h2\u003e\u003cp\u003eBoth machine and deep learning algorithms are able to customize mental health assessment for individual use since they can address multiple variables simultaneously [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. An exhaustive review by Iyortsuun et al (2020) [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] examined 33 studies and concluded that deep learning-based models proved to be more accurate in diagnosing schizophrenia, depression, anxiety, bipolar disorder and post-traumatic stress disorder than conventional approaches. Shatte et al (2019) [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] investigated deep learning\u0026rsquo;s capacity to evaluate intricate data sets, predict the prognosis and enable personalized interventions, highlighting its revolutionary influence on mental health outcome research. Together, these reviews illustrate how AI-based methods are improving diagnostic precision and making more effective, personalized treatment approaches in psychiatric treatment.\u003c/p\u003e\u003cp\u003eNatural language processing (NLP) has proven itself to be a promising domain for psychological studies by drawing on textual data from various sources like social media messages, clinician notes and patient interviews in order to classify, identify and predict mental disorders. Malgaroli et al. (2023) [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] conducted a systematic review on the application of NLP in Mental Health Interventions (MHI) and consequently developed a research framework to address existing challenges. Results indicated a significant increase in NLP-based MHI studies since 2019 where large language models (LLMs) played a dominant role in data analysis. The review reported that text-based features contributed more to model accuracy than audio-based data and digital health platforms were the primary sources of MHI data.\u003c/p\u003e\u003cp\u003eTorfi et al. (2020) [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] performed a survey on the development of natural language processing (NLP) fueled by deep learning with an emphasis on the manner in which computational advancements and access to large linguistic corpora have revolutionized the discipline. Their aim was to examine the role of deep learning models in NLP operations such as semantic analysis, part-of-speech tagging, named entity recognition and sentiment classification. Their approach involved discussing state-of-the-art deep learning methods such as recurrent neural networks (RNNs), convolutional neural networks (CNNs) and transformer-based models such as BERT, noting their application in automating and enhancing linguistic tasks. Results reported that deep learning greatly improves NLP applications, especially in task-dependent applications such as machine translation, text summarization and sentiment analysis. BERT and GPT transformer models showed better language understanding performance than previous models and were able to effectively capture text long-range dependencies. The work also highlighted grand challenges, e.g., access to large quantities of annotated training data, issues of interpretability and ethical risks of biases within AI-generated content.\u003c/p\u003e\u003cp\u003eTeferra et al. (2024) [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] systemically searched from databases such as Semantic Scholar, PubMed and Google Scholar with research focus on screening depression using NLP techniques. The search term \"depression screening,\" \"depression detection,\" and \"natural language processing\" was applied and for inclusion studies were considered to be related if they were addressing the application of NLP techniques in screening or detecting depression. The review concluded that although there are possibilities to improve detection of depression using NLP, significant hurdles still exist in the form of ethical concerns and necessity for incorporating cross-cultural awareness.\u003c/p\u003e\u003cp\u003eUddin et al. (2021) [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] created a depressive symptom predictive system based on deep learning from text data with a Long Short-Term Memory (LSTM)-based Recurrent Neural Network (RNN) model to examine a huge collection of text submitted by youth on a Norwegian online forum. The model was trained using extraction of strong features representing depressive symptoms as defined by medical specialists such as psychiatrists. The explainable AI (XAI) method and Local Interpretable Model-Agnostic Explanations (LIME) was utilized to gain insights into model decisions. Their research focuses on how more improvement is needed in interpretability where the models did not just aim for correct predictions but also aimed to explain their decision-making processes to ensure better transparency and trustworthiness.\u003c/p\u003e\u003cp\u003eMilintsevich et al. (2022) [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] employed a multi-target hierarchical regression model to predict the severity of depression symptoms from DAIC-WOZ interview transcripts. The researchers used S-RoBERTa to generate embeddings for dialog turns and a Bi-LSTM with attention for interviews, trained over 200 epochs. Results demonstrated a Mean Absolute Error (MAE) of 0.438\u0026ndash;0.830 for symptom predictions, a macro-F1 score of 73.9 for binary classification and an MAE of 3.78 for total PHQ-8 scores which is competitive with state-of-the-art models. This symptom-focused NLP approach aligned with symptom network analysis offers significant potential for personalized prediction in clinical settings.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003e1.3 Supremacy of BERT in text classification\u003c/h2\u003e\u003cp\u003eNumerous studies have concluded that family of large language models such as Bidirectional Encoder Representations from Transformers (BERT) model [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] and its variants have established new standards in NLP tasks concerned with mental health analysis and outperform traditional models [21, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. The BERT model is superior to the traditional neural networks and long short-term memory (LSTM) models because it is based on Transformer architecture [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e] which enables it to learn context from the left and right sides of a token/sentence at the same time. Unlike LSTMs, which read inputs sequentially and are plagued by long-range dependency constraints, BERT leverages multi head self-attention mechanisms to capture sentence-level semantics or in technical terms global dependencies more effectively. In addition, whereas typical neural networks and LSTMs usually need task-specific architectures, pre-trained representations of BERT can be fine-tuned with little adaptation on a wide range of NLP tasks with minimal feature engineering and computational costs. Empirical performance shows BERT's state of-the-art performance on several benchmarks which far outperforms prior models in question answering and language inference tasks.\u003c/p\u003e\u003cp\u003eA BERT variant fine-tuned on a clinical dataset can overcome challenges in mental health infrastructure by examining text for patterns that might be missed by clinicians, providing a more comprehensive and impartial evaluation. Through the examination of clinical interview text, it can detect symptoms early on especially in regions with limited mental health professionals which enhances care accessibility in underserved areas and facilitates early intervention. This is in line with the increasing worldwide trend of using AI in healthcare systems to improve outcomes [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\u003ch2\u003e1.4 The present study\u003c/h2\u003e\u003cp\u003eDespite the advances in areas of NLP, deep and machine learning and medicine, a key research gap identified is lack of model interpretability and no reports on generalization to real-world settings. Thus, most of the works have halted at performance metrics and token-level explanations without validating the model in actual contexts. The original studies and systematic reviews undertaken in these areas have suggested to increase interpretability, personalization, and the addressing of bias in order to better use NLP for the detection of depression [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Therefore, given the rising global prevalence of Major Depressive disorder, limitations in healthcare accessibility and infrastructure, suggestions and the gaps identified, the present study aims to (i) assess the efficacy of a BERT variant for predicting depression through text, (ii) compare the model\u0026rsquo;s predictions against a clinician\u0026rsquo;s to assess its generalizability and (iii) further analyze the model\u0026rsquo;s decision making both qualitatively and quantitatively. Thus, the current study adds to the expanding body of evidence for the potential of AI to serve as a useful resource for enhancing mental health care, especially in resource-scarce settings.\u003c/p\u003e\u003c/div\u003e"},{"header":"2. METHOD","content":"\u003cp\u003eThe study employed a mixed method based explanatory sequential research design which aims to assess to what extent an agreement exists between a fine-tuned RoBERTa model and a licensed clinician for predicting depression among Indian adult population. The interviews which were disagreed upon by the raters underwent a thematic analysis to better understand the model\u0026rsquo;s decision and potentially suggest steps to improve feature extraction and feature engineering for future studies.\u003c/p\u003e\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\u003ch2\u003e2.1 Ethics\u003c/h2\u003e\u003cp\u003eThe study was reviewed and approved by the School Ethics Review Board of School of Behavioral Forensics, NFSU, Gandhinagar between November 2024 to January 2025. The data collection proceeded from first week of February 2025 till 22nd April 2025. For clinical population, the permission of interviewing in-ward patients was obtained from nursing staff and informed consent was obtained from the patients with the interviews being conducted in the presence of the nursing staff only. For OPD based patients, the informed consent was obtained from the patient and/or informants and the interviews were conducted in presence of a clinical psychologist. Inorder to obtain consent from non-clinical population, google forms were circulated online.\u003c/p\u003e\u003cp\u003eConfidentiality was maintained throughout the process where patients from clinical group were identified by codes such as C01, C02, etc. and non-clinical group was identified as NC01, NC02, etc.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\u003ch2\u003e2.2 Sample\u003c/h2\u003e\u003cp\u003eConvenient and purposive sampling were used in the current study where available population members in convenience who met the inclusion criteria were sampled. The population of research interest was Indian adults in both clinical and non-clinical settings. The sampling frame consists of 79 individuals with 22 patients from OPD and 26 patients were selected from wards of Hospital for Mental Health, Ahmedabad. The non-clinical population comprised of 31 Indian adults who based on personal judgement of the researcher and their self-report were selected due to no prior history of chief complaints and past diagnosis. The sample consisted of 37 females and 42 males within the age range 18 to 81 years. The clinical group was interviewed in person while the non-clinical group was interviewed either in-person or on video conferencing, based upon the preference given by the participants.\u003c/p\u003e\u003cp\u003e 70 out of 79 interviews were conducted in participants\u0026rsquo; mother tongue. Thus, the interviews underwent both forward translation into English and backward translation by SV and to ensure consistencies, reduce biases and errors in translation.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eDiagnostic status of clinical group (N\u0026thinsp;=\u0026thinsp;48)\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDIAGNOSIS\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eNo. of patients\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAlcohol use disorder (AUD)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e3\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eBipolar Mood disorder-I\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e5\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eBorderline Personality disorder\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCannabis and Tobacco use disorder\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eIntellectual disability (IDD)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMajor depressive disorder (MDD)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMDD\u0026thinsp;+\u0026thinsp;psychotic features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eObsessive-compulsive disorder\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eParanoid schizophrenia\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePending*\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e22\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSchizoaffective\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSchizophrenia\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e5\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSchizophrenia\u0026thinsp;+\u0026thinsp;AUD\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSchizophrenia\u0026thinsp;+\u0026thinsp;IDD\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"2\"\u003e\u003cem\u003e* used for OPD based patients who had not been formally diagnosed till the time of interview\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003e2.3 Dataset\u003c/h2\u003e\u003cp\u003eThe DAIC-WOZ Depression Database is a subset of the Distress Analysis Interview Corpus (DAIC), designed to aid in diagnosing psychological conditions such as depression and PTSD [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. It includes approximately 50 hours of clinical interviews collected for developing a virtual interviewer named Ellie, controlled by a human operator, to identify verbal and nonverbal indicators of mental illness [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The dataset consists of 189 recorded sessions (with some exclusions and special cases), providing audio, video, transcriptions and various extracted features, including facial expressions, gaze and acoustic properties. Transcripts adhere to specific annotation guidelines and some audio sections have been scrubbed to protect participant privacy. The dataset is intended for use in AI-driven mental health assessments and computational analysis of psychological distress. The dataset was a part of AVEC 2017 challenge where a PHQ-8 score of 10 and above indicated depression. The PHQ-8 interview schedule in DAIC-WOZ serves as a supplement to the clinical interviews which are the gold standard for diagnosis [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eThe DAIC-WOZ dataset contains 189 interviews that have been split into training, validation and testing sets. Since 47 interviews in test set were spare with no PHQ binary, the current study used the 107 interviews for training and validating while the remaining 35 interviews for testing the model.\u003c/p\u003e\u003cdiv id=\"Sec9\" class=\"Section3\"\u003e\u003ch2\u003e2.3.1 Data preprocessing\u003c/h2\u003e\u003cp\u003eA significant change from previous studies is not including prompts of the interviewer to train the model. Burdisso et al. (2024) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] found that incorporating the interviewer\u0026rsquo;s prompts into depression detection models from the DAIC-WOZ dataset introduces a bias that significantly improves classification performance but not necessarily due to a better understanding of participants' distress. Models using interviewer\u0026rsquo;s prompts tended to focus on a specific section of interviews where questions about past mental health experiences were asked, effectively exploiting this bias to distinguish between depressed and control participants. By leveraging this shortcut, models achieved an F1 score of 0.90, the highest reported using only textual data.\u003c/p\u003e\u003cp\u003eIn the original dataset, the responses to the same question have been split into multiple samples for the same utterance due to differences in timestamps. At present, the timestamps were removed and responses for a single utterance were merged into one sample due to which the original number of samples were reduced even further.\u003c/p\u003e\u003cp\u003eThe dataset was converted to lowercase, cleaned to remove punctuations and unnecessary utterances (\u0026lsquo;mhm\u0026rsquo; or \u0026lsquo;hmm\u0026rsquo;), spaces, and contractions (don\u0026rsquo;t, won\u0026rsquo;t, etc.) were expanded. Instead of opting for automated lexicon- based approaches [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] for identifying depression markers from the corpus, we decided to opt for manual labelling to ensure complexity, nuances and subtlety of human language is not missed. An exhaustive process was undertaken to categorize responses in both training and test datasets. There are questions that are unable to differentiate between a depressed and non-depressed participants such as- \u0026ldquo;What's your dream job\u0026rdquo;, \u0026ldquo;Where are you from originally\u0026rdquo; and other fillers. Thus, the responses to these filler questions have been automatically categorized as non-depressed samples despite them coming from an individual exhibiting symptoms of depression. For questions which appeared to be clinically relevant SV went through the interviews to manually label responses as either 0 or 1 based on DSM-5\u0026rsquo;s criteria for Major Depressive Disorder. Additionally, social support was also considered as an important feature in line with existing literature [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. AP verified the categorization by reviewing 10 interviews chosen from training split. After the suggested changes and agreement between both the authors on categorization of responses, a set of 2974 samples was used as training and validation set while another set of 1133 samples from 35 interviews was used as the test set.\u003c/p\u003e\u003cp\u003eInorder to affirm that the categorization process was indeed suitable for differentiating between the classes, average semantic similarity scores were computed for both train and test sets. Each statement was tokenized and encoded in batches with embeddings derived from the mean-pooled last hidden state of the model\u0026rsquo;s output whilst taking attention masks into account. Cosine similarity was then computed pairwise: (1) within each class to assess intra-group semantic similarity and (2) across the two classes to evaluate inter-group similarity. For each case, the mean of the similarity matrix (excluding diagonal self-similarity) was reported as the semantic similarity index to provide an extent to which both classes were distinct. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the obtained index values across three conditions for both sets.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eAverage similarity index for final train and test sets\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eComparison\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMean Similarity (Train)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMean Similarity (Test)\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eWithin class 0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.6250\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.6639\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eWithin class 1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.7656\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.7107\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eBetween\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.3880\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.4547\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eFrom Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, it can be inferred that the manual categorization was indeed effective especially when it came to discriminate between the two classes. The similarity within non-depressed class (class 0) was less in both sets than that of depressed class (class 1). This observation can be attributed to variability in responses to questions which addressed participants\u0026rsquo; hobbies, living situations, aspirations, etc. While the stronger cohesion within depressed class exhibits similar usage of language to express common themes that align with symptomatic criteria of depression.\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e\u003ch2\u003e2.4 Measures\u003c/h2\u003e\u003cp\u003eRoBERTa (A Robustly Optimized BERT Pretraining Approach) is a transformer language model that extends BERT with enhanced training methods like longer training durations, elimination of the next sentence prediction task, and dynamic masking. It has 768 dimensions and has been trained on around 125\u0026nbsp;million parameters which also supports up to 512 tokens. The model was trained for five epochs using a linear warm-up and decay schedule where the peak learning rate was set to 3e-5 while applying a constant hidden-layer dropout of 0.2. The original set of 2794 samples consisted of 1972 samples from non-depressed class while 822 samples from depressed class. To address this class imbalance, we used random undersampling for majority class and brought both classes to 1:1 ratio. Finally, an early-stopping callback with a patience of one epoch was employed to stop training once validation loss failed to improve.\u003c/p\u003e\u003cp\u003eThe interview schedule used in current study was a modification of original DAIC-WOZ\u0026rsquo;s interview schedule which is based on PHQ-8. In context of the current research the interviews were to be conducted in clinical settings (OPD and wards) where time efficiency was must due to larger number of patients and resource restraints. This concern worked both ways since patients and/or their informants were aware that there was a problem that needed to be addressed immediately. Thus, the questions asked aimed to directly address the chief complaints and symptoms of patients within a shorter period of time. Hence, the filler questions that aim to establish rapport and enquire clinically irrelevant details have been removed. Dr. AP and Dr. SV who are clinical psychologists, provided their expert judgement on the interview schedule for its content validity. The experts determined if the original 18 items in the modified interview schedule were relevant in Indian context. They agreed on relevance of 16 items out of 18, whereupon after further discussion it was decided to modify the two items to match the appropriate context. Thus, the final interview schedule used in current study had 13 main questions with the last question having five sub-parts as provided in Appendix A.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\u003ch2\u003e2.5 Data analysis\u003c/h2\u003e\u003cp\u003ePerformance metrics- sensitivity, precision, accuracy and macro-averaged F1 scores were reported on the test split. F1 score is of primary importance in the current work since it balances precision and recall, making it relevant in cases of class imbalance. The 79 interviews were examined by AP to determine whether the individuals exhibited indicators of depression and provided as an input to the fine-tuned RoBERTa model inorder to obtain its decision for indicators of depression. Jamovi, an open-source software for quantitative analysis was used to calculate the Cohen\u0026rsquo;s Kappa coefficient for inter-rater reliability between AP and the model.\u003c/p\u003e\u003cp\u003eAdditionally, the interviews upon which agreement couldn\u0026rsquo;t be reached between the two raters, underwent a thorough analysis both qualitatively and quantitatively. For the purpose of qualitative analysis, the interviews on which both the raters disagreed were analyzed following the blueprint of Braun \u0026amp; Clarke\u0026rsquo;s thematic analysis [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] to code and develop themes with an inductive approach. SV reviewed all the eight interviews multiple times to gain familiarity with the data and generate around 94 codes initially which after revision were reduced to 80 codes. SV identified the patterns prevalent across these codes to develop initial themes and sub-themes. After modifications finally a set of six themes emerged from the dataset. The iterative process of mapping the codes, categorizing them into groups to identify and revise the sub-themes and themes was done in an online open-sourced software Taguette by SV. The quantitative analysis was done by obtaining token attribution scores for one randomly selected interview out of the set of misclassified interviews.\u003c/p\u003e\u003c/div\u003e"},{"header":"3. RESULTS","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\u003ch2\u003e3.1 Performance metrics\u003c/h2\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eClassification report of fine-tuned Roberta-base model\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eClass\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003ePrecision\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eRecall\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eF1-Score\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eSupport\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e0 (Not depressed)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.96\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.88\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.92\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e914\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e1 (Depressed)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.63\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.83\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.71\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e219\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAccuracy\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.87\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMacro Avg\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.79\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.86\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e\u003cb\u003e0.82\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eWeighted Avg\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.89\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.87\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.88\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eIn Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, it can be observed that the model has strong Precision of 0.96 and 0.88\u003c/p\u003e\u003cp\u003erecall for non-depressed class while it has a similar recall for depressed class, it has a moderate\u003c/p\u003e\u003cp\u003eprecision of 0.63 indicating the risk of generating false positives. The model has an accuracy of\u003c/p\u003e\u003cp\u003e87% with an F1- score with weighted average of 0.88 and a macro average of 0.82 reflecting an\u003c/p\u003e\u003cp\u003eoverall strong performance despite major class imbalance.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\u003ch2\u003e3.2 Inter-rater reliability estimates\u003c/h2\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eContingency table for rater and responses\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"4\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eModel: Not Depressed\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eModel: Depressed\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eTotal (Rater)\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAP: Not Depressed\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e61\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e8\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e69\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAP: Depressed\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e10\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e10\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTotal (Model)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e61\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e18\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e79\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eOut of 79 interviews 48 interviews belonged to clinical population while 31 interviews were taken with non-clinical population. In clinical population 22 interviews were conducted freshly with patients from OPD and hence there was no formal diagnosis available for them while the rest 28 interviews were conducted with patients from ward who were diagnosed formally on basis of a mutual decision given by psychiatrists and clinical psychologists with assessments and multiple sessions (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Out of these 28 patients there were four patients diagnosed with Major Depressive Disorder including comorbid features and 5 patients were diagnosed with Bipolar Mood Disorder-I without any comorbidities. As depicted in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, there were eight interviews where AP and the model had a disagreement. Out of these, one interview belonged to an inward patient of Alcohol use Disorder while the rest seven interviews belonged to OPD based patients. Thus, it is evident that the classification report obtained from test set of DAIC-WOZ is in alignment with interviews conducted on Indian population where the model is generating false positives while successfully identifying actual cases of depression.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eEstimate of Cohen\u0026rsquo;s Kappa\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eStatistic\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eValue\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eNumber of subjects\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e79\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eNumber of raters\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePercent agreement\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e89%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCohen\u0026rsquo;s kappa (κ)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e0.628\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cem\u003ep\u003c/em\u003e-value\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eFrom Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, it can be concluded that for 79 interviews, the obtained Cohen\u0026rsquo;s Kappa\u003c/p\u003e\u003cp\u003ecoefficient was 0.628 indicating a substantial level of agreement between the two raters. For the\u003c/p\u003e\u003cp\u003eobserved agreement of 89% and the obtained coefficient, the p\u0026thinsp;\u0026lt;\u0026thinsp;0.001 indicating the results are\u003c/p\u003e\u003cp\u003estatistically significant and unlikely due to chance.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e\u003ch2\u003e3.3 Analysis of false positives\u003c/h2\u003e\u003cp\u003eThe eight interviews upon which the model and AP disagreed underwent a qualitative and quantitative analysis to gain a deeper understanding of the inconsistencies in depression prediction. This dual approach was employed to move beyond descriptives such as performance metrics and Cohen's kappa and instead explore the reasons behind the disagreements for further model refinement. By integrating qualitative insights such as thematic analysis with quantitative measures like token attribution scores, the analysis aimed to identify specific patterns and potential biases in the model\u0026rsquo;s predictions.\u003c/p\u003e\u003cp\u003eThe thematic analysis revealed that while many statements aligned with depression, they were not uniquely indicative of it and their cause did not lie solely in affective afflictions but other factors too (social, substance use, physical difficulties, etc.). Below, each theme is discussed in detail, highlighting its contribution to false positives and its interplay with other themes.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cdiv id=\"Sec16\" class=\"Section3\"\u003e\u003ch2\u003e3.3.1 Discussion of themes\u003c/h2\u003e\u003cdiv id=\"Sec17\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.1 Psychological Symptoms\u003c/h2\u003e\u003cp\u003eThe Psychological Symptoms theme includes emotional, behavioral, and cognitive disturbances that superficially resemble depression. Emotional Distress which is characterized by codes like \"I have been feeling empty\" and \"After my engagement was broken, I have felt worthless sometimes,\" suggests significant affliction in mood. However, these afflictions may stem from situational stressors such as relationship breakdowns rather than a depressive disorder. For example, Participant C24's worthlessness following a broken engagement could reflect acute grief rather than chronic depression. Behavioral Disturbances, such as \"I sit idle at one place all day\" or \"(informant) She throw things around the house,\" may indicate irritability associated with depression. Yet, these behaviors could also reflect frustration, anxiety, or personality traits, particularly when paired with interpersonal conflicts. Cognitive disturbances, including \"I cannot concentrate on my work\" and \"I have been overthinking for a month now,\" are similar to cognitive impairment observed in depression but may also signal anxiety or stress-related rumination. The overlap of these symptoms with multiple conditions determines the risk of false positives when models prioritize symptom presence over context and cause.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec18\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.2 Physical Symptoms\u003c/h2\u003e\u003cp\u003ePhysical Symptoms are divided into subthemes of those indicative of depression (e.g., sleep disturbances, psychomotor retardation) and somatic complaints not indicative of depression (e.g., headaches, dizziness). Symptoms like \"I have been slower than before\" (C24) or \"I do not eat\" (C25) are classic depression markers but are not specific to it. For example, Participant C26's sleep issues (\"I cannot sleep when people are always fighting outside\") appear as result from environmental stressors rather than depression. Somatic complaints, such as \"usually dizzy most of the day\" (C21) or \"I am old so I cannot have much energy\" (C32), further complicate diagnosis, as they may reflect physical illnesses, aging, or substance use effects (C26). These findings align with existing literature indicating that somatic symptoms are prevalent in non-depressed populations, particularly among older adults or those with chronic conditions [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e].\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec19\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.3 Interpersonal Factors\u003c/h2\u003e\u003cp\u003eInterpersonal Factors including conflicts and familial relationships play a significant role in manifestation of symptoms. Codes like \"I have frequent fights with my family especially my wife\" (C47) and \"I cook my own food and wash my own clothes\" (C21) suggest social withdrawal and strained relationships which can intensify emotional distress. However, these interpersonal dynamics may be primary causes of distress rather than secondary to a depressive disorder.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec20\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.4 Treatment and History\u003c/h2\u003e\u003cp\u003eThe Treatment and History theme provides critical context for interpreting current symptoms. Codes like \"He [Doctor] said I have a mental illness\" (C32) indicate prior mental health engagement which may bias automated models to flag depression. Similarly, past symptoms such as \"I was under stress\" or \"I cannot remember much but I used to drink a lot\" (C26) suggest a history of past distress or substance use that complicates current symptom interpretation for the model. For example, Participant C26's alcohol use history may explain physical symptoms like vomiting which a model might falsely attribute to depression. This theme highlights the need for longitudinal data to differentiate chronic depressive symptoms from transient or substance-related issues, a challenge for automated systems relying on single-point assessments.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec21\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.5 Response Style\u003c/h2\u003e\u003cp\u003eContradictory Responses, observed in seven of eight participants, are a key contributor to false positives. Statements like \"I am fine... I have been feeling empty\" (C09) or \"I get angry on small things these days... but mostly I am happy\" (C25) reflect inconsistent self-reporting, potentially due to denial, fluctuating moods, or lack of insight. Such contradictions may exaggerate symptom scores in automated models as positive and negative statements are both counted leading to erroneous flagging. This aligns with research suggesting that patient denial or stigma can distort self-reports which can complicate depression diagnosis.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec22\" class=\"Section4\"\u003e\u003ch2\u003e3.3.1.6 Symptoms of Other Disorders\u003c/h2\u003e\u003cp\u003eThe presence of non-depressive symptoms, such as those related to Alcohol Use Disorder (e.g., \"I vomit a lot due to drinking\" in C26) or paranoia (e.g., \"I feel as if they are against me\" in C32), significantly complicates diagnosis. Paranoia, reported by Participants C21 and C32, may indicate conditions like anxiety, personality disorders, or psychosis rather than depression. Similarly, alcohol-related symptoms in C26 overlap with depressive physical complaints but have a distinct etiology. These findings echo studies highlighting the diagnostic challenge of distinguishing depression from other psychiatric conditions with overlapping symptoms. Automated models may fail to weight these non-depressive symptoms appropriately, leading to false positives.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eTheme matrix\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"7\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eID\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003econtradictions\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003ePsychological complaints\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003ePhysical complaints\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eInterpersonal factors\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003ePast history\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c7\"\u003e\u003cp\u003eOther symptoms\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC09\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC21\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC22\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC24\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC25\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC26\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC32\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eC47\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e✔\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eThe identified themes can lead to false positives in the model by emphasizing symptoms that overlap with depression but arise from alternative causes as discussed above. The code matrix illustrates this complexity, showing unique combinations of themes that might confuse a model relying on symptom presence alone. For instance, Participant C09 exhibits Emotional and behavioral disturbances which are classic depression markers. However, presence of contradictory responses suggest unreliable reporting, possibly due to situational distress or denial which the model might misinterpret as depression. Participant C21 exhibits complex cluster of symptoms ranging from Emotional Distress, Physical Complaints, Other Symptoms (e.g., suspiciousness) to Interpersonal Conflicts. In such cases the model might flag this as depression even in the presence of non-depressive symptoms like paranoia that indicate a different condition. Somatic complaints are prevalent across participants (e.g., C24, C25, C32) which might have been attributed to depression by the model but external factors (e.g., noisy environments, aging, or alcohol use, as in \u0026ldquo;I vomit a lot due to drinking\u0026rdquo; for C26) appear to be the root cause. Somatic symptoms of depression often overlap with physical illnesses due to which diagnosing depression becomes complicated (Mitchell et al., 2009). Similarly, Interpersonal Conflict in participants like C47 might amplify emotional distress, leading the model to flag for depression without considering relational context. In the present context the risk of false positives exists because the model fails to weigh the specificity of symptoms or their co-occurrence with non-depressive indicators (e.g., \u0026ldquo;I feel as if they are against me\u0026rdquo; in C32). Contradictory responses further complicate matters as inconsistent self-reports might inflate symptom scores erroneously. Thus, the model\u0026rsquo;s reliance on surface-level symptom matching without contextual nuance increases its risk for predicting false positives.\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Sec23\" class=\"Section3\"\u003e\u003ch2\u003e3.3.2 Attribution scores\u003c/h2\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eFor the purpose of gaining quantitative insight into model\u0026rsquo;s decision, out of eight interviews, one was randomly chosen via fishbowl method to obtain the tokens with fifteen highest attribution scores. The results for C25\u0026rsquo;s interview depict \"irritated\" (+\u0026thinsp;0.6) and \"suspicious\" (+\u0026thinsp;0.55) are the most influential tokens driving the model\u0026rsquo;s prediction of depression, indicating a strong focus on affective symptoms and paranoia. Tokens such as \u0026ldquo;more\u0026rdquo; (+\u0026thinsp;0.25), \"less\" (+\u0026thinsp;0.15) and \"decreased\" (+\u0026thinsp;0.15) are contributing factors to the model\u0026rsquo;s classification, though their influence is moderate compared to higher-scoring tokens. The tokens \"less\" and \u0026ldquo;decreased\u0026rdquo; are often associated with statements like reduced appetite or energy (e.g., \"I eat less\"), which directly signals a decline in typical functioning contributing to the class 1 prediction by emphasizing loss or reduction, a common depressive marker. Similarly, \"more\" (+\u0026thinsp;0.25) indicates an increment in intensity of symptom, further supporting the class 1 prediction by suggesting a worsening state. Individually, these tokens contribute to the prediction by perhaps aligning with hallmark depression indicators like reduced functioning, declining states and periodic emotional struggles, though their moderate attribution scores suggest they are secondary drivers compared to more emotionally charged tokens. Tokens like \u0026ldquo;more\u0026rdquo;, \"anger\", and \"become\" also contribute, though to a lesser extent reflecting attention to affect and markers in change. This aligns with the thematic analysis, as \u0026ldquo;irritated\u0026rdquo; and \"anger\" correspond to Theme of Psychological Symptoms specifically emotional distress, highlighting the model\u0026rsquo;s tendency to overgeneralize affective language without considering situational factors like grief or stress. Similarly, the high attribution score of \"suspicious\" aligns with symptoms of other disorders confirming the challenge in isolating psychotic symptoms when depression indicators are present, despite its ability to differentiate them in their absence.\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e"},{"header":"4. DISCUSSION","content":"\u003cp\u003eThe present study aims to assess how well a RoBERTa model, fine-tuned on DAIC WOZ dataset generalizes to Indian population by assessing inter-rater reliability between a clinical psychologist and the model for indicators of depression in 79 interviews (48 in clinical group and 31 in non-clinical). Further, to gain a deeper understanding behind model\u0026rsquo;s decision making, the interviews which were disagreed upon by both the raters underwent thematic analysis and one interview was randomly selected for obtaining token attribution scores.\u003c/p\u003e\u003cp\u003eDespite the nature of scarce data and manual labelling which might have led to biases in both test and training sets, the RoBERTa model has achieved a macro F1 score of 0.82, performing in accordance with studies using text classification to predict depression [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]. The overall accuracy of the model is 0.87, implying that 87% of the total predictions across both classes are correct. While this seems high on its own, accuracy alone can be misleading especially in imbalanced datasets such as that in current study where non-depressed has significantly more instances than depressed. Therefore, it's important to also consider the macro-averaged F1-score of 0.82 (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e), which balances precision and recall. For the non-depressed class, the model demonstrates a strong precision of 0.96, which is that when it classifies an individual as non-depressed, it is correct a majority of the time. Its recall for the same class is also quite high at 0.88 which implies it gets 88% of all actual Class 0 instances correct. As compared, the depressed class has much less precision of only 0.63, meaning a higher level of false positives. Still, its recall is relatively stronger at 0.83, showing that it catches most actual cases of depression accurately. This trade-off between precision and recall for depressed class means that although the model is accurate at identifying depressed people when they exist, it is less accurate in validating those predictions because it has a greater number of false alarms. Thus, the model may benefit from further tuning or resampling techniques.\u003c/p\u003e\u003cp\u003eOut of 79 interviews conducted the model and rater disagreed over eight interviews with a significant Cohen\u0026rsquo;s Kappa of 0.628, where all these interviews according to the clinician fell under non-depressed category but the model predicted the opposite class (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). The results from fine tuning the model are reflected in its generalizability where it can be observed that the model\u0026rsquo;s ability to catch all cases of depression whilst producing false positives is a significant limitation to overcome. Qualitative (thematic analysis) and quantitative (attribution scores) analysis of these eight interviews reveals that symptoms mimicking depression often arise from situational stressors, substance use or other disorders, leading to false positives in automated models. The thematic analysis identified six key themes (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e): Psychological Symptoms, Physical Symptoms, Interpersonal Factors, Treatment and History, Response Style, and Symptoms of Other Disorders; revealing that symptoms resembling depression such as emotional distress (e.g., \"I have been feeling empty\"), somatic complaints (e.g., \"I cannot sleep\") and interpersonal conflicts (e.g., \"I have frequent fights with my family\"), often arise from situational stressors, substance use, or other psychological conditions like anxiety or paranoia rather than depression. For example, Participant C24\u0026rsquo;s reported worthlessness was linked to a recent broken engagement, suggesting acute grief rather than chronic depression, while Participant C26\u0026rsquo;s sleep disturbances were attributed to environmental disturbances rather than a depressive condition. The quantitative analysis of token attribution scores for Participant C25\u0026rsquo;s interview (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) clarified the model\u0026rsquo;s decision-making process. Tokens such as \"irritated\" (+\u0026thinsp;0.60), \"suspicious\" (+\u0026thinsp;0.55), \"less\" (+\u0026thinsp;0.15), and \"decreased\" (+\u0026thinsp;0.15) were highly influential in driving the model\u0026rsquo;s depression prediction, reflecting its focus on affective and functional decline markers. However, the high attribution score for \"suspicious\" aligns with the findings of thematic analysis indicating the model\u0026rsquo;s challenge in distinguishing paranoia or anxiety-related symptoms from depression. Similarly, tokens like \"less\" and \"decreased,\" often tied to reduced appetite or energy correspond to the somatic symptoms theme but lack contextual specificity leading to erroneous flagging when these symptoms stem from non-depressive causes like aging or substance use. Contradictory self-reports, prevalent in seven of the eight disagreed cases further complicate predictions, as the model may overemphasize both positive and negative statements.\u003c/p\u003e\u003cdiv id=\"Sec25\" class=\"Section2\"\u003e\u003ch2\u003e4.1 Limitations and future implications\u003c/h2\u003e\u003cp\u003eIn context of the present study, the fine-tuned RoBERTa model demonstrates strong potential for use of AI in screening depression among Indian adults within clinical settings while closely aligning with a clinicians' assessments. However, there are certain limitations that can be overcome to improve the results such as the lack of a clinical dataset specific to the Indian context, significant class imbalance in training and test sets and the \u0026lsquo;Lost in Translation\u0026rsquo; effect i.e. reliance on manual categorization and transcription of interviews which might introduce biases and errors. The model's performance is limited by its inability to incorporate audio and visual cues and its decision criteria could be refined to identify specific areas of disturbance rather than solely detecting depression indicators based on symptoms.\u003c/p\u003e\u003cp\u003eThe results offer practical implications for AI in clinical environments, enabling rapid depression screening via short interviews aligned with DSM-5 criteria in high-demand settings. Future studies can leverage multilingual clinical BERT variants to mitigate translation errors and incorporate audio and visual cues to reduce manual input, potentially enhancing screening and diagnostic processes. The model operates efficiently on standard laptops and desktops, requiring no advanced hardware and henceforth, future iterations incorporating additional features must prioritize accessibility. Expanding rater diversity beyond clinicians, such as including psychiatrists and psychiatric social workers, could further validate the model\u0026rsquo;s reliability across different domains.\u003c/p\u003e\u003c/div\u003e"},{"header":"5. CONCLUSION","content":"\u003cp\u003e\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003eThe current study examined the potential of AI in clinical settings through a fine-tuned RoBERTa model for predicting depression among Indian adults, the decisions of which aligned closely with the decision of a clinical psychologist. Despite robust performance, false positives arose due to symptoms of depression overlapping with situational stressors and other disorders. This problem emerged due to inconsistent reporting by patients, situational stressors and other psychological conditions overlapping with classic symptoms of depression.\u003c/p\u003e\u003cp\u003eThese findings align closely with existing literature leveraging BERT variants and DAIC-WOZ dataset which have highlighted the significance of text-based detection of depression and other disorders. However, as discussed earlier, the study noted challenges such as risk of false positives which can be countered by an exhaustive analysis of model\u0026rsquo;s decision making to further improve its explainability, choice of datasets and feature extraction techniques.\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"APPENDIX","content":"\u003cp\u003e\u003cb\u003eA: The interview schedule used in current study\u003c/b\u003e\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e\u003ccolgroup cols=\"1\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003e1. How are you doing today\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e2. How close are you to your family\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e3. Do you have a close relationship with anyone else other than your family\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e4. Over the past two weeks have you been frequently experiencing anger or irritability that\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003efeels out of control\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e5. How easy is it for you to get a good night's sleep\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e6. Over the past two weeks has your interest or pleasure in doing things changed\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e7. How have you been feeling for the past two weeks\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e8. How is your energy level for the past two weeks\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e9. Over the past two weeks have you been overeating or do you not feel hungry at all\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e10. Within the past two weeks have you ever felt worthless or guilty\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e11. Over the past two weeks can you concentrate properly on the activities you usually do\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e12. Over the past two weeks have you noticed any change in your movement, are you too\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eslow or have you been restless and fidgety\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e13. Have you previously seen a doctor regarding these problems. Follow up questions:\u003c/p\u003e\u003cp\u003ea. What decision was given by the doctor\u003c/p\u003e\u003cp\u003eb. How long ago was your disease identified\u003c/p\u003e \u003cp\u003ec. What were your symptoms\u003c/p\u003e\u003cp\u003ed. What got you to seek help\u003c/p\u003e\u003cp\u003ee. Did you think you had a problem before you found out\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eFunding\u003c/h2\u003e\u003cp\u003eThis research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eS.V. worked on Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Visualization, Writing \u0026ndash; original draft and editing. Dr. A.P. supervised the work and assisted with Conceptualization, Methodology, Resources, Project administration, Formal Analysis, Validation and Writing \u0026ndash; review. Dr. S.V. co-supervised and assisted with Conceptualization, Validation and Writing- review\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eAlthough the sets of interviews used to train, test and calculate inter-rater rater reliability are not available due to ethics surrounding patient confidentiality, the code used to generate results is open available in author's Git repository at https://github.com/sv6121096/RoBERTa-for-depression\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAlhuwaydi AM (2024) Exploring the role of artificial intelligence in mental healthcare: current trends and future directions\u0026ndash;a narrative review for a comprehensive insight. Risk Manage Healthc Policy 1339\u0026ndash;1348. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2147/rmhp.s461562\u003c/span\u003e\u003cspan address=\"10.2147/rmhp.s461562\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAmerican Psychiatric Association (2022) \u003cem\u003eDiagnostic and statistical manual of mental disorders\u003c/em\u003e (5th ed., text rev.). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1176/appi.books.9780890425787\u003c/span\u003e\u003cspan address=\"10.1176/appi.books.9780890425787\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBains N, Abdijadid S (2023), April 10 \u003cem\u003eMajor depressive disorder\u003c/em\u003e. In StatPearls. StatPearls Publishing. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ncbi.nlm.nih.gov/books/NBK559078/\u003c/span\u003e\u003cspan address=\"https://www.ncbi.nlm.nih.gov/books/NBK559078/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBraun V, Clarke V (2006) Using thematic analysis in psychology. Qualitative Res Psychol 3:77\u0026ndash;101. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1191/1478088706qp063oa\u003c/span\u003e\u003cspan address=\"10.1191/1478088706qp063oa\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBurdisso S, Reyes-Ram\u0026iacute;rez E, Villatoro-Tello E, S\u0026aacute;nchez-Vega F, L\u0026oacute;pez-Monroy P, Motlicek P (2024) DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews. \u003cem\u003earXiv preprint arXiv:2404.14463\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCha J, Kim S, Park E (2022) A lexicon-based approach to examine depression detection in social media: the case of Twitter and university community. Humanit Soc Sci Commun 9:325. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1057/s41599-022-01313-2\u003c/span\u003e\u003cspan address=\"10.1057/s41599-022-01313-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCruz-Gonzalez P, He AW, Lam EP, Ng IMC, Li MW, Hou R, Chan JN, Sahni Y, Guasch V, Miller N, Lau T (2025) Artificial intelligence in mental health care: a systematic review of diagnosis, monitoring, and intervention applications. Psychol Med 55:e18 \u0026amp; S\u0026aacute;nchez Vida\u0026ntilde;a, D. I. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1017/S0033291724003295\u003c/span\u003e\u003cspan address=\"10.1017/S0033291724003295\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDeSouza DD, Robin J, Gumus M, Yeung A (2021) Natural language processing as an emerging tool to detect late-life depression. Front Psychiatry 12:719125. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fpsyt.2021.719125\u003c/span\u003e\u003cspan address=\"10.3389/fpsyt.2021.719125\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDeVault D, Artstein R, Benn G, Dey T, Fast E, Gainer A, Georgila K, Gratch J, Hartholt A, Lhommet M, Lucas G, Marsella S, Morbini F, Nazarian A, Scherer S, Stratou G, Suri A, Traum D, Wood R, Xu Y, Rizzo A, Morency L-P (2014) SimSensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u0026rsquo;14), Paris\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDevlin J, Chang MW, Lee K, Toutanova K (2019), June Bert: Pre-training of deep bidirectional transformers for language understanding. In \u003cem\u003eProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)\u003c/em\u003e (pp. 4171\u0026ndash;4186). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1810.04805\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1810.04805\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDinkel H, Wu M, Yu K (2019) Text-based depression detection: What triggers an alert. \u003cem\u003earXiv preprint arXiv:1904.05154\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDubey A, Chahar R, Narang S (2024) A Comparative Analysis of Machine Learning Algorithms with Bagging and Boosting for Mental Health Assessment. Int J Electron Commun Eng 11:20\u0026ndash;39. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.14445/23488549/IJECE-V11I6P103\u003c/span\u003e\u003cspan address=\"10.14445/23488549/IJECE-V11I6P103\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDucharme J (2025), January 14 \u003cem\u003eDismissed and disbelieved, some Long COVID patients are pushed into psychiatric wards\u003c/em\u003e. \u003cem\u003eTime\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://time.com/7206080/long-covid-psychiatric-wards/\u003c/span\u003e\u003cspan address=\"https://time.com/7206080/long-covid-psychiatric-wards/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGratch J, Artstein R, Lucas GM, Stratou G, Scherer S, Nazarian A, Wood R, Boberg J, DeVault D, Marsella S, Traum DR The Distress Analysis Interview Corpus of human and computer interviews. InLREC 2014 May (pp. 3123\u0026ndash;3128)\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eIyortsuun NK, Kim SH, Jhon M, Yang HJ, Pant S (2023) A Review of Machine Learning and Deep Learning Approaches on Mental Health Diagnosis. Healthc (Basel Switzerland) 11(3):285. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/healthcare11030285\u003c/span\u003e\u003cspan address=\"10.3390/healthcare11030285\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKarmen C, Hsiung RC, Wetter T (2015) Screening internet forum participants for depression symptoms by assembling and enhancing multiple NLP methods. Comput Methods Programs Biomed 120(1):27\u0026ndash;36\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMalgaroli M, Hull TD, Zech JM, Althoff T (2023) Natural language processing for mental health interventions: a systematic review and research framework. Translational Psychiatry 13(1):309\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMao K, Wu Y, Chen J (2023) A systematic review on automated clinical depression diagnosis. npj Mental Health Res 2:20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s44184-023-00040-z\u003c/span\u003e\u003cspan address=\"10.1038/s44184-023-00040-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMilintsevich K, Sirts K, Dias Ga\u0026euml;l (2022) Towards Automatic Text-Based Estimation of Depression through Symptom Prediction. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.21203/rs.3.rs-2096670/v1\u003c/span\u003e\u003cspan address=\"10.21203/rs.3.rs-2096670/v1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMitchell AJ, Vaze A, Rao S (2009) Clinical diagnosis of depression in primary care: a meta-analysis. Lancet (London England) 374(9690):609\u0026ndash;619. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S0140-6736(09)60879-5\u003c/span\u003e\u003cspan address=\"10.1016/S0140-6736(09)60879-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMoradi M, Samwald M (2022) Deep learning, natural language processing, and explainable artificial intelligence in the biomedical domain. \u003cem\u003earXiv preprint arXiv:2202.12678\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNguyen T, Yates A, Zirikly A, Desmet B, Cohan A (2022) Improving the generalizability of depression detection by leveraging clinical interview schedules. arXiv preprint arXiv:2204.10432\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003ePoświata R, Perełkiewicz M (2022) OPI@LT-EDI-ACL2022: Detecting signs of depression from social media text using RoBERTa pre-trained language models. In \u003cem\u003eProceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion\u003c/em\u003e (pp. 276\u0026ndash;282). Association for Computational Linguistics. https://aclanthology.org/2022.ltedi-1.40/\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShatte ABR, Hutchinson DM, Teague SJ (2019) Machine learning in mental health: a scoping review of methods and applications. Psychol Med 49(9):1426\u0026ndash;1448. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1017/S0033291719000151\u003c/span\u003e\u003cspan address=\"10.1017/S0033291719000151\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSmith L, Hill N, Kokanovic R (2015) Experiences of depression, the role of social support and its impact on health outcomes. J Ment Health 24(6):342\u0026ndash;346. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3109/09638237.2014.954693\u003c/span\u003e\u003cspan address=\"10.3109/09638237.2014.954693\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSpitzer RL, Williams JB, Kroenke K, Linzer M, deGruy FV 3rd, Hahn SR, Brody D, Johnson JG (1994) Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA 272(22):1749\u0026ndash;1756\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eStewart RA, Patel TA, McDermott KA, Cougle JR (2022) Functional and structural social support in DSM-5 mood and anxiety disorders: A population-based study. J Affect Disord 308:528\u0026ndash;534. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jad.2022.04.026\u003c/span\u003e\u003cspan address=\"10.1016/j.jad.2022.04.026\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTadesse MM, Lin H, Xu B, Yang L (2019) Detection of depression-related posts in reddit social media forum. Ieee Access 7:44883\u0026ndash;44893. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ACCESS.2019.2909180\u003c/span\u003e\u003cspan address=\"10.1109/ACCESS.2019.2909180\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTeferra BG, Rueda A, Pang H, Valenzano R, Samavi R, Krishnan S, Bhat V (2024) Screening for Depression Using Natural Language Processing: Literature Review. Interact J Med Res 13:e55067. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2196/55067\u003c/span\u003e\u003cspan address=\"10.2196/55067\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTorfi A, Shirvani R, Keneshloo Y, Tavvaf, Nader, Fox, Edward (2020) Natural Language Processing Advancements By Deep Learning: A Survey. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.2003.01200\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2003.01200\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eUddin MZ, Dysthe KK, F\u0026oslash;lstad A et al (2022) Deep learning for prediction of depressive symptoms in a large textual dataset. Neural Comput Applic 34:721\u0026ndash;744. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00521-021-06426-4\u003c/span\u003e\u003cspan address=\"10.1007/s00521-021-06426-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., \u0026hellip; Polosukhin,I. (2017). Attention is all you need. Advances in neural information processing systems, 30.https://doi.org/10.48550/arXiv.1706.03762\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eZhang J, Guo Y (2024) Multilevel depression status detection based on fine-grained prompt learning. Pattern Recogn Lett 178:167\u0026ndash;173. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.patrec.2024.01.005\u003c/span\u003e\u003cspan address=\"10.1016/j.patrec.2024.01.005\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eZhang X, Li C, Chen W et al (2025) Optimizing depression detection in clinical doctor-patient interviews using a multi-instance learning framework. Sci Rep 15:6637. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-025-90117-w\u003c/span\u003e\u003cspan address=\"10.1038/s41598-025-90117-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Depression, AI, Deep learning, Mental health, Natural Language Processing","lastPublishedDoi":"10.21203/rs.3.rs-7431380/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7431380/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn recent years, the developments in Artificial intelligence (AI) has reshaped several industries and professions. Given the rising prevalence of depression and the inability of existing health infrastructure to address this, the current study was undertaken to investigate the potential of AI in clinical settings. The first step is assessing how language can provide insights into psychological states of individuals. For the same purpose RoBERTa, a transformers based deep learning model was fine-tuned on DAIC-WOZ dataset to predict depression among Indian adults. Additionally, interviews of Indian adults were conducted and analyzed by both the model and a clinical psychologist to predict indicators of depression. The model achieved a macro F1-score of 0.82 on test split of DAIC-WOZ, indicating robust performance despite class imbalance. Cohen\u0026rsquo;s kappa of 0.628 indicated a substantial agreement was reached between both the model and the rater on the interviews. However, as revealed by the thematic analysis and attribution scores for interviews which were disagreed upon, the model\u0026rsquo;s tendency to generate false positives highlights the need for enhanced contextual analysis. These findings reveal that the language of depression is universal in its essence while emphasizing the necessity for culturally tailored datasets and multimodal approaches to improve predictions in resource constraints.\u003c/p\u003e","manuscriptTitle":"Inter-rater Reliability of an LLM in Predicting Depression Among Indian Adults","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-07 13:07:58","doi":"10.21203/rs.3.rs-7431380/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b75df231-be6f-405c-9b57-544f1af080cb","owner":[],"postedDate":"November 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-01-11T03:38:32+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-07 13:07:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7431380","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7431380","identity":"rs-7431380","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}