A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies

A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies Panagiotis Kaliosis, Adithya V. Ganesan, Oscar N.E. Kjell, Whitney Ringwald, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8376581/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 11 You are reading this latest preprint version Abstract Large language models (LLMs) are increasingly being used in a zero-shot fashion to assess mental health conditions, yet we have limited knowledge on what factors affect their accuracy. In this study, we utilize a clinical dataset of natural language narratives and self-reported PTSD severity scores from 1,437 individuals to comprehensively evaluate the performance of 11 state-of-the-art LLMs. To understand the factors affecting accuracy, we systematically varied (i) contextual knowledge like subscale definitions, distribution summary, and interview questions, and (ii) modeling strategies including zero-shot vs few shot, amount of reasoning effort, model sizes, structured subscales vs direct scalar prediction, output rescaling and nine ensemble methods. Our findings indicate that (a) LLMs are most accurate when provided with detailed construct definitions and context of the narrative; (b) increased reasoning effort leads to better estimation accuracy; (c) performance of open-weight models (Llama, Deepseek), plateau beyond 70B parameters while closed-weight (o3-mini, gpt-5) models improve with newer generations; and (d) best performance is achieved when ensembling a supervised model with the zero-shot LLMs. Taken together, the results suggest choice of contextual knowledge and modeling strategies is important for deploying LLMs to accurately assess mental health. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Post Traumatic Stress Disorder Large Language Models Prompting Strategies Zero-shot Inference Full Text Additional Declarations Competing interest reported. ONEK co-founded and holds shares in a start-up that uses language-based assessments to diagnose mental health problems. The authors report no additional biomedical financial interests or potential conflicts of interest. Supplementary Files SupplementaryASystematicEvaluationofLargeLanguageModelsforPTSDSeverityEstimationTheRoleofContextualKnowledgeandModelingStrategies.pdf Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 17 Mar, 2026 Reviews received at journal 12 Mar, 2026 Reviewers agreed at journal 08 Mar, 2026 Reviewers agreed at journal 06 Mar, 2026 Reviews received at journal 28 Jan, 2026 Reviewers agreed at journal 18 Jan, 2026 Reviewers agreed at journal 01 Jan, 2026 Reviewers invited by journal 24 Dec, 2025 Editor assigned by journal 22 Dec, 2025 Submission checks completed at journal 22 Dec, 2025 First submitted to journal 16 Dec, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8376581","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":565152042,"identity":"8403c8fb-4dd4-4f76-afdc-8b9c1fbac741","order_by":0,"name":"Panagiotis Kaliosis","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFUlEQVRIie2RvWrDMBCAzxjiRcUdVVrsVzgwNGTKq8gY3EUpGT2ZTM7i7vZbpG8gMLSLSlZDlxRDp1I0dvBQ/5T+gEwzFqoP9HNCH3cnARgMfxEClmD94my+jjrYx9AqMCpE9CEeqQwbyo5U5qQ6iEOSXvhl0yhoW29OV3dKJTG4DkedsrjJUDBZEXyML6mVYbAorqOykBzO8hetgntAEWaC4DmfgbXBcFfzwD7JEsBanwX3juqUlPiltLvCvinLKeUh77PYBGoCFGafCgekE73kcj32IrtewiwIUL5GViFjQuXzWv9iV7dPb0m69LdVo1TreXi/qkAlkeduo522sB8R66fTYR5/53dlwBVTlw0Gg+Gf8g5efWObhXCqkQAAAABJRU5ErkJggg==","orcid":"","institution":"Stony Brook University","correspondingAuthor":true,"prefix":"","firstName":"Panagiotis","middleName":"","lastName":"Kaliosis","suffix":""},{"id":565152044,"identity":"8c6edf1c-6510-4b2e-ba9a-a31e9a5c79b9","order_by":1,"name":"Adithya V. Ganesan","email":"","orcid":"","institution":"Stony Brook University","correspondingAuthor":false,"prefix":"","firstName":"Adithya","middleName":"V.","lastName":"Ganesan","suffix":""},{"id":565152045,"identity":"cb593897-6706-421f-bb43-ab4416e4a694","order_by":2,"name":"Oscar N.E. Kjell","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"prefix":"","firstName":"Oscar","middleName":"N.E.","lastName":"Kjell","suffix":""},{"id":565152046,"identity":"455f2728-5d0d-4d4d-b858-386b6ea36ce4","order_by":3,"name":"Whitney Ringwald","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Whitney","middleName":"","lastName":"Ringwald","suffix":""},{"id":565152047,"identity":"533b412e-3636-412b-a531-aad4116d7fc8","order_by":4,"name":"Scott Feltman","email":"","orcid":"","institution":"Stony Brook University","correspondingAuthor":false,"prefix":"","firstName":"Scott","middleName":"","lastName":"Feltman","suffix":""},{"id":565152048,"identity":"3fd9c796-10f6-4f63-8793-5f92cd98737d","order_by":5,"name":"Melissa A. Carr","email":"","orcid":"","institution":"Stony Brook Medicine","correspondingAuthor":false,"prefix":"","firstName":"Melissa","middleName":"A.","lastName":"Carr","suffix":""},{"id":565152049,"identity":"68a01737-c398-4d03-aeaf-d8bda5e36a98","order_by":6,"name":"Dimitris Samaras","email":"","orcid":"","institution":"Stony Brook University","correspondingAuthor":false,"prefix":"","firstName":"Dimitris","middleName":"","lastName":"Samaras","suffix":""},{"id":565152050,"identity":"eaec445a-90ae-4734-82a7-64c81d63a16a","order_by":7,"name":"Camilo Ruggero","email":"","orcid":"","institution":"The University of Texas at Dallas","correspondingAuthor":false,"prefix":"","firstName":"Camilo","middleName":"","lastName":"Ruggero","suffix":""},{"id":565152051,"identity":"d8bf3547-de08-45e2-83bd-73850322117f","order_by":8,"name":"Benjamin J. Luft","email":"","orcid":"","institution":"Stony Brook Medicine","correspondingAuthor":false,"prefix":"","firstName":"Benjamin","middleName":"J.","lastName":"Luft","suffix":""},{"id":565152052,"identity":"61ae3349-bc35-4f46-90db-1c533b1c25b0","order_by":9,"name":"Roman Kotov","email":"","orcid":"","institution":"Stony Brook Medicine","correspondingAuthor":false,"prefix":"","firstName":"Roman","middleName":"","lastName":"Kotov","suffix":""},{"id":565152053,"identity":"0a5db7a9-e354-4e21-96cf-6c2ebfd6361e","order_by":10,"name":"H. Andrew Schwartz","email":"","orcid":"","institution":"Stony Brook University","correspondingAuthor":false,"prefix":"","firstName":"H.","middleName":"Andrew","lastName":"Schwartz","suffix":""}],"badges":[],"createdAt":"2025-12-16 13:23:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8376581/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8376581/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98990607,"identity":"8e674cd0-9584-47e9-b274-5d679c7b5b30","added_by":"auto","created_at":"2025-12-25 10:14:32","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":12357,"visible":true,"origin":"","legend":"","description":"","filename":"3f248a6be53848ffba7c9f90ad880bd2.json","url":"https://assets-eu.researchsquare.com/files/rs-8376581/v1/2c1751e4458887d3097b14d4.json"},{"id":99313188,"identity":"a5448c25-3a15-488f-bfec-464ace110d10","added_by":"auto","created_at":"2025-12-31 16:19:53","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":508173,"visible":true,"origin":"","legend":"","description":"","filename":"ASystematicEvaluationofLargeLanguageModelsforPTSDSeverityEstimationTheRoleofContextualKnowledgeandModelingStrategies2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8376581/v1_covered_9e70df79-fd78-486c-beba-51c58b1b355b.pdf"},{"id":98990609,"identity":"d3be3291-8e50-47ea-ad82-c01b022c14bd","added_by":"auto","created_at":"2025-12-25 10:14:32","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":286005,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryASystematicEvaluationofLargeLanguageModelsforPTSDSeverityEstimationTheRoleofContextualKnowledgeandModelingStrategies.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8376581/v1/df8517cfcdccd772b5fc3f86.pdf"}],"financialInterests":"Competing interest reported. ONEK co-founded and holds shares in a start-up that uses language-based assessments to diagnose mental health problems. The authors report no additional biomedical financial interests or potential conflicts of interest.","formattedTitle":"A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Post Traumatic Stress Disorder, Large Language Models, Prompting Strategies, Zero-shot Inference","lastPublishedDoi":"10.21203/rs.3.rs-8376581/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8376581/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Large language models (LLMs) are increasingly being used in a zero-shot fashion to assess mental health conditions, yet we have limited knowledge on what factors affect their accuracy. In this study, we utilize a clinical dataset of natural language narratives and self-reported PTSD severity scores from 1,437 individuals to comprehensively evaluate the performance of 11 state-of-the-art LLMs. To understand the factors affecting accuracy, we systematically varied (i) contextual knowledge like subscale definitions, distribution summary, and interview questions, and (ii) modeling strategies including zero-shot vs few shot, amount of reasoning effort, model sizes, structured subscales vs direct scalar prediction, output rescaling and nine ensemble methods. Our findings indicate that (a) LLMs are most accurate when provided with detailed construct definitions and context of the narrative; (b) increased reasoning effort leads to better estimation accuracy; (c) performance of open-weight models (Llama, Deepseek), plateau beyond 70B parameters while closed-weight (o3-mini, gpt-5) models improve with newer generations; and (d) best performance is achieved when ensembling a supervised model with the zero-shot LLMs. Taken together, the results suggest choice of contextual knowledge and modeling strategies is important for deploying LLMs to accurately assess mental health.","manuscriptTitle":"A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-25 10:14:27","doi":"10.21203/rs.3.rs-8376581/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-03-17T06:01:08+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-13T02:46:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"122124958829706783816045036107557226493","date":"2026-03-08T05:13:42+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"336594108632836530176489525207944256490","date":"2026-03-06T09:14:28+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-28T21:32:11+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"207099066928517643740998394142410626690","date":"2026-01-18T21:03:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"233887718128603235879971969183520211199","date":"2026-01-01T17:01:16+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-24T09:22:01+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-12-22T19:29:47+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-12-22T05:29:27+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2025-12-16T13:14:38+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c026e5ac-811b-4aa9-8438-f6c8120d84c1","owner":[],"postedDate":"December 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":60169880,"name":"Physical sciences/Mathematics and computing"},{"id":60169881,"name":"Biological sciences/Psychology"},{"id":60169882,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-03-17T06:11:29+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-25 10:14:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8376581","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8376581","identity":"rs-8376581","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}