QoQ-Med3: Robust Multimodal Clinical Analysis Foundation Model with Reasoning

QoQ-Med3: Robust Multimodal Clinical Analysis Foundation Model with Reasoning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article QoQ-Med3: Robust Multimodal Clinical Analysis Foundation Model with Reasoning David Dai, Jeannie She, Jiaee Cheong, Xing Han, Carl Harris, Haowen Wei, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8391230/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 20 You are reading this latest preprint version Abstract Multimodal reasoning–based foundation models (MRFMs) hold considerable promise for addressing key challenges in medicalpractice, yet their readiness for real-world deployment remains insufficiently explored. To bridge this gap, we developedtwo MRFMs (QoQ-Med3 and QoQ-Med3-MIMIC) and systematically evaluated their (i) generalizability to previously unseenclinical modalities and tasks, (ii) transferability to held-out datasets collected across different clinical sites, and (iii) robustnessto real-world challenges like cross-site heterogeneity. Our results demonstrate that these models can learn transferablerepresentations across modalities, tasks, and heterogeneous clinical datasets. QoQ-Med3 achieves an overall balancedaccuracy of 71.3% and a F1 of 0.349, superceding all open source and closed source models including GPT-4o, with particularly pronounced gains in understudied modalities such as ultrasound and mammography. The model trained on public clinical dataonly generalized to the both the held-out MIMIC-IV and the private JHU PMAP dataset collected at Johns Hopkins Universityhospital. In addition, the extrinsic hallucination rates are reduced by 78.2 percent after training. Collectively, our findingshighlight both the potential of multimodal reasoning-based clinical foundation models and the critical next steps required tomake them robust and reliable for real-world deployment. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Full Text Additional Declarations No competing interests reported. Supplementary Files SupplementnpjQoQMed3RobustMultimodalDiagnosisFoundationModelwithReasoning.pdf Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 11 Mar, 2026 Reviews received at journal 06 Mar, 2026 Reviews received at journal 25 Feb, 2026 Reviews received at journal 09 Feb, 2026 Reviewers agreed at journal 04 Feb, 2026 Reviews received at journal 28 Jan, 2026 Reviews received at journal 28 Jan, 2026 Reviews received at journal 28 Jan, 2026 Reviewers agreed at journal 27 Jan, 2026 Reviews received at journal 27 Jan, 2026 Reviewers agreed at journal 27 Jan, 2026 Reviewers agreed at journal 27 Jan, 2026 Reviewers agreed at journal 26 Jan, 2026 Reviewers agreed at journal 26 Jan, 2026 Reviewers agreed at journal 27 Dec, 2025 Reviewers agreed at journal 26 Dec, 2025 Reviewers invited by journal 26 Dec, 2025 Editor assigned by journal 22 Dec, 2025 Submission checks completed at journal 22 Dec, 2025 First submitted to journal 18 Dec, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8391230","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":566489296,"identity":"a00a438c-ada0-48cf-88dc-3ffcb4ab86ce","order_by":0,"name":"David Dai","email":"","orcid":"","institution":"Massachusetts Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Dai","suffix":""},{"id":566489297,"identity":"b7772b2a-1fed-443d-9c41-510771195cc9","order_by":1,"name":"Jeannie She","email":"","orcid":"","institution":"Massachusetts Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Jeannie","middleName":"","lastName":"She","suffix":""},{"id":566489298,"identity":"a5309394-c02f-4ab7-b176-4ad1d89422d5","order_by":2,"name":"Jiaee Cheong","email":"","orcid":"","institution":"Harvard University","correspondingAuthor":false,"prefix":"","firstName":"Jiaee","middleName":"","lastName":"Cheong","suffix":""},{"id":566489299,"identity":"0faab86f-fa0e-47ac-8ecc-5a4d1a1438af","order_by":3,"name":"Xing Han","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"prefix":"","firstName":"Xing","middleName":"","lastName":"Han","suffix":""},{"id":566489300,"identity":"48cca0ef-f3e4-489b-a3e5-1b576fbb20a7","order_by":4,"name":"Carl Harris","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"prefix":"","firstName":"Carl","middleName":"","lastName":"Harris","suffix":""},{"id":566489301,"identity":"7661d4db-a5b3-422d-873f-361395c0bb6a","order_by":5,"name":"Haowen Wei","email":"","orcid":"","institution":"Massachusetts Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Haowen","middleName":"","lastName":"Wei","suffix":""},{"id":566489304,"identity":"3575a39f-7363-46a2-ace0-e364c6c91e4e","order_by":6,"name":"Farzan Vahedifard","email":"","orcid":"","institution":"Harvard University","correspondingAuthor":false,"prefix":"","firstName":"Farzan","middleName":"","lastName":"Vahedifard","suffix":""},{"id":566489312,"identity":"89ed854f-537f-4a35-8046-2ac09e9e6428","order_by":7,"name":"Suchi Saria","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"prefix":"","firstName":"Suchi","middleName":"","lastName":"Saria","suffix":""},{"id":566489313,"identity":"bc44eef9-0ff3-41b7-b624-100cde6e27e8","order_by":8,"name":"Robert Stevens","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"prefix":"","firstName":"Robert","middleName":"","lastName":"Stevens","suffix":""},{"id":566489314,"identity":"eb286f2c-099b-4000-a6a2-0c74563500d2","order_by":9,"name":"Paul Liang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtklEQVRIiWNgGAWjYFAC5oYDQFKOgRnEYSNKCyNYizEDMzMJWkBkYgMDsVrMpRsbD/zcUZu+tp3/AMOHssOEtVjOOdhwsPfM8dxth5kZGGecI0KLwY3EhgO8bcfAWph524jUcvBv27F0M5CWv8RqOczbVpMA1sJIjBaQXw7Lth0wBDrM4GDPuXTCWsylmw9/fNtWJ292/uDDBz/KrIlwmASYgrjnAGH1CC11RCkeBaNgFIyCEQoAq6ZBcZUrOUIAAAAASUVORK5CYII=","orcid":"","institution":"Massachusetts Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"Paul","middleName":"","lastName":"Liang","suffix":""}],"badges":[],"createdAt":"2025-12-18 05:23:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8391230/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8391230/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":99205231,"identity":"30360a5f-d3df-47b4-bbf8-7bf182cddda0","added_by":"auto","created_at":"2025-12-30 06:19:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10255375,"visible":true,"origin":"","legend":"","description":"","filename":"npjQoQMed3RobustMultimodalDiagnosisFoundationModelwithReasoningnosup.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8391230/v1/fd2ccf89291f19998ccb083f.pdf"},{"id":99205232,"identity":"a307211b-4345-4f45-9ecd-02b9765e4e15","added_by":"auto","created_at":"2025-12-30 06:19:43","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10765,"visible":true,"origin":"","legend":"","description":"","filename":"9983690b35d3425c9da596dcdc581a56.json","url":"https://assets-eu.researchsquare.com/files/rs-8391230/v1/950d700efffe8ab9aea7ef49.json"},{"id":99205230,"identity":"a22c1768-2a2d-4fe1-aa93-7690ce56c11f","added_by":"auto","created_at":"2025-12-30 06:19:43","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8659702,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementnpjQoQMed3RobustMultimodalDiagnosisFoundationModelwithReasoning.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8391230/v1/ac7075d42312b3175a9c3856.pdf"},{"id":99317849,"identity":"05b725bb-12b3-4703-941a-851aa57bf24d","added_by":"auto","created_at":"2025-12-31 16:30:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":6083844,"visible":true,"origin":"","legend":"","description":"","filename":"npjQoQMed3RobustMultimodalDiagnosisFoundationModelwithReasoningnosup.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8391230/v1_covered_270732ce-02c7-4cc1-b9e5-cd2466a4ebed.pdf"},{"id":99205233,"identity":"c4a9f7f6-7385-45ae-b5e7-11cb63f56fd9","added_by":"auto","created_at":"2025-12-30 06:19:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":8659702,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementnpjQoQMed3RobustMultimodalDiagnosisFoundationModelwithReasoning.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8391230/v1/9256703ff158af3477afc33e.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"QoQ-Med3: Robust Multimodal Clinical Analysis Foundation Model with Reasoning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8391230/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8391230/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Multimodal reasoning–based foundation models (MRFMs) hold considerable promise for addressing key challenges in medicalpractice, yet their readiness for real-world deployment remains insufficiently explored. To bridge this gap, we developedtwo MRFMs (QoQ-Med3 and QoQ-Med3-MIMIC) and systematically evaluated their (i) generalizability to previously unseenclinical modalities and tasks, (ii) transferability to held-out datasets collected across different clinical sites, and (iii) robustnessto real-world challenges like cross-site heterogeneity. Our results demonstrate that these models can learn transferablerepresentations across modalities, tasks, and heterogeneous clinical datasets. QoQ-Med3 achieves an overall balancedaccuracy of 71.3% and a F1 of 0.349, superceding all open source and closed source models including GPT-4o, with particularly pronounced gains in understudied modalities such as ultrasound and mammography. The model trained on public clinical dataonly generalized to the both the held-out MIMIC-IV and the private JHU PMAP dataset collected at Johns Hopkins Universityhospital. In addition, the extrinsic hallucination rates are reduced by 78.2 percent after training. Collectively, our findingshighlight both the potential of multimodal reasoning-based clinical foundation models and the critical next steps required tomake them robust and reliable for real-world deployment.","manuscriptTitle":"QoQ-Med3: Robust Multimodal Clinical Analysis Foundation Model with Reasoning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-30 06:19:38","doi":"10.21203/rs.3.rs-8391230/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-03-11T23:23:37+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-06T22:03:14+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-02-26T01:51:53+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-02-09T10:18:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"286043745718161249205449890659531372759","date":"2026-02-04T19:42:10+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-28T19:33:42+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-28T17:27:46+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-28T15:04:07+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"279823821037624811700490575262180000207","date":"2026-01-27T17:23:05+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-27T08:03:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"152545428902422281903391715542436204648","date":"2026-01-27T07:50:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"99646746188972688093091380576510297071","date":"2026-01-27T06:11:43+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"304583681037650665842476258624313508671","date":"2026-01-27T04:24:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"306190312305546549423027393300390947540","date":"2026-01-27T04:23:07+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"280230880636075000459009095764431205440","date":"2025-12-28T03:48:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"303660308627683935830306921359377536435","date":"2025-12-26T18:15:32+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-26T07:33:36+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-12-23T01:52:06+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-12-22T09:32:02+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2025-12-18T05:14:57+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"916ef907-4125-403b-9f43-fb63676f8a06","owner":[],"postedDate":"December 30th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":60298274,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":60298275,"name":"Health sciences/Health care"},{"id":60298276,"name":"Physical sciences/Mathematics and computing"},{"id":60298277,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-05-15T18:23:32+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-30 06:19:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8391230","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8391230","identity":"rs-8391230","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}