A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model

A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model Hao Chen, Zhe Xu, Ziyi LIU, Junlin HOU, Ma Jiabo, Cheng Jin, Yihui Wang, and 9 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7578325/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Pathology serves as the cornerstone of modern medicine. The integration of multimodal data, particularly the combination of pathological images and textual context, is increasingly essential for accurate and efficient diagnosis. Multimodal large language models (MLLMs) hold promise for the development of interactive assistant systems capable of supporting pathologists and enhancing diagnostic workflows. However, current MLLM approaches in pathology demonstrate significantly constrained reasoning capabilities, primarily due to their reliance on expensive chain-of-thought annotations. Additionally, existing methods remain limited to simplex application of visual question answering (VQA) at the region-of-interest (ROI) level, failing to address the full spectrum of diagnostic needs such as ROI classification, detection, segmentation, whole-slide-image (WSI) classification and VQA in clinical practice. In this study, we present SmartPath-R1, a versatile MLLM capable of simultaneously addressing both ROI-level and WSI-level tasks while demonstrating robust pathological reasoning capability. Our framework combines scale-dependent supervised fine-tuning and task-aware reinforcement fine-tuning, which circumvents the requirement for chain-of-thought supervision by leveraging the intrinsic knowledge within MLLM. Furthermore, SmartPath-R1 integrates multiscale and multitask analysis through a mixture-of-experts mechanism, enabling dynamic processing for diverse tasks. We curate a large-scale dataset comprising 2.3M ROI and 188K WSI instruction samples for training and evaluation. Extensive experiments across 72 tasks validate the effectiveness and superiority of the proposed approach. This work represents a significant step toward developing versatile, reasoning-enhanced AI systems for precision pathology. Biological sciences/Computational biology and bioinformatics/Software Health sciences/Health care/Diagnosis Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7578325","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":522145697,"identity":"297d931a-adbe-461b-b068-c912b3453cce","order_by":0,"name":"Hao Chen","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAsUlEQVRIiWNgGAWjYBACxhkMjA9gHGaGA8RpYTaAaSBOC4MEA5sEaVqYZzc/q/jxx4bB4Pz5A8wFZ4hx2JxjZjd729IYDG4kMzDPuEGMlhk5bLcZGw4DtQAdxvOBSC3FDH/+Ax12mAQtzAxsBxgMDgAdxkOcw9KMJXvbknkkbyQbHOYhxvuGM5Iffvjxx06O7/zBh495jhGjpQFC84CIA0RoYGCQJ0rVKBgFo2AUjGwAAITwNJZ5nBIOAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-8400-3780","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Hao","middleName":"","lastName":"Chen","suffix":""},{"id":522145698,"identity":"40f90455-c424-4ae6-bb20-6384f5bb63ba","order_by":1,"name":"Zhe Xu","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Zhe","middleName":"","lastName":"Xu","suffix":""},{"id":522145699,"identity":"3854366b-e0d7-496a-8d59-a6af51293d6c","order_by":2,"name":"Ziyi LIU","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Ziyi","middleName":"","lastName":"LIU","suffix":""},{"id":522145700,"identity":"f4a90a5b-f7a9-4c61-a3ee-3e76a7c29379","order_by":3,"name":"Junlin HOU","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Junlin","middleName":"","lastName":"HOU","suffix":""},{"id":522145701,"identity":"0bbd81be-87af-40b9-8c46-d982be956941","order_by":4,"name":"Ma Jiabo","email":"","orcid":"https://orcid.org/0000-0001-8532-4466","institution":"Hong Kong University","correspondingAuthor":false,"prefix":"","firstName":"Ma","middleName":"","lastName":"Jiabo","suffix":""},{"id":522145702,"identity":"e4cc604c-ddc6-46f1-99c0-dc1604a64522","order_by":5,"name":"Cheng Jin","email":"","orcid":"https://orcid.org/0000-0002-3522-3592","institution":"Department of Computer Science and Engineering, The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Cheng","middleName":"","lastName":"Jin","suffix":""},{"id":522145703,"identity":"8b0c40ed-55f5-4e99-ad5e-da0fddaa96d8","order_by":6,"name":"Yihui Wang","email":"","orcid":"https://orcid.org/0009-0002-7606-9816","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yihui","middleName":"","lastName":"Wang","suffix":""},{"id":522145704,"identity":"ba39d5fb-8690-4e2f-8fed-6883861581c3","order_by":7,"name":"Zhixuan CHEN","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Zhixuan","middleName":"","lastName":"CHEN","suffix":""},{"id":522145705,"identity":"8219415d-d449-4219-8986-d1d96df38bde","order_by":8,"name":"Zhengyu ZHANG","email":"","orcid":"","institution":"Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Zhengyu","middleName":"","lastName":"ZHANG","suffix":""},{"id":522145706,"identity":"7025067f-618e-4dd4-a371-7befe8ce1958","order_by":9,"name":"Fuxiang HUANG","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Fuxiang","middleName":"","lastName":"HUANG","suffix":""},{"id":522145707,"identity":"8c90a755-4039-4fc4-8d16-31d7b061ca63","order_by":10,"name":"Zhengrui GUO","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Zhengrui","middleName":"","lastName":"GUO","suffix":""},{"id":522145708,"identity":"a21758fa-237e-4a56-8ab9-8ad94b0c165f","order_by":11,"name":"Fengtao ZHOU","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Fengtao","middleName":"","lastName":"ZHOU","suffix":""},{"id":522145709,"identity":"868fb33f-3f13-49d7-b753-2a73ce6b454c","order_by":12,"name":"Yingxue XU","email":"","orcid":"https://orcid.org/0000-0002-9657-3107","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yingxue","middleName":"","lastName":"XU","suffix":""},{"id":522145710,"identity":"09b1c52a-40b6-4428-b804-8208523c43cc","order_by":13,"name":"Xi WANG","email":"","orcid":"","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Xi","middleName":"","lastName":"WANG","suffix":""},{"id":522145711,"identity":"3020476c-9a65-4615-ba69-972a2e214ed8","order_by":14,"name":"Ronald Chan","email":"","orcid":"","institution":"The Chinese University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Ronald","middleName":"","lastName":"Chan","suffix":""},{"id":522145712,"identity":"a9d02aab-d447-4a60-861b-fab0e206ca36","order_by":15,"name":"Li Liang","email":"","orcid":"https://orcid.org/0000-0001-5302-2754","institution":"Department of Pathology, School of Basic Medical Sciences, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Li","middleName":"","lastName":"Liang","suffix":""}],"badges":[],"createdAt":"2025-09-10 03:35:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7578325/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7578325/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97897882,"identity":"cde3b51c-1ded-4650-831c-0f3573c20957","added_by":"auto","created_at":"2025-12-10 15:38:25","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":14085950,"visible":true,"origin":"","legend":"Article File","description":"","filename":"SmartPathR1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7578325/v1_covered_644f0523-4b67-460d-b435-33b67738320e.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7578325/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7578325/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Pathology serves as the cornerstone of modern medicine. The integration of multimodal data, particularly the combination of pathological images and textual context, is increasingly essential for accurate and efficient diagnosis. Multimodal large language models (MLLMs) hold promise for the development of interactive assistant systems capable of supporting pathologists and enhancing diagnostic workflows. However, current MLLM approaches in pathology demonstrate significantly constrained reasoning capabilities, primarily due to their reliance on expensive chain-of-thought annotations. Additionally, existing methods remain limited to simplex application of visual question answering (VQA) at the region-of-interest (ROI) level, failing to address the full spectrum of diagnostic needs such as ROI classification, detection, segmentation, whole-slide-image (WSI) classification and VQA in clinical practice. In this study, we present SmartPath-R1, a versatile MLLM capable of simultaneously addressing both ROI-level and WSI-level tasks while demonstrating robust pathological reasoning capability. Our framework combines scale-dependent supervised fine-tuning and task-aware reinforcement fine-tuning, which circumvents the requirement for chain-of-thought supervision by leveraging the intrinsic knowledge within MLLM. Furthermore, SmartPath-R1 integrates multiscale and multitask analysis through a mixture-of-experts mechanism, enabling dynamic processing for diverse tasks. We curate a large-scale dataset comprising 2.3M ROI and 188K WSI instruction samples for training and evaluation. Extensive experiments across 72 tasks validate the effectiveness and superiority of the proposed approach. This work represents a significant step toward developing versatile, reasoning-enhanced AI systems for precision pathology.","manuscriptTitle":"A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-01 09:08:33","doi":"10.21203/rs.3.rs-7578325/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0990d2af-72ef-4eec-92f5-2e9c408f0b90","owner":[],"postedDate":"October 1st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":55494833,"name":"Biological sciences/Computational biology and bioinformatics/Software"},{"id":55494834,"name":"Health sciences/Health care/Diagnosis"}],"tags":[],"updatedAt":"2025-12-11T16:18:33+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-01 09:08:33","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7578325","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7578325","identity":"rs-7578325","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}