An agentic multimodal AI framework for end-to-end breast cancer staging and biomarker profiling

An agentic multimodal AI framework for end-to-end breast cancer staging and biomarker profiling | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article An agentic multimodal AI framework for end-to-end breast cancer staging and biomarker profiling Yang Liu, Shaohua Chen, Guiyun Zhang, Yue Wu, Ruirui Cao, Shanshan Zhang, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8661837/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Breast cancer care spans screening, diagnosis and molecular stratification, yet most AI systems remain siloed into single tasks and single modalities. A unified, agentic system that can invoke specialized models across modalities could streamline end-to-end decision support. Methods We curated a cohort of 923 patients with paired radiology and pathology data and trained five CNN backbones (ResNet, DenseNet, EfficientNet, RegNet and MobileNetV3) for nine clinically relevant tasks, including T/N/M and clinical staging, histological grade, ER/PR/HER2 status and Ki-67 expression. We further implemented a late-fusion transformer and an orchestration layer that selects the appropriate single-modality or fused model per query, and evaluated performance using AUROC, accuracy and F1-score. Results Across tasks, the best single-modality models achieved AUROCs from 0.606 to 0.990, with strong performance for histological grade (AUROC 0.950; accuracy 0.909) and HER2 status (AUROC 0.810; accuracy 0.762). Multimodal fusion consistently improved discrimination over the best single modality (mean ΔAUROC 0.016), reaching AUROC 0.964 and accuracy 0.924 for grade, AUROC 0.831 for HER2 and AUROC 0.806 for N staging. Conclusion Together, these results show that agent-guided multimodal modelling can deliver robust, task-adaptive predictions spanning staging and biomarker profiling within a single framework. By enabling modular deployment—from screening-time risk triage to pathology-informed stratification—this approach provides a practical foundation for scalable, clinically integrated breast cancer decision support. Health sciences/Biomarkers Biological sciences/Cancer Biological sciences/Computational biology and bioinformatics Health sciences/Oncology Agentic multimodal AI Breast cancer staging Biomarker profiling Radiology–pathology fusion Deployment-efficient deep learning Full Text Additional Declarations No competing interests reported. Supplementary Files supplementarytables.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8661837","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":580889052,"identity":"42723e21-4bbd-4b28-892f-fd7f20de100d","order_by":0,"name":"Yang Liu","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Liu","suffix":""},{"id":580889053,"identity":"2fffa436-0b30-4d86-82d6-fe11fc8348c0","order_by":1,"name":"Shaohua Chen","email":"","orcid":"","institution":"Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Shaohua","middleName":"","lastName":"Chen","suffix":""},{"id":580889054,"identity":"deb6b275-4234-4199-b86c-d4e949c491fb","order_by":2,"name":"Guiyun Zhang","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Guiyun","middleName":"","lastName":"Zhang","suffix":""},{"id":580889055,"identity":"3f878fff-e7f7-4c58-ac04-1fd7a875f73f","order_by":3,"name":"Yue Wu","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Wu","suffix":""},{"id":580889056,"identity":"7553782d-867d-43b9-b478-ea5b6161524a","order_by":4,"name":"Ruirui Cao","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Ruirui","middleName":"","lastName":"Cao","suffix":""},{"id":580889057,"identity":"888bed58-830c-4786-91f7-3a6bc9a8e90a","order_by":5,"name":"Shanshan Zhang","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Shanshan","middleName":"","lastName":"Zhang","suffix":""},{"id":580889058,"identity":"3b77d5bd-1016-4252-a757-8b081aec8615","order_by":6,"name":"Yufo Chen","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yufo","middleName":"","lastName":"Chen","suffix":""},{"id":580889059,"identity":"f699168f-c4d4-4622-b26a-5916634d118d","order_by":7,"name":"Rui Wang","email":"","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":false,"prefix":"","firstName":"Rui","middleName":"","lastName":"Wang","suffix":""},{"id":580889060,"identity":"7957c860-95e6-495e-acd1-61291f327dc0","order_by":8,"name":"Erkang Tian","email":"","orcid":"","institution":"West China Stomatological Hospital of Sichuan University Medical University","correspondingAuthor":false,"prefix":"","firstName":"Erkang","middleName":"","lastName":"Tian","suffix":""},{"id":580889061,"identity":"8f2a0ee4-1a30-4f31-8f3d-55b3ecfe552a","order_by":9,"name":"Yumei Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIiWNgGAWjYBACPmYwJcHDz95+4EBChYQcPyEtbFAtMpI9ZxIPPDhjYSzZQEgLlLYxuJFgfPBhW0XiBoJa2HnMpG5UWPAYnDmQcCBxngTjBgbmh49u4HUYj5l0zhkJHsnjjQcOJG6TYDZnYDM2ziGkJbdNgocPbMs2CTbLBh42acJa/knwMNxIMDiQOEeCx+AAUVoaJHgEwFoaJCSI0MJWbJ1zDOiXnjMJBxKOSRhINhPwCz//4Y23c2rq7IFRefjjj5q6+n725oeP8WlhYOAwQBNgxqscBNgfEFQyCkbBKBgFIxwAAEb1R7PFVUVnAAAAAElFTkSuQmCC","orcid":"","institution":"The First Affiliated Hospital of Bengbu Medical University","correspondingAuthor":true,"prefix":"","firstName":"Yumei","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2026-01-21 16:08:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8661837/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8661837/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":102261464,"identity":"dcc02957-b25c-4836-8e3b-a0567b7aeabe","added_by":"auto","created_at":"2026-02-10 00:39:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":999122,"visible":true,"origin":"","legend":"","description":"","filename":"revisedmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8661837/v1_covered_31b26455-c2c0-45bc-8807-4d505c6d9af8.pdf"},{"id":101283809,"identity":"6a0b9d1a-f489-4754-b9bb-eaa243b0fe53","added_by":"auto","created_at":"2026-01-28 06:04:33","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":14497,"visible":true,"origin":"","legend":"","description":"","filename":"supplementarytables.docx","url":"https://assets-eu.researchsquare.com/files/rs-8661837/v1/4da527ad860911b472062ead.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"An agentic multimodal AI framework for end-to-end breast cancer staging and biomarker profiling","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Agentic multimodal AI, Breast cancer staging, Biomarker profiling, Radiology–pathology fusion, Deployment-efficient deep learning","lastPublishedDoi":"10.21203/rs.3.rs-8661837/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8661837/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eBreast cancer care spans screening, diagnosis and molecular stratification, yet most AI systems remain siloed into single tasks and single modalities. A unified, agentic system that can invoke specialized models across modalities could streamline end-to-end decision support.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eWe curated a cohort of 923 patients with paired radiology and pathology data and trained five CNN backbones (ResNet, DenseNet, EfficientNet, RegNet and MobileNetV3) for nine clinically relevant tasks, including T/N/M and clinical staging, histological grade, ER/PR/HER2 status and Ki-67 expression. We further implemented a late-fusion transformer and an orchestration layer that selects the appropriate single-modality or fused model per query, and evaluated performance using AUROC, accuracy and F1-score.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eAcross tasks, the best single-modality models achieved AUROCs from 0.606 to 0.990, with strong performance for histological grade (AUROC 0.950; accuracy 0.909) and HER2 status (AUROC 0.810; accuracy 0.762). Multimodal fusion consistently improved discrimination over the best single modality (mean ΔAUROC 0.016), reaching AUROC 0.964 and accuracy 0.924 for grade, AUROC 0.831 for HER2 and AUROC 0.806 for N staging.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eTogether, these results show that agent-guided multimodal modelling can deliver robust, task-adaptive predictions spanning staging and biomarker profiling within a single framework. By enabling modular deployment\u0026mdash;from screening-time risk triage to pathology-informed stratification\u0026mdash;this approach provides a practical foundation for scalable, clinically integrated breast cancer decision support.\u003c/p\u003e","manuscriptTitle":"An agentic multimodal AI framework for end-to-end breast cancer staging and biomarker profiling","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-28 06:04:24","doi":"10.21203/rs.3.rs-8661837/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"07dd29d8-7e2f-4b7b-be93-f96b0eecac1f","owner":[],"postedDate":"January 28th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":61793168,"name":"Health sciences/Biomarkers"},{"id":61793169,"name":"Biological sciences/Cancer"},{"id":61793170,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":61793171,"name":"Health sciences/Oncology"}],"tags":[],"updatedAt":"2026-02-10T00:39:24+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-28 06:04:24","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8661837","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8661837","identity":"rs-8661837","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}