Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks

Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks Wenya Xie, Qingying Xiao, Yu Zheng, Xidong Wang, Junying Chen, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6763537/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 13 You are reading this latest preprint version Abstract The rise of large language models (LLMs) has profoundly influenced health-care by offering medical advice, diagnostic suggestions, and more. However, their deployment directly toward patients poses substantial risks, as limited domain knowledge may result in misleading or erroneous outputs. To address this challenge , we propose repositioning LLMs as clinical assistants that collaborate with experienced physicians rather than interacting with patients directly. We begin with a two-stage inspiration–feedback survey to identify real-world needs in clinical workflows. Guided by this, we construct DoctorFLAN, a large-scale Chi-nese medical dataset comprising 92,000 Q&A instances across 22 clinical tasks and 27 specialties. To evaluate model performance in doctor-facing applications, 1 we introduce DoctorFLAN-test (550 single-turn Q&A items) and DotaBench (74 multi-turn conversations mimicking realistic scenarios). Experimental results with over ten popular LLMs demonstrate that DoctorFLAN notably improves the performance of open-source LLMs in medical contexts, facilitating their alignment with physician workflows and complementing existing patient-oriented models. This work contributes a valuable resource and framework for advancing doctor-centered medical LLM development. Biological sciences/Computational biology and bioinformatics/Data integration Biological sciences/Computational biology and bioinformatics/Data mining Large Language Models Healthcare AI Medical Assistants Clinical Workflow Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 24 Jun, 2025 Reviews received at journal 24 Jun, 2025 Reviews received at journal 23 Jun, 2025 Reviewers agreed at journal 23 Jun, 2025 Reviews received at journal 17 Jun, 2025 Reviewers agreed at journal 03 Jun, 2025 Reviewers agreed at journal 31 May, 2025 Reviewers agreed at journal 30 May, 2025 Reviewers agreed at journal 29 May, 2025 Reviewers invited by journal 29 May, 2025 Editor assigned by journal 29 May, 2025 Submission checks completed at journal 28 May, 2025 First submitted to journal 27 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6763537","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":464624259,"identity":"2a58b95a-d1bb-498f-9460-4e8d0310c242","order_by":0,"name":"Wenya Xie","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Wenya","middleName":"","lastName":"Xie","suffix":""},{"id":464624260,"identity":"f4eeaea5-f248-422c-a74c-fd44c8a44b82","order_by":1,"name":"Qingying Xiao","email":"","orcid":"","institution":"Shenzhen Research Institute of Big Data","correspondingAuthor":false,"prefix":"","firstName":"Qingying","middleName":"","lastName":"Xiao","suffix":""},{"id":464624261,"identity":"b2f66093-f8c7-4023-a60a-a0a7754eb6fd","order_by":2,"name":"Yu Zheng","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Zheng","suffix":""},{"id":464624262,"identity":"94a6cdfe-19e9-4655-b970-d02120235739","order_by":3,"name":"Xidong Wang","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Xidong","middleName":"","lastName":"Wang","suffix":""},{"id":464624263,"identity":"5e792498-8b3f-4aa5-aa7d-830e652ae27c","order_by":4,"name":"Junying Chen","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Junying","middleName":"","lastName":"Chen","suffix":""},{"id":464624264,"identity":"607e3bb1-03ac-4948-ae81-bb34869cfd7d","order_by":5,"name":"Ke Ji","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Ke","middleName":"","lastName":"Ji","suffix":""},{"id":464624265,"identity":"7362994a-ddd1-4375-a087-01730407439f","order_by":6,"name":"Anningzhe Gao","email":"","orcid":"","institution":"Shenzhen Research Institute of Big Data","correspondingAuthor":false,"prefix":"","firstName":"Anningzhe","middleName":"","lastName":"Gao","suffix":""},{"id":464624267,"identity":"f0fe5705-b43a-4c2e-8ec5-6bb7dd693e07","order_by":7,"name":"Prayag Tiwari","email":"","orcid":"","institution":"Halmstad University","correspondingAuthor":false,"prefix":"","firstName":"Prayag","middleName":"","lastName":"Tiwari","suffix":""},{"id":464624268,"identity":"20abcddb-96df-4556-b408-9a4a3b536d0e","order_by":8,"name":"Xiang Wan","email":"","orcid":"","institution":"Shenzhen Research Institute of Big Data","correspondingAuthor":false,"prefix":"","firstName":"Xiang","middleName":"","lastName":"Wan","suffix":""},{"id":464624270,"identity":"5d7a53c3-3438-4ae4-b3b8-d0f71a2d4307","order_by":9,"name":"Feng Jiang","email":"","orcid":"","institution":"Shenzhen Research Institute of Big Data","correspondingAuthor":false,"prefix":"","firstName":"Feng","middleName":"","lastName":"Jiang","suffix":""},{"id":464624271,"identity":"98538d55-1c77-4cd8-b9ef-37f11aea6cc5","order_by":10,"name":"Benyou Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8ElEQVRIiWNgGAWjYJACZih1gIGxIQHISCBaC1sCyVp4DIjTYnD87OHXBRV3Evslcj6/5t2RxsDPnmPA8HMHHi1n8tKsZ5x5ljhzRu42a94zOQySPW8MGHvP4NZidiDHzJi37XDihhu524CMCgaDGzkGzIxteLScfwPU8g+kJecZWIs9QS03cowf8zaAtTA/5m3LYTCQIKDF/sYbM+YZxw4bz+x5ZsY4ty2NR+LMs4KDvXi0SPbnGH8uqDks28+e/PjD27ZkOf725I0PfuLRAgRsEkDCsQHK4AEJHcCrARiTH0AOhDFGwSgYBaNgFGAAAKaBWB5giZv1AAAAAElFTkSuQmCC","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":true,"prefix":"","firstName":"Benyou","middleName":"","lastName":"Wang","suffix":""}],"badges":[],"createdAt":"2025-05-28 02:53:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6763537/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6763537/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":83742844,"identity":"1a2aea7f-c978-4c09-9674-b427727184b4","added_by":"auto","created_at":"2025-06-02 02:03:18","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":934389,"visible":true,"origin":"","legend":"","description":"","filename":"npjEnablingDoctorCentricMedicalAIwithLLMsthroughWorkflowAlignedTasksandBenchmarks.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6763537/v1_covered_25c9c9e6-f4c2-4a29-8851-858ade12aa23.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-health-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)","snPcode":"44401","submissionUrl":"https://submission.springernature.com/new-submission/44401/3","title":"npj Health Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Large Language Models, Healthcare AI, Medical Assistants, Clinical Workflow","lastPublishedDoi":"10.21203/rs.3.rs-6763537/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6763537/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The rise of large language models (LLMs) has profoundly influenced health-care by offering medical advice, diagnostic suggestions, and more. However, their deployment directly toward patients poses substantial risks, as limited domain knowledge may result in misleading or erroneous outputs. To address this challenge , we propose repositioning LLMs as clinical assistants that collaborate with experienced physicians rather than interacting with patients directly. We begin with a two-stage inspiration–feedback survey to identify real-world needs in clinical workflows. Guided by this, we construct DoctorFLAN, a large-scale Chi-nese medical dataset comprising 92,000 Q\u0026A instances across 22 clinical tasks and 27 specialties. To evaluate model performance in doctor-facing applications, 1 we introduce DoctorFLAN-test (550 single-turn Q\u0026A items) and DotaBench (74 multi-turn conversations mimicking realistic scenarios). Experimental results with over ten popular LLMs demonstrate that DoctorFLAN notably improves the performance of open-source LLMs in medical contexts, facilitating their alignment with physician workflows and complementing existing patient-oriented models. This work contributes a valuable resource and framework for advancing doctor-centered medical LLM development.","manuscriptTitle":"Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-02 01:55:12","doi":"10.21203/rs.3.rs-6763537/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-06-24T22:36:48+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-24T22:34:22+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-24T01:55:52+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"111780100576070012238413825806638855435","date":"2025-06-24T00:06:00+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-17T12:10:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"56659618095500670632560855220490095045","date":"2025-06-03T19:47:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"218935748481564782952588972212329197740","date":"2025-05-31T17:46:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"320316772992713655443043229162844576906","date":"2025-05-31T00:27:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"190257282017125010923794066035643415776","date":"2025-05-29T20:15:28+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-05-29T10:56:22+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-29T10:50:57+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-28T10:34:22+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Health Systems","date":"2025-05-28T02:41:13+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-health-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)","snPcode":"44401","submissionUrl":"https://submission.springernature.com/new-submission/44401/3","title":"npj Health Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"1b469e81-a6d2-4cb1-a1dc-2f0415938ae2","owner":[],"postedDate":"June 2nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":49332709,"name":"Biological sciences/Computational biology and bioinformatics/Data integration"},{"id":49332710,"name":"Biological sciences/Computational biology and bioinformatics/Data mining"}],"tags":[],"updatedAt":"2025-08-13T15:53:34+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-02 01:55:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6763537","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6763537","identity":"rs-6763537","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}