Advancing Conversational Diagnostic AI with Multimodal Reasoning

Advancing Conversational Diagnostic AI with Multimodal Reasoning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Biological Sciences - Article Advancing Conversational Diagnostic AI with Multimodal Reasoning Ryutaro Tanno, Khaled Saab, Jan Freyberg, Chunjong Park, Tim Strother, and 31 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6713863/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract AI systems based on Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms can permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in conversation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal medical data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process which emulates that of experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, double-blind study of chat based consultations with 25 patient actors in the style of an Objective Structured Clinical Examination (OSCE). We constructed 105 evaluation scenarios, using common artifacts such as smartphone photos of skin conditions, ECG tracings, and PDFs of clinical documents across a diversity of conditions and demographics. We designed a rubric to assess multiple aspects of the multimodal capability of AMIE and PCPs, in addition to other clinically meaningful axes such as history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. Evaluation by 18 specialists revealed superior performance of AMIE relative to PCPs in handling and reasoning about multimodal data on 7 out of 9 axes while displaying similar superior performance in the non-multimodal metrics, including diagnostic accuracy, on 29 out of 32 axes. These results demonstrate clear progress in multimodal conversational diagnostic AI, although real-world translation necessitates further research. Health sciences/Health care Health sciences/Diseases Full Text Additional Declarations Yes there is potential Competing Interest. Content related to my submission was recently published by this journal Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6713863","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Biological Sciences - Article","associatedPublications":[],"authors":[{"id":461886540,"identity":"17042800-8440-4df7-9294-129218f74450","order_by":0,"name":"Ryutaro Tanno","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAwElEQVRIiWNgGAWjYPACGxDBeIA4xWxgMg1MkqTlMAlazOc3P3tcUHHerp+9/cKBDwx2croNBLTIHGMzN55x5nbyzJ4zBQdnMCQbmxGyS4KNwUyat+12ssGNnITDPAwHErcR1sL+DajlHETLH+K08IBsOWBncCP9ADAQiNKSUybNcyY5QbLnDMPBHgNi/MJ8fJs0T4WdPT97+8MHPyrs5AhqgYHEBgYeAwYGAyKVg4A9AwP7AxLUj4JRMApGwUgCAIsLQFSPpNACAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-8107-6730","institution":"Google DeepMind","correspondingAuthor":true,"prefix":"","firstName":"Ryutaro","middleName":"","lastName":"Tanno","suffix":""},{"id":461886541,"identity":"69de855a-c36c-4971-b936-4e48bba309e9","order_by":1,"name":"Khaled Saab","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Khaled","middleName":"","lastName":"Saab","suffix":""},{"id":461886542,"identity":"1375570e-bbb5-404a-9e72-7e95d183bc7f","order_by":2,"name":"Jan Freyberg","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Jan","middleName":"","lastName":"Freyberg","suffix":""},{"id":461886543,"identity":"6664dc79-3dcf-48b2-8502-20fe12bcb778","order_by":3,"name":"Chunjong Park","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Chunjong","middleName":"","lastName":"Park","suffix":""},{"id":461886544,"identity":"43cb76fc-2aaa-43ae-8f8e-5aab06f368db","order_by":4,"name":"Tim Strother","email":"","orcid":"https://orcid.org/0009-0005-4427-2707","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Tim","middleName":"","lastName":"Strother","suffix":""},{"id":461886545,"identity":"19431e2e-e119-47a4-9cf3-cdee771c2e8a","order_by":5,"name":"Yong Cheng","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Yong","middleName":"","lastName":"Cheng","suffix":""},{"id":461886546,"identity":"21e6be0a-91ce-4ed9-bd78-c6f368fe0eca","order_by":6,"name":"Wei-Hung Weng","email":"","orcid":"https://orcid.org/0000-0003-2232-0390","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Wei-Hung","middleName":"","lastName":"Weng","suffix":""},{"id":461886547,"identity":"3143f99d-5617-4882-9960-4ae575a949e3","order_by":7,"name":"David Barrett","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Barrett","suffix":""},{"id":461886548,"identity":"22410f7c-a5fc-4591-b499-9afea85ec19d","order_by":8,"name":"David Stutz","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Stutz","suffix":""},{"id":461886549,"identity":"d80f7074-599a-4c67-b82b-dfa66b7bde0d","order_by":9,"name":"Nenad Tomasev","email":"","orcid":"https://orcid.org/0000-0003-1624-0220","institution":"DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Nenad","middleName":"","lastName":"Tomasev","suffix":""},{"id":461886550,"identity":"ee47d926-28ac-41c1-a875-b0e77bfe9c82","order_by":10,"name":"Anil Palepu","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Anil","middleName":"","lastName":"Palepu","suffix":""},{"id":461886551,"identity":"16db5c68-f6a4-4d3e-b10f-8c8379096b4c","order_by":11,"name":"Valentin Liévin","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Valentin","middleName":"","lastName":"Liévin","suffix":""},{"id":461886552,"identity":"9272abff-6623-4071-9383-5e21a862f457","order_by":12,"name":"Yash Sharma","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Yash","middleName":"","lastName":"Sharma","suffix":""},{"id":461886553,"identity":"8660d1a6-f8d5-4244-9584-21a0ee8899f6","order_by":13,"name":"Abdullah Ahmed","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Abdullah","middleName":"","lastName":"Ahmed","suffix":""},{"id":461886554,"identity":"577b5cc0-3f54-4aac-837d-030ce30f8b10","order_by":14,"name":"Elahe Vedadi","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Elahe","middleName":"","lastName":"Vedadi","suffix":""},{"id":461886555,"identity":"246325ec-1713-44f8-8c49-5f859312ebc6","order_by":15,"name":"Roma Ruparel","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Roma","middleName":"","lastName":"Ruparel","suffix":""},{"id":461886556,"identity":"13dbbe44-d9d0-451c-b993-c7b7dfab4eb4","order_by":16,"name":"Kimberly Kanada","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Kimberly","middleName":"","lastName":"Kanada","suffix":""},{"id":461886557,"identity":"597abf67-5b3a-4d30-969e-47493f488ae9","order_by":17,"name":"Cian Hughes","email":"","orcid":"https://orcid.org/0000-0001-6901-0985","institution":"Google Health","correspondingAuthor":false,"prefix":"","firstName":"Cian","middleName":"","lastName":"Hughes","suffix":""},{"id":461886558,"identity":"2a97f2c6-c90a-4e3c-9cec-1f19a2f5b731","order_by":18,"name":"Yun Liu","email":"","orcid":"https://orcid.org/0000-0003-4079-8275","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Yun","middleName":"","lastName":"Liu","suffix":""},{"id":461886559,"identity":"ff6a545a-9b18-48e5-a62c-182e7ed191a5","order_by":19,"name":"Geoff Brown","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Geoff","middleName":"","lastName":"Brown","suffix":""},{"id":461886560,"identity":"09f1212f-972b-406d-8b1d-b3b9aed45ade","order_by":20,"name":"Yang Gao","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Gao","suffix":""},{"id":461886561,"identity":"ab6f56e9-c3c3-44c6-a6a8-62c8f45d5adc","order_by":21,"name":"Sean Li","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Sean","middleName":"","lastName":"Li","suffix":""},{"id":461886562,"identity":"01516bb7-e619-42b3-a907-d2547e4c1995","order_by":22,"name":"S. Sara Mahdavi","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"S.","middleName":"Sara","lastName":"Mahdavi","suffix":""},{"id":461886563,"identity":"17980b0c-bb18-4ade-b311-3d6e3845937f","order_by":23,"name":"James Manyika","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"James","middleName":"","lastName":"Manyika","suffix":""},{"id":461886564,"identity":"ad6ec824-78a9-47a7-a071-df9d8d050bdf","order_by":24,"name":"Katherine Chou","email":"","orcid":"","institution":"Google, Inc.","correspondingAuthor":false,"prefix":"","firstName":"Katherine","middleName":"","lastName":"Chou","suffix":""},{"id":461886565,"identity":"f800d76b-012d-4783-98eb-57b972e74b3b","order_by":25,"name":"Yossi Matias","email":"","orcid":"https://orcid.org/0000-0003-3960-6002","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Yossi","middleName":"","lastName":"Matias","suffix":""},{"id":461886566,"identity":"36a2ca44-f945-412b-8e80-74d123630c4d","order_by":26,"name":"Avinatan Hassidim","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Avinatan","middleName":"","lastName":"Hassidim","suffix":""},{"id":461886567,"identity":"21924de9-062b-4cdf-bf35-04ab399039fc","order_by":27,"name":"Dale Webster","email":"","orcid":"https://orcid.org/0000-0002-3023-8824","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Dale","middleName":"","lastName":"Webster","suffix":""},{"id":461886568,"identity":"33d3194f-1b6c-48c2-9138-0ee6e5748f3a","order_by":28,"name":"Pushmeet Kohli","email":"","orcid":"https://orcid.org/0000-0002-7466-7997","institution":"DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Pushmeet","middleName":"","lastName":"Kohli","suffix":""},{"id":461886569,"identity":"2652250f-c9fc-47a9-8d66-d28a9da22002","order_by":29,"name":"Ali Eslami","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Ali","middleName":"","lastName":"Eslami","suffix":""},{"id":461886570,"identity":"b42d5479-fd65-4b24-ace7-623c51c473ec","order_by":30,"name":"Joelle Barral","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Joelle","middleName":"","lastName":"Barral","suffix":""},{"id":461886571,"identity":"97190f34-ea88-43f4-b5a7-09b78d77d4a1","order_by":31,"name":"Adam Rodman","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Adam","middleName":"","lastName":"Rodman","suffix":""},{"id":461886572,"identity":"6e8c02f1-a1c8-4dbf-8217-6009aed74208","order_by":32,"name":"Vivek Natarajan","email":"","orcid":"https://orcid.org/0000-0001-7849-2074","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Vivek","middleName":"","lastName":"Natarajan","suffix":""},{"id":461886573,"identity":"41018c31-6fd5-4246-9f5f-a6a58fe29e63","order_by":33,"name":"Mike Schaekermann","email":"","orcid":"https://orcid.org/0000-0002-1735-9680","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Mike","middleName":"","lastName":"Schaekermann","suffix":""},{"id":461886574,"identity":"02bd3e0d-ba9b-44be-bd75-6dcccf77e075","order_by":34,"name":"Tao Tu","email":"","orcid":"https://orcid.org/0000-0001-9191-7938","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Tao","middleName":"","lastName":"Tu","suffix":""},{"id":461886575,"identity":"f2dea99e-fb83-4971-a84f-6dcc64a51fb5","order_by":35,"name":"Alan Karthikesalingam","email":"","orcid":"https://orcid.org/0009-0000-4958-5976","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Alan","middleName":"","lastName":"Karthikesalingam","suffix":""}],"badges":[],"createdAt":"2025-05-21 07:41:53","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6713863/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6713863/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":83883639,"identity":"709c8cb4-e2dd-4d24-b093-209fcf078fb2","added_by":"auto","created_at":"2025-06-04 06:14:28","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":9820413,"visible":true,"origin":"","legend":"Article File","description":"","filename":"AdvancingConversationalDiagnosticAIwithMultimodalReasoning.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6713863/v1_covered_bb9b9d41-b93f-433f-99ab-5ad2a06e62a7.pdf"}],"financialInterests":"\u003cb\u003eYes\u003c/b\u003e there is potential Competing Interest.\nContent related to my submission was recently published by this journal","formattedTitle":"Advancing Conversational Diagnostic AI with Multimodal Reasoning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6713863/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6713863/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"AI systems based on Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms can permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in conversation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal medical data, and reason about this precisely during consultations. \r\n\r\nLeveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process which emulates that of experienced clinicians. \r\n\r\nWe compared AMIE to primary care physicians (PCPs) in a randomized, double-blind study of chat based consultations with 25 patient actors in the style of an Objective Structured Clinical Examination (OSCE). We constructed 105 evaluation scenarios, using common artifacts such as smartphone photos of skin conditions, ECG tracings, and PDFs of clinical documents across a diversity of conditions and demographics. We designed a rubric to assess multiple aspects of the multimodal capability of AMIE and PCPs, in addition to other clinically meaningful axes such as history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. Evaluation by 18 specialists revealed superior performance of AMIE relative to PCPs in handling and reasoning about multimodal data on 7 out of 9 axes while displaying similar superior performance in the non-multimodal metrics, including diagnostic accuracy, on 29 out of 32 axes. These results demonstrate clear progress in multimodal conversational diagnostic AI, although real-world translation necessitates further research.","manuscriptTitle":"Advancing Conversational Diagnostic AI with Multimodal Reasoning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-04 06:06:08","doi":"10.21203/rs.3.rs-6713863/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-medicine","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"nm","sideBox":"Learn more about [Nature Medicine](http://www.nature.com/nm/)","snPcode":"","submissionUrl":"","title":"Nature Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"dcaa791f-ea8f-4ea4-a9c6-9b4c19d647f9","owner":[],"postedDate":"June 4th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":49046427,"name":"Health sciences/Health care"},{"id":49046428,"name":"Health sciences/Diseases"}],"tags":[],"updatedAt":"2025-06-04T06:06:08+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-04 06:06:08","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6713863","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6713863","identity":"rs-6713863","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}