Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation Ryutaro Tanno, David Barrett, Andrew Sellergren, Sumedh Ghaisas, and 22 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3940387/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 07 Nov, 2024 Read the published version in Nature Medicine → Version 1 posted You are reading this latest preprint version Abstract Radiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors in report delivery. While recent progress in automated report generation with vision-language models offers clear potential to ameliorate this situation, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of AI-generated reports. In this study, we build a state-of-the-art report generation system for chest radiographs, Flamingo-CXR , by fine-tuning a well-known vision-language foundation model on radiology data. To measure the quality of the AI-generated reports, we perform an expert evaluation, that is largest in scale and diversity to date, by engaging a group of 27 certified radiologists in the United States and India to provide detailed assessment of AI-generated and human written reports from an intensive care setting as well as an inpatient setting. We observe a wide distribution of preferences across the panel, ranging from full consensus to dissensus, across clinical settings and regions, with 55.4% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for outpatient x-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. For reports that contain errors we develop an assistive setting, the first demonstration of clinician-AI collaboration for radiology report composition, and we observe a synergistic improvement across all clinical settings. Altogether, these nuanced evaluations reveal disparities between the AI system and radiologists, identify areas for potential clinical utility and pave the way toward a collaborative system that enhances clinical accuracy of radiology reporting. Health sciences/Medical research Health sciences/Health care/Medical imaging radiology report generation clinician-AI collaboration vision-language models Full Text Additional Declarations Yes there is potential Competing Interest. This study was funded by Alphabet Inc and/or a subsidiary thereof (‘Alphabet’). All authors are employed by Google except Danielle Belgrave. Danielle Belgrave is currently at GSK but was employed by Google when the research was conducted. Cite Share Download PDF Status: Published Journal Publication published 07 Nov, 2024 Read the published version in Nature Medicine → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3940387","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":271985313,"identity":"a7a23158-2b3b-4fc1-8333-a59829f026ac","order_by":0,"name":"Ryutaro Tanno","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAwElEQVRIiWNgGAWjYPACGxDBeIA4xWxgMg1MkqTlMAlazOc3P3tcUHHerp+9/cKBDwx2croNBLTIHGMzN55x5nbyzJ4zBQdnMCQbmxGyS4KNwUyat+12ssGNnITDPAwHErcR1sL+DajlHETLH+K08IBsOWBncCP9ADAQiNKSUybNcyY5QbLnDMPBHgNi/MJ8fJs0T4WdPT97+8MHPyrs5AhqgYHEBgYeAwYGAyKVg4A9AwP7AxLUj4JRMApGwUgCAIsLQFSPpNACAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-8107-6730","institution":"Google DeepMind","correspondingAuthor":true,"prefix":"","firstName":"Ryutaro","middleName":"","lastName":"Tanno","suffix":""},{"id":271985314,"identity":"45a7b228-35b7-4aca-b26d-f0fc6a3d278d","order_by":1,"name":"David Barrett","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Barrett","suffix":""},{"id":271985315,"identity":"e62b3b71-459b-4aab-a63a-2bb8c75f90b2","order_by":2,"name":"Andrew Sellergren","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Andrew","middleName":"","lastName":"Sellergren","suffix":""},{"id":271985316,"identity":"99a341b3-fe7b-46c7-91b1-2ab32ab20640","order_by":3,"name":"Sumedh Ghaisas","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Sumedh","middleName":"","lastName":"Ghaisas","suffix":""},{"id":271985317,"identity":"ff18c428-2ae1-4437-a8f3-503a833bb148","order_by":4,"name":"Sumanth Dathathri","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Sumanth","middleName":"","lastName":"Dathathri","suffix":""},{"id":271985318,"identity":"9e730db9-0830-4c3a-a1a9-d38ad1a58e56","order_by":5,"name":"Abigail See","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Abigail","middleName":"","lastName":"See","suffix":""},{"id":271985319,"identity":"a0e4d27c-3321-4e4e-8230-112559c8b562","order_by":6,"name":"Johannes Welbl","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Johannes","middleName":"","lastName":"Welbl","suffix":""},{"id":271985320,"identity":"11b5683c-91de-423c-8624-9f2c0b621abf","order_by":7,"name":"Karan Singhal","email":"","orcid":"https://orcid.org/0009-0001-0286-609X","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Karan","middleName":"","lastName":"Singhal","suffix":""},{"id":271985321,"identity":"5df33d0f-0e25-4429-a5e6-b353812adcec","order_by":8,"name":"Shekoofeh Azizi","email":"","orcid":"https://orcid.org/0000-0002-7447-6031","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Shekoofeh","middleName":"","lastName":"Azizi","suffix":""},{"id":271985322,"identity":"d45209ec-e4b4-41e4-b473-4930b234828a","order_by":9,"name":"Tao Tu","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Tao","middleName":"","lastName":"Tu","suffix":""},{"id":271985323,"identity":"08db2a73-8e59-4728-b28a-c3ab9d33b49c","order_by":10,"name":"Mike Schaekermann","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Mike","middleName":"","lastName":"Schaekermann","suffix":""},{"id":271985324,"identity":"8847ecaa-40a5-4d0f-89aa-a253aa2a415a","order_by":11,"name":"Rhys May","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Rhys","middleName":"","lastName":"May","suffix":""},{"id":271985325,"identity":"a1488b1f-3294-4c7d-a2c5-774a585262c5","order_by":12,"name":"Roy Lee","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"Roy","middleName":"","lastName":"Lee","suffix":""},{"id":271985326,"identity":"2ea2f1bc-0f97-4f1d-82b9-c7dbfc1dd7f1","order_by":13,"name":"SiWai Man","email":"","orcid":"","institution":"Google Research","correspondingAuthor":false,"prefix":"","firstName":"SiWai","middleName":"","lastName":"Man","suffix":""},{"id":271985327,"identity":"835bef8f-1668-4a31-af08-68adaacb9f16","order_by":14,"name":"Zahra Ahmed","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Zahra","middleName":"","lastName":"Ahmed","suffix":""},{"id":271985328,"identity":"6b96b049-a2aa-4d15-82bd-27243c04c8c1","order_by":15,"name":"S. Sara Mahdavi","email":"","orcid":"","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"S.","middleName":"Sara","lastName":"Mahdavi","suffix":""},{"id":271985329,"identity":"67e1bd30-853a-4d20-9957-b25167ab61aa","order_by":16,"name":"Yossi Matias","email":"","orcid":"https://orcid.org/0000-0003-3960-6002","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Yossi","middleName":"","lastName":"Matias","suffix":""},{"id":271985330,"identity":"a8963382-e476-4d89-9967-7117101fd990","order_by":17,"name":"Joelle Barral","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Joelle","middleName":"","lastName":"Barral","suffix":""},{"id":271985331,"identity":"0857147e-4b27-41fa-8326-6ff9caa70c4b","order_by":18,"name":"Ali Eslami","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Ali","middleName":"","lastName":"Eslami","suffix":""},{"id":271985332,"identity":"2b19771e-5bb1-48f6-a811-6b65513e080c","order_by":19,"name":"Danielle Belgrave","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Danielle","middleName":"","lastName":"Belgrave","suffix":""},{"id":271985333,"identity":"17aba1e6-3df9-4353-929f-46cdaa3e9ac7","order_by":20,"name":"Vivek Natarajan","email":"","orcid":"https://orcid.org/0000-0001-7849-2074","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Vivek","middleName":"","lastName":"Natarajan","suffix":""},{"id":271985334,"identity":"8ce03e4a-b7cd-49a7-a3ef-88bd246a580b","order_by":21,"name":"Shravya Shetty","email":"","orcid":"https://orcid.org/0000-0003-3783-3172","institution":"Google Health","correspondingAuthor":false,"prefix":"","firstName":"Shravya","middleName":"","lastName":"Shetty","suffix":""},{"id":271985335,"identity":"b390dba7-521b-4577-9563-3bbaf3aa03bc","order_by":22,"name":"Pushmeet Kohli","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Pushmeet","middleName":"","lastName":"Kohli","suffix":""},{"id":271985336,"identity":"c4be988f-49f2-4a4a-91d5-f5f252dda3f0","order_by":23,"name":"Po-Sen Huang","email":"","orcid":"","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Po-Sen","middleName":"","lastName":"Huang","suffix":""},{"id":271985337,"identity":"7b12127f-ef4c-4a68-8070-447fb8c332b6","order_by":24,"name":"Alan Karthikesalingam","email":"","orcid":"https://orcid.org/0009-0000-4958-5976","institution":"Google","correspondingAuthor":false,"prefix":"","firstName":"Alan","middleName":"","lastName":"Karthikesalingam","suffix":""},{"id":271985338,"identity":"7ecaa280-e76e-4cf8-a8be-e2e86715218c","order_by":25,"name":"Ira Ktena","email":"","orcid":"https://orcid.org/0000-0001-6677-6547","institution":"Google DeepMind","correspondingAuthor":false,"prefix":"","firstName":"Ira","middleName":"","lastName":"Ktena","suffix":""}],"badges":[],"createdAt":"2024-02-08 15:30:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3940387/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3940387/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41591-024-03302-1","type":"published","date":"2024-11-07T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":68524299,"identity":"866b0997-e356-4e65-ba82-d10543ccfb89","added_by":"auto","created_at":"2024-11-08 08:11:38","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1903746,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3940387/v1_covered_f8042add-71ca-46c4-aea9-b9d5ecb60a95.pdf"}],"financialInterests":"\u003cb\u003eYes\u003c/b\u003e there is potential Competing Interest.\nThis study was funded by Alphabet Inc and/or a subsidiary thereof (‘Alphabet’). All authors are employed by Google except Danielle Belgrave. Danielle Belgrave is currently at GSK but was employed by Google when the research was conducted.","formattedTitle":"Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"radiology report generation, clinician-AI collaboration, vision-language models","lastPublishedDoi":"10.21203/rs.3.rs-3940387/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3940387/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRadiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors in report delivery. While recent progress in automated report generation with vision-language models offers clear potential to ameliorate this situation, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of AI-generated reports. In this study, we build a state-of-the-art report generation system for chest radiographs, \u003cem\u003eFlamingo-CXR\u003c/em\u003e, by fine-tuning a well-known vision-language foundation model on radiology data. To measure the quality of the AI-generated reports, we perform an expert evaluation, that is largest in scale and diversity to date, by engaging a group of 27 certified radiologists in the United States and India to provide detailed assessment of AI-generated and human written reports from an intensive care setting as well as an inpatient setting. We observe a wide distribution of preferences across the panel, ranging from full consensus to dissensus, across clinical settings and regions, with 55.4% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for outpatient x-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. For reports that contain errors we develop an assistive setting, the first demonstration of clinician-AI collaboration for radiology report composition, and we observe a synergistic improvement across all clinical settings. Altogether, these nuanced evaluations reveal disparities between the AI system and radiologists, identify areas for potential clinical utility and pave the way toward a collaborative system that enhances clinical accuracy of radiology reporting.\u003c/p\u003e","manuscriptTitle":"Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-02-12 03:28:37","doi":"10.21203/rs.3.rs-3940387/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-medicine","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"nm","sideBox":"Learn more about [Nature Medicine](http://www.nature.com/nm/)","snPcode":"","submissionUrl":"","title":"Nature Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"dcaa791f-ea8f-4ea4-a9c6-9b4c19d647f9","owner":[],"postedDate":"February 12th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":28680856,"name":"Health sciences/Medical research"},{"id":28680857,"name":"Health sciences/Health care/Medical imaging"}],"tags":[],"updatedAt":"2024-11-08T08:11:27+00:00","versionOfRecord":{"articleIdentity":"rs-3940387","link":"https://doi.org/10.1038/s41591-024-03302-1","journal":{"identity":"nature-medicine","isVorOnly":false,"title":"Nature Medicine"},"publishedOn":"2024-11-07 05:00:00","publishedOnDateReadable":"November 7th, 2024"},"versionCreatedAt":"2024-02-12 03:28:37","video":"","vorDoi":"10.1038/s41591-024-03302-1","vorDoiUrl":"https://doi.org/10.1038/s41591-024-03302-1","workflowStages":[]},"version":"v1","identity":"rs-3940387","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3940387","identity":"rs-3940387","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}