Leveraging Large Language Models to Construct Feedback from Medical Multiple-Choice Questions

Leveraging Large Language Models to Construct Feedback from Medical Multiple-Choice Questions | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Leveraging Large Language Models to Construct Feedback from Medical Multiple-Choice Questions Mihaela Tomova, Iván Roselló Atanet, Victoria Sehy, Miriam Sieg, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4706589/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Exams like the formative Progress Test Medizin can enhance their effectiveness by offering feedback beyond numerical scores. Content-based feedback, which encompasses relevant information from exam questions, can be valuable for students by offering them insight into their performance on the current exam, as well as serving as study aids and tools for revision.Our goal was to utilize Large Language Models (LLMs) in preparing content-based feedback for the Progress Test Medizin and evaluate their effectiveness in this task.We utilize two popular LLMs and conduct a comparative assessment by performing textual similarity on the generated outputs. Furthermore, we study via a survey how medical practitioners and medical educators assess the capabilities of LLMs and perceive the usage of LLMs for the task of generating content-based feedback for PTM exams.Our findings show that both examined LLMs performed similarly. Both have their own advantages and disadvantages. Our survey results indicate that one LLM produces slightly better outputs; however, this comes at a cost since it is a paid service, while the other is free to use. Overall, medical practitioners and educators who participated in the survey find the generated feedback relevant and useful, and they are open to using LLMs for such tasks in the future.We conclude that while the content-based feedback generated by the LLM may not be perfect, it nevertheless can be considered a valuable addition to the numerical feedback currently provided. Large Language Models Data Analysis Natural Language Processing Machine Learning Feedback Full Text Additional Declarations No competing interests reported. Supplementary Files supplementarymaterialAHSE24.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4706589","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":328012429,"identity":"70746144-f107-4285-bf39-ca2e47d6e945","order_by":0,"name":"Mihaela Tomova","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABTElEQVRIie3Pv2rCQBzA8V8InMu1XSNCfYLCBSEgpPoqdwTiYv+AS8DBlEBcAl3TpX0FXVwbOarLQddIF6UvEDehQpvU1GobO3e47/BLLncfwgHIZP8xFafj+vOV7G2cZiOa5yvtJ0kP4y1Bm0ct24hoAYFDhAXZLCBn/aNxkpAGNEve8NVxeBWq3tNg6TdajxpVItY2L09CqsTJlhj82CqHxAKMJx1dCK67CNmzO9+6CDQKERvZHS2maj3cIZhUMFF7gdY2yje+rbgIGy9YqCm5SlLCmRtTVMG7pPa2Jj3AOWnmpNfCm7+8s4eMrHeJUQHCv4jJNsThNCcRG2QE9kg9IFPAIr2LK0zLR7Y9C52pHoh5Riw2FAuvHnyTZ1GLV04XSn1vuHAd7fxW5ZM4Id1qqU/VZDlqsPupNY5XcDhU8E1x/wAymUwm+90HL0SAHuOggxgAAAAASUVORK5CYII=","orcid":"","institution":"Ilmenau University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Mihaela","middleName":"","lastName":"Tomova","suffix":""},{"id":328012431,"identity":"48c0857e-9ecd-40bc-bf2f-125ed3128307","order_by":1,"name":"Iván Roselló Atanet","email":"","orcid":"","institution":"Charité - University Medicine Berlin","correspondingAuthor":false,"prefix":"","firstName":"Iván","middleName":"Roselló","lastName":"Atanet","suffix":""},{"id":328012437,"identity":"db06bf93-b3c1-415c-b06d-7e38ee82194b","order_by":2,"name":"Victoria Sehy","email":"","orcid":"","institution":"Charité - University Medicine Berlin","correspondingAuthor":false,"prefix":"","firstName":"Victoria","middleName":"","lastName":"Sehy","suffix":""},{"id":328012447,"identity":"4beb60f5-bada-4b78-b7d8-b0f71fe6a317","order_by":3,"name":"Miriam Sieg","email":"","orcid":"","institution":"Charité - University Medicine Berlin","correspondingAuthor":false,"prefix":"","firstName":"Miriam","middleName":"","lastName":"Sieg","suffix":""},{"id":328012452,"identity":"c6c34851-4fe0-48e5-9197-9563c36583d7","order_by":4,"name":"Maren März","email":"","orcid":"","institution":"Charité - University Medicine Berlin","correspondingAuthor":false,"prefix":"","firstName":"Maren","middleName":"","lastName":"März","suffix":""},{"id":328012457,"identity":"187fdd45-5006-46c9-9a10-3dd8d7b300c7","order_by":5,"name":"Patrick Mäder","email":"","orcid":"","institution":"Ilmenau University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Patrick","middleName":"","lastName":"Mäder","suffix":""}],"badges":[],"createdAt":"2024-07-08 15:17:01","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4706589/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4706589/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62869110,"identity":"5926cb57-b10a-4990-9f08-e669e066a9d8","added_by":"auto","created_at":"2024-08-20 12:32:03","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2180961,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4706589/v1_covered_f15530da-4e39-44ae-917f-c48992ddc6f8.pdf"},{"id":61544918,"identity":"68b6151e-4e64-44f6-b0fa-16db26d4a299","added_by":"auto","created_at":"2024-08-01 04:34:22","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":4563244,"visible":true,"origin":"","legend":"","description":"","filename":"supplementarymaterialAHSE24.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4706589/v1/525a9951f243e5e9577b1ea9.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Leveraging Large Language Models to Construct Feedback from Medical Multiple-Choice Questions","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models, Data Analysis, Natural Language Processing, Machine Learning, Feedback","lastPublishedDoi":"10.21203/rs.3.rs-4706589/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4706589/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Exams like the formative Progress Test Medizin can enhance their effectiveness by offering feedback beyond numerical scores. Content-based feedback, which encompasses relevant information from exam questions, can be valuable for students by offering them insight into their performance on the current exam, as well as serving as study aids and tools for revision.Our goal was to utilize Large Language Models (LLMs) in preparing content-based feedback for the Progress Test Medizin and evaluate their effectiveness in this task.We utilize two popular LLMs and conduct a comparative assessment by performing textual similarity on the generated outputs. Furthermore, we study via a survey how medical practitioners and medical educators assess the capabilities of LLMs and perceive the usage of LLMs for the task of generating content-based feedback for PTM exams.Our findings show that both examined LLMs performed similarly. Both have their own advantages and disadvantages. Our survey results indicate that one LLM produces slightly better outputs; however, this comes at a cost since it is a paid service, while the other is free to use. Overall, medical practitioners and educators who participated in the survey find the generated feedback relevant and useful, and they are open to using LLMs for such tasks in the future.We conclude that while the content-based feedback generated by the LLM may not be perfect, it nevertheless can be considered a valuable addition to the numerical feedback currently provided.","manuscriptTitle":"Leveraging Large Language Models to Construct Feedback from Medical Multiple-Choice Questions","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-01 04:34:16","doi":"10.21203/rs.3.rs-4706589/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"46405ec2-2ba8-457c-a708-5ab266f3c4cc","owner":[],"postedDate":"August 1st, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-08-29T11:14:29+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-01 04:34:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4706589","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4706589","identity":"rs-4706589","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}