Benchmarking Large Language Models on USMLE: Evaluating ChatGPT, DeepSeek, Grok, and Qwen in Clinical Reasoning and Medical Licensing Scenarios

Benchmarking Large Language Models on USMLE: Evaluating ChatGPT, DeepSeek, Grok, and Qwen in Clinical Reasoning and Medical Licensing Scenarios | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Benchmarking Large Language Models on USMLE: Evaluating ChatGPT, DeepSeek, Grok, and Qwen in Clinical Reasoning and Medical Licensing Scenarios Md Kamrul Siam, Angel Varela, Md Jobair Hossain Faruk, Jerry Q. Cheng, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6651111/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 03 Dec, 2025 Read the published version in Scientific Reports → Version 1 posted 14 You are reading this latest preprint version Abstract Artificial intelligence (AI) is transforming healthcare by assisting with intricate clinical reasoning and diagnosis. Recent research demonstrates that large language models (LLMs), such as ChatGPT and DeepSeek, possess considerable potential in medical comprehension. This study meticulously evaluates the clinical reasoning capabilities of four advanced LLMs, including ChatGPT, DeepSeek, Grok, and Qwen, utilizing the United States Medical Licensing Examination (USMLE) as a standard benchmark. We assess 376 publicly accessible USMLE sample exam questions (Step 1, Step 2 CK, Step 3) from the most recent booklet released in July 2023. We analyze model performance across four question categories—text-only, text with image, text with mathematical reasoning, and integrated text-image-mathematical reasoning—and measure model accuracy at three USMLE steps. Our findings indicate that on Step 2 CK, DeepSeek consistently outperforms other models, achieving a peak accuracy of 93%. Despite ChatGPT’s little latency, the restricted convergence in error patterns suggests that ensemble approaches might enhance effectiveness. Grok and Qwen demonstrate reduced and less dependable accuracy throughout all steps. These findings point out the importance of LLMs in clinical reasoning in medical licensing scenarios. However, we also emphasize that these procedures require improvement to ensure their safe and effective integration into practical healthcare processes. Health sciences/Diseases Health sciences/Health care Health sciences/Health occupations Health sciences/Medical research Large Language Models (LLMs) Clinical Reasoning USMLE Medical Licensing Examination Diagnostic Decision Support Full Text Additional Declarations No competing interests reported. Supplementary Files USMLEsupplementary.xlsx Step1.pdf Step2CK.pdf Step3.pdf Cite Share Download PDF Status: Published Journal Publication published 03 Dec, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 24 Jul, 2025 Reviews received at journal 21 Jul, 2025 Reviews received at journal 19 Jul, 2025 Reviews received at journal 12 Jul, 2025 Reviewers agreed at journal 05 Jul, 2025 Reviewers agreed at journal 04 Jul, 2025 Reviews received at journal 03 Jul, 2025 Reviewers agreed at journal 22 Jun, 2025 Reviewers agreed at journal 10 Jun, 2025 Reviewers invited by journal 10 Jun, 2025 Editor assigned by journal 09 Jun, 2025 Editor invited by journal 26 May, 2025 Submission checks completed at journal 24 May, 2025 First submitted to journal 12 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6651111","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":469469155,"identity":"308e1531-3989-4e59-b761-008ae6d88bc6","order_by":0,"name":"Md Kamrul Siam","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAyUlEQVRIiWNgGAWjYBACCQbGBgYGAwk5BjCDgZlYLRUWxkAtjQ1EagGBMxWJDRBriNAi2X+48dPNNon07RLJ7Q8YKqxBevEDaYnEZuncNoncnTMSgQ47k05Yi5wEYwNYy4YbQC2MbYeJ0MJ/sPk3UEu6AVjLPyK0SDMktknnnJFIgGhpIEKL5IzENuucCgnDDWceNs5IOJZuTFCLxPnjj2/nGNTJGxxPf/DhQ421LEEtCCCQwMCQQLxyEOA/QJr6UTAKRsEoGDkAAO70QzKODWaJAAAAAElFTkSuQmCC","orcid":"","institution":"New York Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"Md","middleName":"Kamrul","lastName":"Siam","suffix":""},{"id":469469156,"identity":"e8620c30-ccb9-46ad-91ca-ce9fd3a818a2","order_by":1,"name":"Angel Varela","email":"","orcid":"","institution":"Colegio Maria Cano I.E.D","correspondingAuthor":false,"prefix":"","firstName":"Angel","middleName":"","lastName":"Varela","suffix":""},{"id":469469159,"identity":"6413ab7e-dab1-4eee-863e-5ccd7ec850ca","order_by":2,"name":"Md Jobair Hossain Faruk","email":"","orcid":"","institution":"New York Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Md","middleName":"Jobair Hossain","lastName":"Faruk","suffix":""},{"id":469469165,"identity":"a373bc49-839d-4354-84c0-1f46082bf52c","order_by":3,"name":"Jerry Q. Cheng","email":"","orcid":"","institution":"New York Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Jerry","middleName":"Q.","lastName":"Cheng","suffix":""},{"id":469469168,"identity":"09022d2e-f327-4b9f-9c33-ff775d3e20aa","order_by":4,"name":"Huanying Gu","email":"","orcid":"","institution":"New York Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Huanying","middleName":"","lastName":"Gu","suffix":""},{"id":469469173,"identity":"cb5be68c-4a97-43ee-b982-afc7fb99db1c","order_by":5,"name":"Abdullah Al Maruf","email":"","orcid":"","institution":"Bangladesh University of Business and Technology","correspondingAuthor":false,"prefix":"","firstName":"Abdullah","middleName":"Al","lastName":"Maruf","suffix":""},{"id":469469176,"identity":"97658304-61d3-456a-bc87-22bbc09afb2c","order_by":6,"name":"Zeyar Aung","email":"","orcid":"","institution":"Khalifa University","correspondingAuthor":false,"prefix":"","firstName":"Zeyar","middleName":"","lastName":"Aung","suffix":""}],"badges":[],"createdAt":"2025-05-13 03:53:36","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6651111/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6651111/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-31010-4","type":"published","date":"2025-12-03T15:56:54+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":97723735,"identity":"b52cc205-e92e-4e0c-b6c7-0a259741fe5a","added_by":"auto","created_at":"2025-12-08 15:59:40","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":582925,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6651111/v1_covered_89c93a60-1705-4ee3-963f-e79c0660ecd0.pdf"},{"id":84449167,"identity":"b100ce8a-019e-4ff4-8451-ad73eca9de53","added_by":"auto","created_at":"2025-06-12 06:28:03","extension":"xlsx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":21645,"visible":true,"origin":"","legend":"","description":"","filename":"USMLEsupplementary.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-6651111/v1/c0fd458c9ca82b0288ff3b3a.xlsx"},{"id":84449171,"identity":"8749f8b4-e51b-44c0-ab2f-96518945cdeb","added_by":"auto","created_at":"2025-06-12 06:28:03","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":1825143,"visible":true,"origin":"","legend":"","description":"","filename":"Step1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6651111/v1/d5798f593bbc3b39282e81b5.pdf"},{"id":84450463,"identity":"47d82902-6008-4987-9272-7e068b9aaafb","added_by":"auto","created_at":"2025-06-12 06:44:03","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":2113629,"visible":true,"origin":"","legend":"","description":"","filename":"Step2CK.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6651111/v1/53350ab79c7aa318dfae545f.pdf"},{"id":84449173,"identity":"02b4f619-c2e6-43b1-b7e4-ac089b458449","added_by":"auto","created_at":"2025-06-12 06:28:03","extension":"pdf","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":1763539,"visible":true,"origin":"","legend":"","description":"","filename":"Step3.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6651111/v1/1ead3ac4351e699b216d524d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Benchmarking Large Language Models on USMLE: Evaluating ChatGPT, DeepSeek, Grok, and Qwen in Clinical Reasoning and Medical Licensing Scenarios","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models (LLMs), Clinical Reasoning, USMLE, Medical Licensing Examination, Diagnostic Decision Support","lastPublishedDoi":"10.21203/rs.3.rs-6651111/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6651111/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Artificial intelligence (AI) is transforming healthcare by assisting with intricate clinical reasoning and diagnosis. Recent research demonstrates that large language models (LLMs), such as ChatGPT and DeepSeek, possess considerable potential in medical comprehension. This study meticulously evaluates the clinical reasoning capabilities of four advanced LLMs, including ChatGPT, DeepSeek, Grok, and Qwen, utilizing the United States Medical Licensing Examination (USMLE) as a standard benchmark. We assess 376 publicly accessible USMLE sample exam questions (Step 1, Step 2 CK, Step 3) from the most recent booklet released in July 2023. We analyze model performance across four question categories—text-only, text with image, text with mathematical reasoning, and integrated text-image-mathematical reasoning—and measure model accuracy at three USMLE steps. Our findings indicate that on Step 2 CK, DeepSeek consistently outperforms other models, achieving a peak accuracy of 93%. Despite ChatGPT’s little latency, the restricted convergence in error patterns suggests that ensemble approaches might enhance effectiveness. Grok and Qwen demonstrate reduced and less dependable accuracy throughout all steps. These findings point out the importance of LLMs in clinical reasoning in medical licensing scenarios. However, we also emphasize that these procedures require improvement to ensure their safe and effective integration into practical healthcare processes.","manuscriptTitle":"Benchmarking Large Language Models on USMLE: Evaluating ChatGPT, DeepSeek, Grok, and Qwen in Clinical Reasoning and Medical Licensing Scenarios","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-12 06:27:59","doi":"10.21203/rs.3.rs-6651111/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-07-24T08:34:32+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-22T02:00:23+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-19T14:39:35+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-12T11:11:34+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"74371460915621473006188495779738994239","date":"2025-07-05T04:08:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"330768709569173946237323252293708839637","date":"2025-07-04T12:41:29+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-03T17:56:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"198211535093647328405698686518905762114","date":"2025-06-23T01:06:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"124124068038278379633166520329462814775","date":"2025-06-10T12:22:26+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-10T10:46:21+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-06-09T07:08:43+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-05-27T03:12:01+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-24T12:36:24+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-05-13T03:42:28+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a4ecab9b-07bb-4ce9-b2ff-d5bf5fe5f882","owner":[],"postedDate":"June 12th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":49846725,"name":"Health sciences/Diseases"},{"id":49846726,"name":"Health sciences/Health care"},{"id":49846727,"name":"Health sciences/Health occupations"},{"id":49846728,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2025-12-08T15:58:43+00:00","versionOfRecord":{"articleIdentity":"rs-6651111","link":"https://doi.org/10.1038/s41598-025-31010-4","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-12-03 15:56:54","publishedOnDateReadable":"December 3rd, 2025"},"versionCreatedAt":"2025-06-12 06:27:59","video":"","vorDoi":"10.1038/s41598-025-31010-4","vorDoiUrl":"https://doi.org/10.1038/s41598-025-31010-4","workflowStages":[]},"version":"v1","identity":"rs-6651111","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6651111","identity":"rs-6651111","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}