Blind Expert Evaluation of Open-Weight LLMs for Thyroid Cancer Patient Education in a Non-English Setting: GPT-OSS-20B vs MedGemma-27B-Instruct

Blind Expert Evaluation of Open-Weight LLMs for Thyroid Cancer Patient Education in a Non-English Setting: GPT-OSS-20B vs MedGemma-27B-Instruct | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Blind Expert Evaluation of Open-Weight LLMs for Thyroid Cancer Patient Education in a Non-English Setting: GPT-OSS-20B vs MedGemma-27B-Instruct Mehmet Poyrazer, Hüseyin Yağcı, Alican Bozdaş, Ayşe Münevver Mühürdaroğlu, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8729661/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 12 You are reading this latest preprint version Abstract Background: Non-English and resource-constrained clinical contexts are underrepresented in current large language models (LLM) benchmarking, making it uncertain whether medical specialization improves patient-facing education when models are deployed locally. Open-weight LLMs are increasingly used for patient education, yet it remains unclear whether medical domain specialization improves patient-facing answers in non-English settings. We compared a general-purpose open-weight model (GPT-OSS-20B) with a medically specialized open-weight model (MedGemma-27B-Instruct) for thyroid cancer patient education in Turkish. Methods : Sixty Turkish patient questions about thyroid cancer were answered by both models. Five endocrinologists, blinded to model identity and study hypotheses, rated each response on 5-point Likert scales for Accuracy, Completeness, Clarity, Clinical Utility, and Satisfaction. Primary inference used per-question median ratings (N = 60 paired observations per criterion) with Wilcoxon signed-rank tests and Holm adjustment; effect size was rank-biserial correlation (RBC), and location shift was estimated with Hodges–Lehmann differences. Inter-rater reliability was assessed using ICC (2, k), and ceiling-aware summaries included perfect-score and top-box analyses. Results : GPT-OSS-20B achieved higher question-level median ratings than MedGemma-27B-Instruct across all five criteria after Holm correction. The largest differences were observed for Satisfaction (median 5.0 vs 4.0; RBC = 0.788; Holm-adjusted p < 0.001) and Completeness (median 5.0 vs 4.0; RBC = 0.599; Holm-adjusted p < 0.001). Inter-rater reliability was good and comparable across models (ICC (2, k) ≈ 0.74–0.80). Ceiling aware reporting showed consistently higher perfect-score proportions for GPT-OSS-20B across criteria, with the most pronounced gaps in Satisfaction and Completeness. Conclusions : In this first head-to-head comparison of open-weight LLMs for thyroid cancer patient education in Turkish, the general-purpose GPT-OSS-20B significantly outperformed the medically fine-tuned MedGemma-27B-Instruct across all evaluation criteria. These findings suggest that medical domain specialization does not necessarily yield superior patient-facing educational content in non-English settings and that general-purpose open-weight models may offer advantages for patient education tasks in resource-constrained contexts. Biological sciences/Cancer Health sciences/Endocrinology Health sciences/Health care Health sciences/Medical research Health sciences/Oncology Thyroid cancer Large language models Open-weight models Blind expert evaluation Patient education Full Text Additional Declarations No competing interests reported. Supplementary Files figureS1.pdf Figure S1. Category-wise distributions of question-level median expert ratings for both models across all evaluation criteria. Panels show (A) Accuracy, (B) Clarity, (C) Completeness, (D) Clinical Utility, and (E) Satisfaction. For each category, boxplots summarize the distribution of question-level median ratings (median and IQR; whiskers = 1.5×IQR), and overlaid points represent individual questions. Scores are based on medians across five endocrinologists per question (N = 60 questions total), grouped into six categories (Basic Information & Diagnosis n = 12; Treatment & Surgery n = 15; Follow-up & Prognosis n = 8; Lifestyle & Nutrition n = 8; Postoperative Life n = 12; Special Situations n = 5). The category-wise plots are descriptive/exploratory and provided to visualize potential heterogeneity and ceiling effects. TableS1.docx.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 11 May, 2026 Reviewers agreed at journal 27 Apr, 2026 Reviews received at journal 14 Mar, 2026 Reviews received at journal 11 Mar, 2026 Reviewers agreed at journal 10 Mar, 2026 Reviewers agreed at journal 10 Mar, 2026 Reviewers agreed at journal 10 Mar, 2026 Reviewers invited by journal 10 Mar, 2026 Editor invited by journal 03 Feb, 2026 Editor assigned by journal 30 Jan, 2026 Submission checks completed at journal 30 Jan, 2026 First submitted to journal 29 Jan, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8729661","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":605113545,"identity":"75a7315a-943b-4970-aadb-4aaf585c65fd","order_by":0,"name":"Mehmet Poyrazer","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGUlEQVRIiWNgGAWjYDCCA0DM2IAswg7iGVgQ1sIDF+EBCRlIkKJFIgFM4tTBd/sA64aPO7bJ2Us3H5OuqKmTM5/5/OqGHwUSDPzt3QnYtEieS2C7OfPMbWMemWNpkmeOsRnL3M4pu9kDdJjEmbMbsGkxOMPAdpu37XZij0SOmWQDG0/iDOmctBs8QC0GErm4tfxtu10P0fJPon6G5Jm0m38IaWFsu53AA9LS2GaQICHBfuw2PlskzzC23extu23Yc+dYsmVjX4LhDJ4cttsyBhI8uPzCd4b52I2fbbfl2Wc3H7zZ8K1OXoL9+LObb/7YyPG392LVgoh6CQYWaFzwGIBJ7MqRgQQD8wcIi/0BYdWjYBSMglEwkgAANTRk2TDtw2EAAAAASUVORK5CYII=","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":true,"prefix":"","firstName":"Mehmet","middleName":"","lastName":"Poyrazer","suffix":""},{"id":605113546,"identity":"4bf0d045-c5cc-4317-8ba9-b67da3a94a16","order_by":1,"name":"Hüseyin Yağcı","email":"","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":false,"prefix":"","firstName":"Hüseyin","middleName":"","lastName":"Yağcı","suffix":""},{"id":605113547,"identity":"afee4100-af78-4eb9-91e4-cf1ec0275e21","order_by":2,"name":"Alican Bozdaş","email":"","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":false,"prefix":"","firstName":"Alican","middleName":"","lastName":"Bozdaş","suffix":""},{"id":605113548,"identity":"c573aa3b-ede3-46a6-b47b-0d84657fb521","order_by":3,"name":"Ayşe Münevver Mühürdaroğlu","email":"","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":false,"prefix":"","firstName":"Ayşe","middleName":"Münevver","lastName":"Mühürdaroğlu","suffix":""},{"id":605113549,"identity":"78b29576-df24-4c0b-9ae6-2ab3a3348e4d","order_by":4,"name":"Çağatay Emir Önder","email":"","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":false,"prefix":"","firstName":"Çağatay","middleName":"Emir","lastName":"Önder","suffix":""},{"id":605113550,"identity":"fb566ffa-1dc2-46b1-8bcf-2278d55f39e5","order_by":5,"name":"Sevde Nur Fırat","email":"","orcid":"","institution":"University of Health Sciences, Ankara Training and Research Hospital","correspondingAuthor":false,"prefix":"","firstName":"Sevde","middleName":"Nur","lastName":"Fırat","suffix":""}],"badges":[],"createdAt":"2026-01-29 09:45:56","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8729661/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8729661/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104782455,"identity":"7d8d3eac-0503-4ce0-a3ed-00158ef39172","added_by":"auto","created_at":"2026-03-17 07:57:20","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":551514,"visible":true,"origin":"","legend":"","description":"","filename":"makalebmc.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8729661/v1_covered_a35e27fd-b95f-466b-9f0d-f0df31c49aa4.pdf"},{"id":104588623,"identity":"cfde544c-2971-48c3-849a-97c527da33bd","added_by":"auto","created_at":"2026-03-13 16:26:23","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":48878,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFigure S1. \u003c/strong\u003eCategory-wise distributions of question-level median expert ratings for both models across all evaluation criteria. Panels show (A) Accuracy, (B) Clarity, (C) Completeness, (D) Clinical Utility, and (E) Satisfaction. For each category, boxplots summarize the distribution of question-level median ratings (median and IQR; whiskers = 1.5×IQR), and overlaid points represent individual questions. Scores are based on medians across five endocrinologists per question (N = 60 questions total), grouped into six categories (Basic Information \u0026amp; Diagnosis n = 12; Treatment \u0026amp; Surgery n = 15; Follow-up \u0026amp; Prognosis n = 8; Lifestyle \u0026amp; Nutrition n = 8; Postoperative Life n = 12; Special Situations n = 5). The category-wise plots are descriptive/exploratory and provided to visualize potential heterogeneity and ceiling effects.\u003c/p\u003e","description":"","filename":"figureS1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8729661/v1/eef8948a006ae3ec452e53c3.pdf"},{"id":104588624,"identity":"e23117b5-02e1-4421-a21a-77b54990be6e","added_by":"auto","created_at":"2026-03-13 16:26:23","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":36579,"visible":true,"origin":"","legend":"","description":"","filename":"TableS1.docx.docx","url":"https://assets-eu.researchsquare.com/files/rs-8729661/v1/ee9d78f8ff27a0e3bb25d350.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Blind Expert Evaluation of Open-Weight LLMs for Thyroid Cancer Patient Education in a Non-English Setting: GPT-OSS-20B vs MedGemma-27B-Instruct","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Thyroid cancer, Large language models, Open-weight models, Blind expert evaluation, Patient education","lastPublishedDoi":"10.21203/rs.3.rs-8729661/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8729661/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground: \u003c/strong\u003eNon-English and resource-constrained clinical contexts are underrepresented in current large language models (LLM) benchmarking, making it uncertain whether medical specialization improves patient-facing education when models are deployed locally. Open-weight LLMs are increasingly used for patient education, yet it remains unclear whether medical domain specialization improves patient-facing answers in non-English settings. We compared a general-purpose open-weight model (GPT-OSS-20B) with a medically specialized open-weight model (MedGemma-27B-Instruct) for thyroid cancer patient education in Turkish.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: Sixty Turkish patient questions about thyroid cancer were answered by both models. Five endocrinologists, blinded to model identity and study hypotheses, rated each response on 5-point Likert scales for Accuracy, Completeness, Clarity, Clinical Utility, and Satisfaction. Primary inference used per-question median ratings (N = 60 paired observations per criterion) with Wilcoxon signed-rank tests and Holm adjustment; effect size was rank-biserial correlation (RBC), and location shift was estimated with Hodges–Lehmann differences. Inter-rater reliability was assessed using ICC (2, k), and ceiling-aware summaries included perfect-score and top-box analyses.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: GPT-OSS-20B achieved higher question-level median ratings than MedGemma-27B-Instruct across all five criteria after Holm correction. The largest differences were observed for Satisfaction (median 5.0 vs 4.0; RBC = 0.788; Holm-adjusted p \u0026lt; 0.001) and Completeness (median 5.0 vs 4.0; RBC = 0.599; Holm-adjusted p \u0026lt; 0.001). Inter-rater reliability was good and comparable across models (ICC (2, k) ≈ 0.74–0.80). Ceiling aware reporting showed consistently higher perfect-score proportions for GPT-OSS-20B across criteria, with the most pronounced gaps in Satisfaction and Completeness.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e: In this first head-to-head comparison of open-weight LLMs for thyroid cancer patient education in Turkish, the general-purpose GPT-OSS-20B significantly outperformed the medically fine-tuned MedGemma-27B-Instruct across all evaluation criteria. These findings suggest that medical domain specialization does not necessarily yield superior patient-facing educational content in non-English settings and that general-purpose open-weight models may offer advantages for patient education tasks in resource-constrained contexts.\u003c/p\u003e","manuscriptTitle":"Blind Expert Evaluation of Open-Weight LLMs for Thyroid Cancer Patient Education in a Non-English Setting: GPT-OSS-20B vs MedGemma-27B-Instruct","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-13 16:26:19","doi":"10.21203/rs.3.rs-8729661/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-11T06:57:42+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"5508448384231522922491046122312235406","date":"2026-04-27T15:01:52+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-14T21:05:17+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-12T01:37:13+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"138180824923343981237214170587429656872","date":"2026-03-10T19:33:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"268203084817013843306468984681343154625","date":"2026-03-10T14:30:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"300973840835648595773995261156302981170","date":"2026-03-10T13:31:26+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-10T13:22:26+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-02-03T14:05:57+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-01-30T13:22:01+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-01-30T13:17:34+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2026-01-29T08:56:45+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cc5b6d75-828f-48d7-9eb5-71c95a6a3fb2","owner":[],"postedDate":"March 13th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-11T06:57:42+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":64397731,"name":"Biological sciences/Cancer"},{"id":64397732,"name":"Health sciences/Endocrinology"},{"id":64397733,"name":"Health sciences/Health care"},{"id":64397734,"name":"Health sciences/Medical research"},{"id":64397735,"name":"Health sciences/Oncology"}],"tags":[],"updatedAt":"2026-05-18T12:25:14+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-13 16:26:19","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8729661","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8729661","identity":"rs-8729661","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}