Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal Study

Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal Study Burcu Yeliz KOLLAYAN, Tuğba CEBECİ This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9030646/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 10 You are reading this latest preprint version Abstract Background: Oral cancer is still a major global health problem. It is among the most common malignancies worldwide and continues to cause significant morbidity and mortality. As patients increasingly consult large language models (LLMs) for health information prior to professional evaluation, assessing the clinical safety and reliability of AI-generated responses in oral oncology has become essential. Methods: This prospective longitudinal comparative study evaluated two advanced LLMs (Google Gemini Pro and xAI Grok-1) over a 7-day period. Twenty standardized Turkish-language oral cancer–related patient scenarios were submitted daily to each model, yielding 280 total responses. Scientific accuracy and completeness were assessed using a 5-point Likert scale by two independent oral and maxillofacial radiologists. Objective readability was measured using validated Turkish formulas (Ateşman and Bezirci–Yılmaz). Referral safety was evaluated as a binary outcome. Temporal stability was assessed using Cronbach’s alpha, and inter-model agreement was analyzed using intraclass correlation coefficients (ICC(2,1)). Results: Mean scientific accuracy scores were 3.52 ± 0.57 for Gemini and 3.39 ± 0.68 for Grok (p = 0.072). Completeness scores were 3.40 ± 0.70 and 3.25 ± 0.78, respectively (p = 0.091). Grok generated significantly longer sentences (14.83 ± 1.16 vs. 12.61 ± 0.49; p = 0.0005), although overall readability indices did not differ significantly. Referral-safe responses were observed in 90.0% of Gemini and 92.1% of Grok outputs (p = 0.536). Temporal reliability was high (Gemini α = 0.942; Grok α = 0.886). Inter-model agreement was moderate for scientific accuracy (ICC = 0.58) and completeness (ICC = 0.50). Conclusions: Contemporary LLMs demonstrated moderate-to-high scientific accuracy and strong referral safety in oral cancer–related scenarios. While they appear to favor clinical caution over false reassurance, variability in linguistic structure and inter-model agreement highlights the need for clinician oversight. LLMs may serve as informational adjuncts but should not replace professional evaluation in suspected oral malignancies. oral cancer artificial intelligence large language models patient communication readability Introduction Oral cancer remains an important global health problem and is among the most common malignancies worldwide. Despite advances in diagnosis and treatment, it continues to cause considerable morbidity and mortality, particularly when detected at advanced stages (1). Although early-stage detection is associated with survival rates exceeding 80%, delayed diagnosis significantly reduces prognosis and increases treatment-related morbidity (2-4). Despite advancements in screening and awareness programs, delayed presentation remains common, often due to the asymptomatic nature of early oral mucosal changes (5,6). At the same time, the rapid digitalization of healthcare has significantly changed the way patients search for medical information. Many individuals now consult online sources before seeking professional care, and more recently large language models (LLMs) have become part of this information-seeking process. This shift is particularly pronounced in oncology-related fears, where benign anatomical variations such as torus palatinus or frictional keratosis are frequently misinterpreted as malignancies. In societies with heightened cancer anxiety, digital tools now play a pivotal role in shaping early health perceptions and care-seeking behavior (7-9,16). Recent studies have evaluated the quality of AI-generated medical responses; still, most investigations remain cross-sectional and model-specific, primarily focusing on ChatGPT-based systems (7-13). Comparative analyses between emerging AI architectures remain scarce, and longitudinal assessments of response stability are largely absent. In addition, existing literature predominantly evaluates English-language outputs, leaving a significant gap in understanding how LLMs perform in non-English populations—particularly in languages with distinct linguistic structures such as Turkish. Beyond scientific accuracy, patient safety in oncology communication depends on two critical dimensions: readability and referral safety. Readability determines whether information is accessible to the general population, while referral safety reflects whether the system appropriately directs patients to professional evaluation in potentially malignant scenarios. Failure in either domain may contribute to false reassurance or unnecessary anxiety. For this reason, the aim of the present longitudinal study was to investigate whether contemporary large language models are able to provide clinically safe and psychologically balanced guidance in oral cancer-related scenarios. Unlike prior studies, this research integrates: Objective Turkish-language readability metrics (Ateşman and Bezirci–Yılmaz indices (23, 24)), Inter-model agreement analysis using Intraclass Correlation Coefficient, Temporal reliability assessment using Cronbach’s alpha, A multi-threshold clinical validity framework, And referral safety evaluation as a patient-protection indicator. By combining linguistic analysis with clinical safety thresholds, this study aims to determine whether contemporary LLMs can provide stable, safe, and population-appropriate guidance in the context of oral oncology. Materials and Methods This study used a prospective longitudinal comparative design to assess the performance, stability, and clinical safety of two advanced large language models: Google Gemini (Pro version) and xAI Grok (Grok-1). To assess temporal consistency and response reproducibility, the evaluation was conducted over a continuous 7-day period. Each model was prompted independently and repeatedly using standardized patient scenarios. A total of 20 standardized Turkish-language questions were developed to simulate real-world patient concerns regarding oral cancer. The 20 standardized patient scenarios used in this study are presented in Table 1. The original questions were developed in Turkish. The scenarios included: Benign anatomical variations (e.g., torus palatinus, linea alba) Chronic trauma-related lesions Potentially malignant symptoms (non-healing ulcer, unexplained bleeding) Systemic signs (lymphadenopathy, unexplained mobility) Common oncological myths (e.g., “Does biopsy spread cancer?”) The questions were reviewed and validated by two independent oral and maxillofacial radiologists to ensure clinical relevance and construct validity. Each question was submitted to both Gemini and Grok: Once daily for 7 consecutive days In newly initiated chat sessions With chat history cleared before each interaction This approach minimized contextual memory bias and ensured independent response generation. All responses were archived in a structured database for blinded evaluation. Scientific accuracy was assessed by two expert reviewers using a 5-point Likert scale. In cases of discrepancy, consensus was achieved through joint discussion before final score assignment. The consensus score was used for all statistical analyses. 1= Very poor 2 = Poor 3 = Acceptable 4 = Good 5 = Excellent Although a 5-point Likert scale was used, no response received a score of 5 (“excellent”) in our dataset; observed scores ranged from 1 to 4. Disagreements were resolved by consensus through joint discussion, and consensus scores were used for statistical analyses. Evaluation criteria included: Alignment with current oral oncology guidelines Absence of misinformation Correct interpretation of symptoms Clinical appropriateness of explanations Completeness was evaluated using the same 5-point Likert scale. Assessment focused on: Inclusion of differential diagnoses Mention of red-flag signs Discussion of risk factors Adequacy of patient counseling information Unlike subjective readability scoring, linguistic complexity was measured using validated Turkish readability formulas: Ateşman Readability Formula A Turkish adaptation of the Flesch Reading Ease score (23). Formula: 198.825 − (40.175 × average syllables per word) − (2.610 × average sentence length) Bezirci–Yılmaz Readability Index A syllable-based Turkish readability metric estimating educational level (24). For each response, the following linguistic parameters were automatically computed: Word count Sentence count Mean sentence length Average syllables per word Ateşman score Bezirci–Yılmaz index This approach ensured objective and reproducible readability assessment. Referral safety was evaluated as a binary variable (Yes/No). A response was considered “Referral Safe” if it explicitly recommended professional clinical evaluation in scenarios suggestive of potentially malignant or serious conditions. All statistical analyses were performed using SPSS 22 software (IBM Corporation, Armonk, NY, USA). Temporal stability was assessed using Cronbach’s alpha (α). Internal consistency across the 7-day period was calculated separately for each model. Agreement between Gemini and Grok was evaluated using the ICC. A two-way mixed-effects model with absolute agreement ICC(2,1) was applied. The Friedman test was used to assess score variation over time. The Wilcoxon signed-rank test was applied to compare Likert-based domains between models. Mean readability metrics were compared descriptively and inferentially between models. Objective readability metrics between Gemini and Grok were compared using independent samples t-tests following normality assessment with the Shapiro–Wilk test. Mean sentence length, Ateşman score, and Bezirci–Yılmaz index were analyzed separately (23, 24). Statistical significance was set at p < 0.05. Clinical utility was assessed at three predefined thresholds: Low validity: ≥2 points Moderate validity: ≥3 points High validity: ≥4 points A p-value < 0.05 was considered statistically significant. Results Descriptive Overview A total of 20 standardized patient scenarios were submitted daily to each AI system over a 7-day period, yielding 140 responses per model and 280 responses overall. Objective Readability Overall readability metrics are presented in Table 2. Mean sentence length was significantly higher in Grok responses (14.83 ± 1.16) compared to Gemini (12.61 ± 0.49) (p = 0.0005). No statistically significant differences were observed between models in Ateşman scores (59.23 ± 1.33 vs. 57.67 ± 3.16; p = 0.254) or Bezirci–Yılmaz index values (9.23 ± 0.38 vs. 9.15 ± 0.88; p = 0.814). Scientific Accuracy The overall mean scientific accuracy score was 3.52 ± 0.57 for Gemini and 3.39 ± 0.68 for Grok (Table 3). The difference between models was not statistically significant (Wilcoxon signed-rank test, p = 0.072). Completeness The overall completeness score was 3.40 ± 0.70 for Gemini and 3.25 ± 0.78 for Grok (Table 3). No statistically significant difference was observed between models (Wilcoxon signed-rank test, p = 0.091). Reliability Temporal reliability analysis demonstrated high internal consistency for both models (Gemini α = 0.942; Grok α = 0.886) (Table 4). Inter-model agreement was evaluated using a two-way mixed-effects absolute agreement model ICC(2,1). ICC values were interpreted according to the criteria proposed by Koo and Li (14). Overall ICC values were: Scientific accuracy: 0.58 (95% CI: 0.21–0.81) Completeness: 0.50 (95% CI: 0.09–0.77) Readability: 0.44 (95% CI: 0.00–0.73) These findings indicate moderate agreement for accuracy and completeness, and low-to-moderate agreement for readability (Table 5). Referral Safety Referral-safe responses were observed in 90.0% of Gemini outputs and 92.1% of Grok outputs (Table 6). The difference between models was not statistically significant (p = 0.536). Threshold Analysis At the ≥2 threshold, 100% of Gemini and 99.3% of Grok responses met basic validity criteria. At the ≥3 threshold, 89.2% of Gemini and 84.5% of Grok responses met moderate validity criteria. At the ≥4 threshold, 55.4% of Gemini and 48.2% of Grok responses achieved high validity (Table 7). Discussion The growing use of large language models in healthcare communication has created both opportunities and uncertainties, particularly in the context of oncology-related patient education (8, 15). As individuals increasingly rely on AI systems for symptom interpretation and risk assessment, the potential for both beneficial triage support and unintended anxiety amplification warrants systematic evaluation (10, 21, 22). Despite growing literature on AI performance in medicine, evidence regarding longitudinal stability and referral safety in oral oncology remains limited. In oral oncology, these issues are especially important. Delayed diagnosis remains one of the most critical determinants of poor prognosis, especially in cases presenting with subtle or initially asymptomatic mucosal changes (1-3). Digital systems that underestimate persistent ulcers, unexplained bleeding, or lymphadenopathy could inadvertently contribute to diagnostic delay. Conversely, overly alarming communication may intensify cancer-related anxiety and influence health-seeking behavior in unpredictable ways. Despite increasing evaluation of AI systems in general medical domains, longitudinal data examining clinical safety, referral behavior, and linguistic balance in oral cancer contexts remain limited. In the present study, we systematically evaluated whether contemporary LLMs can provide clinically safe and psychologically balanced guidance in oral cancer–related patient scenarios. By integrating scientific accuracy, completeness, referral safety, Turkish readability metrics, and inter-model agreement across a 7-day period, this study offers a multidimensional evaluation of AI performance in oral oncology communication. Clinical Accuracy and Referral Safety Both Gemini and Grok demonstrated moderate-to-high scientific accuracy and completeness, with no statistically significant differences between models. These findings are consistent with recent oncology-focused evaluations of AI chatbots, which report generally acceptable factual alignment but variability in depth and framing (10, 15). Importantly, referral-safe responses were observed in more than 90% of high-risk scenarios in both systems. In oral oncology, delayed diagnosis remains a critical contributor to poor prognosis (1-3). Digital tools that incorrectly minimize red-flag symptoms such as persistent ulcers or unexplained bleeding could potentially delay care. The high referral rates observed in this study suggest that both models adopt a precautionary stance rather than providing false reassurance, which is a clinically desirable pattern. However, inter-model agreement for scientific accuracy (ICC = 0.58) and completeness (ICC = 0.50) was only moderate. This indicates that while both systems generally provide acceptable information, the structure, emphasis, and explanatory depth vary between models. Such variability may influence patient interpretation and perceived urgency. Beyond accuracy alone, another critical dimension of AI-based medical communication is the reliability of information provided to patients outside the clinical setting. Previous studies have shown that individuals increasingly rely on online platforms and AI-driven tools when interpreting symptoms or seeking preliminary health advice (21,22). In oncology-related contexts, this behavior can influence the timing of professional consultation and may shape patient perceptions of disease severity. Recent investigations evaluating AI chatbots in cancer-related queries have demonstrated that these systems are often able to provide broadly accurate medical information, although variability in response depth and contextual interpretation remains a concern (10,15). In line with these findings, the present study observed generally acceptable levels of scientific accuracy and completeness in responses generated by both LLMs. However, moderate inter-model agreement suggests that different AI architectures may frame similar clinical scenarios in distinct ways, which could potentially influence patient interpretation and decision-making. Readability and Patient Comprehension One strength of this study is the use of validated Turkish readability indices (Ateşman and Bezirci–Yılmaz), addressing a gap in non-English AI evaluation (18,20, 24, 25). While overall readability scores were comparable between models, Grok generated significantly longer sentences. In health communication, linguistic complexity is associated with increased cognitive load and potential misunderstanding, particularly in populations with limited health literacy (19). In the context of cancer-related concerns, even subtle differences in sentence structure may affect how patients perceive severity and risk. For this reason, readability assessment should extend beyond formula-based scores and consider structural characteristics that influence comprehension. Another dimension that deserves attention is the potential impact of AI-generated information on patient behavior and emotional response. Online health information has been shown to significantly affect health-related decision making, particularly among individuals experiencing uncertainty or anxiety about possible disease symptoms (21). In oncology contexts, ambiguous or overly complex information may increase anxiety, while overly reassuring responses may delay professional evaluation. The findings of the present study suggest that both LLMs generally adopted a precautionary communication style by recommending professional consultation in the majority of high-risk scenarios. Such referral-oriented responses may help encourage timely clinical assessment, which is essential in conditions such as oral cancer where early diagnosis substantially improves prognosis (1-3). Nevertheless, the variability observed in linguistic structure and explanatory detail highlights the importance of continued clinical oversight and further research exploring how patients interpret and act upon AI-generated medical guidance. Anxiety Amplification Versus Clinical Caution This study also explored whether AI-generated responses might contribute to unnecessary anxiety or, conversely, provide inappropriate reassurance. Although patient anxiety was not directly measured, referral behavior and linguistic features provide indirect insight. The high referral rates indicate that both models avoid minimizing potentially malignant symptoms, which is essential for patient safety. Nevertheless, variability in framing and sentence complexity may shape emotional interpretation. Prior research has shown that online health information can influence anxiety levels depending on tone and clarity (21,22). Although current LLMs appear to adopt a cautious approach rather than providing false reassurance, the psychological impact of repeated AI exposure in oncology-related contexts still requires further investigation. Temporal Stability Unlike many cross-sectional AI evaluations, this study incorporated a 7-day longitudinal design. Cronbach’s alpha values demonstrated high internal consistency for both models (Gemini α = 0.942; Grok α = 0.886), indicating stable performance over time. This is particularly relevant, as patients may consult AI tools repeatedly before seeking professional care. Nevertheless, moderate ICC values demonstrate that consistency within a model does not equate to agreement between models, underscoring the absence of standardized oncologic reasoning across AI architectures. Clinical Implications From an oral oncology perspective, early recognition and timely referral remain essential to improving outcomes (1-3). The findings suggest that contemporary LLMs may serve as informational adjuncts that encourage professional evaluation rather than replace it (10, 15, 17). However, clinicians should remain aware that AI-generated information varies in framing and depth. While LLMs may support awareness and triage, they should not substitute clinical examination, biopsy, or specialist assessment in suspected malignancies. As AI systems become increasingly integrated into patient decision-making pathways, ensuring that responses remain both clinically safe and psychologically balanced will be critical. Limitations This study evaluated only two LLM systems and one language (Turkish), which may limit generalizability. Anxiety amplification was inferred indirectly rather than measured using validated psychological instruments. Future research should incorporate patient-reported outcomes and qualitative assessment to better understand emotional impact. Additionally, real patient behavioral outcomes were not evaluated. Conclusion Contemporary large language models showed moderate-to-high scientific accuracy and generally strong referral safety. They appear to adopt a precautionary communication style rather than providing potentially misleading reassurance. However, inter-model variability and linguistic differences highlight the need for clinician oversight and continued patient-centered research as AI tools become more prominent in oncology-related health communication. Clinicians should remain aware that while LLMs may encourage appropriate referrals, they should not replace professional evaluation in suspected oral malignancies. Declarations Acknowledgements Not applicable. Author contributions BYK: Conceptualization, Methodology, Investigation, Writing – Original Draft, Writing – Review & Editing. TC: Conceptualization, Methodology, Investigation, Writing – Original Draft, Writing – Review & Editing. Funding statement The author(s) received no financial support for the research, authorship, and/or publication of this article. Data availability statement The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request. Ethics Approval and consent to participate This study did not involve human participants, patient data, or clinical records. The analysis was conducted solely on AI-generated responses. Therefore, institutional ethics committee approval was not required in accordance with local regulations. Consent for publication Not applicable. Competing interests The author(s) declare no competing interests. Clinical trial number: Not applicable. References Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(3):209-249. doi:10.3322/caac.21660. Abati S, Bramati C, Bondi S, Lissoni A, Trimarchi M. Oral cancer and precancer: a narrative review on the relevance of early diagnosis. Int J Environ Res Public Health. 2020;17(24):9160. doi:10.3390/ijerph17249160. Dhanuthai K, Rojanawatsirivej S, Thosaporn W, et al. Oral cancer: a multicenter study. Med Oral Patol Oral Cir Bucal. 2018;23(1):e23. doi:10.4317/medoral.21999. Ünal Erzurumlu Z, Gökkurt Yılmaz BN, Şahiner SP, Sadık E. Bir grup hastanın ağız kanseri ve risk faktörlerinin farkındalığının değerlendirilmesi. Selcuk Dent J. 2025;12(3):372-376. Alqahtani WS, Almufareh NA, Al-Johani HA, Alotaibi RK, Juliana CI, Aljarba NH, et al. Oral and oropharyngeal cancers and possible risk factors across Gulf Cooperation Council countries: a systematic review. World J Oncol. 2020;11(4):173-181. Keser G, Öztürk M, Namdar Pekiner F. Assessment of awareness and knowledge of oral cancer among tobacco-using dental patients. Clin Exp Health Sci. 2021;11(2):279-284. Dergaa I, Fekih-Romdhane F, Hallit S, Loch AA, Glenn JM, Fessi MS, et al. ChatGPT is not ready yet for use in providing mental health assessment and interventions. Front Psychiatry. 2024;14:1277756. Sun G, Zhou YH. AI in healthcare: navigating opportunities and challenges in digital communication. Front Digit Health. 2023;5:1291132. Daniel T, de Chevigny A, Champrigaud A, Valette J, Sitbon M, Jardin M, et al. Answering hospital caregivers’ questions at any time: proof-of-concept study of an artificial intelligence–based chatbot in a French hospital. JMIR Hum Factors. 2022;9(4):e39102. Pan A, Musheyev D, Bockelman D, Loeb S, Kabarriti AE. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol. 2023;9(10):1437-1440. doi:10.1001/jamaoncol.2023.2947. Erden Y, Temel MH, Bağcıer F. Artificial intelligence insights into osteoporosis: assessing ChatGPT’s information quality and readability. Arch Osteoporos. 2024;19(1):17. Nian PP, Saleet J, Magruder M, Wellington IJ, Choueka J, Houten JK, et al. ChatGPT as a source of patient information for lumbar spinal fusion and laminectomy: a comparative analysis against Google web search. Clin Spine Surg. 2024;37(10):E394-E403. Musheyev D, Pan A, Kabarriti AE, Loeb S, Borin JF. Quality of information about kidney stones from artificial intelligence chatbots. J Endourol. 2024;38(10):1056-1061. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012. Chen D, Parsa R, Swanson K, Nunez JJ, Critch A, Bitterman DS, et al. Large language models in oncology: a review. BMJ Oncol. 2025;4(1):e000759. doi:10.1136/bmjonc-2025-000759. Teplinsky E, Ponce SB, Drake EK, Garcia AM, Loeb S, van Londen GJ, et al. Online medical misinformation in cancer: distinguishing fact from fiction. JCO Oncol Pract. 2022;18(8):584-589. doi:10.1200/OP.21.00764. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44-56. doi:10.1038/s41591-018-0300-7. Abreu AA, Murimwa GZ, Farah E, Stewart JW, Zhang L, Rodriguez J, et al. Enhancing readability of online patient-facing content: the role of AI chatbots in improving cancer information accessibility. J Natl Compr Canc Netw. 2024;22(2):e237334. doi:10.6004/jnccn.2023.7334. van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44(1):85-93. doi:10.1111/j.1365-2923.2009.03498.x. Szabó P, Bíró É, Kósa K. Readability and comprehension of printed patient education materials. Front Public Health. 2021;9:725840. doi:10.3389/fpubh.2021.725840. Bujnowska-Fedak MM, Węgierek P. The impact of online health information on patient health behaviours and making decisions concerning health. Int J Environ Res Public Health. 2020;17(3):880. doi:10.3390/ijerph17030880. Link E, Baumann E, Klimmt C. Explaining online information seeking behaviors in people with different health statuses: German representative cross-sectional survey. J Med Internet Res. 2021;23(12):e25963. Ateşman E. Türkçede okunabilirliğin ölçülmesi. Dil Dergisi. 1997;58:71-74. Bezirci B, Yılmaz AE. Türkçe için yeni bir okunabilirlik ölçütü: Bezirci-Yılmaz formülü. Gazi Univ J Sci. 2010;23(3):353-358. Tables Tables are available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files Tables.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 07 Apr, 2026 Reviews received at journal 28 Mar, 2026 Reviewers agreed at journal 23 Mar, 2026 Reviews received at journal 18 Mar, 2026 Reviewers agreed at journal 15 Mar, 2026 Reviewers invited by journal 13 Mar, 2026 Editor invited by journal 05 Mar, 2026 Editor assigned by journal 05 Mar, 2026 Submission checks completed at journal 05 Mar, 2026 First submitted to journal 04 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9030646","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":606368145,"identity":"68f6eed1-6192-468b-8e91-d9d23d45e15a","order_by":0,"name":"Burcu Yeliz KOLLAYAN","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA0klEQVRIiWNgGAWjYBACAygtB2EYWBCvxdiAgRnElSBeS+IGsBYGIrSYSyQfk66ouZO+nb3/6IYfBRIM/O3dCXi1WM5IS5M8c+xZ7s6ew2w3e4AOkzhzdgN+h93IMTZsYDucu+FGMtsNHqAWA4lcQlryPxs2/DucbgDUcvMPcVpyGB82th1OAGm5TZQtlj3PDB829h023HDmsNltGQMJHoJ+MWdPfnCw4dtheYPjjc9uvvljI8ff3otfC4NAAiqfB79yEOA/QFjNKBgFo2AUjHAAAI5SSkLSErLgAAAAAElFTkSuQmCC","orcid":"","institution":"Mehmet Akif Ersoy University","correspondingAuthor":true,"prefix":"","firstName":"Burcu","middleName":"Yeliz","lastName":"KOLLAYAN","suffix":""},{"id":606368148,"identity":"b50c1402-e94c-42de-b12f-5788c79f2fc8","order_by":1,"name":"Tuğba CEBECİ","email":"","orcid":"","institution":"Marmara University","correspondingAuthor":false,"prefix":"","firstName":"Tuğba","middleName":"","lastName":"CEBECİ","suffix":""}],"badges":[],"createdAt":"2026-03-04 13:10:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9030646/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9030646/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104700868,"identity":"a8cefebb-933a-4fe6-b340-d09a38c3ef93","added_by":"auto","created_at":"2026-03-16 08:29:39","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":572874,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9030646/v1/7cf3fa24-60bb-48f2-ad6d-b959bbcb6422.pdf"},{"id":104700808,"identity":"a3a6a117-a0f7-4795-9168-89fdc9ae6686","added_by":"auto","created_at":"2026-03-16 08:29:24","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":21987,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-9030646/v1/be1cfc81678e8f88713c035b.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal Study","fulltext":[{"header":"Introduction","content":"\u003cp\u003eOral cancer remains an important global health problem and is among the most common malignancies worldwide. Despite advances in diagnosis and treatment, it continues to cause considerable morbidity and mortality, particularly when detected at advanced stages (1). Although early-stage detection is associated with survival rates exceeding 80%, delayed diagnosis significantly reduces prognosis and increases treatment-related morbidity (2-4). Despite advancements in screening and awareness programs, delayed presentation remains common, often due to the asymptomatic nature of early oral mucosal changes (5,6).\u003c/p\u003e\n\u003cp\u003eAt the same time, the rapid digitalization of healthcare has significantly changed the way patients search for medical information. Many individuals now consult online sources before seeking professional care, and more recently large language models (LLMs) have become part of this information-seeking process. This shift is particularly pronounced in oncology-related fears, where benign anatomical variations such as torus palatinus or frictional keratosis are frequently misinterpreted as malignancies. In societies with heightened cancer anxiety, digital tools now play a pivotal role in shaping early health perceptions and care-seeking behavior (7-9,16).\u003c/p\u003e\n\u003cp\u003eRecent studies have evaluated the quality of AI-generated medical responses; still, most investigations remain cross-sectional and model-specific, primarily focusing on ChatGPT-based systems (7-13). Comparative analyses between emerging AI architectures remain scarce, and longitudinal assessments of response stability are largely absent. In addition, existing literature predominantly evaluates English-language outputs, leaving a significant gap in understanding how LLMs perform in non-English populations\u0026mdash;particularly in languages with distinct linguistic structures such as Turkish.\u003c/p\u003e\n\u003cp\u003eBeyond scientific accuracy, patient safety in oncology communication depends on two critical dimensions: readability and referral safety. Readability determines whether information is accessible to the general population, while referral safety reflects whether the system appropriately directs patients to professional evaluation in potentially malignant scenarios. Failure in either domain may contribute to false reassurance or unnecessary anxiety.\u003c/p\u003e\n\u003cp\u003eFor this reason, the aim of the present longitudinal study was to investigate whether contemporary large language models are able to provide clinically safe and psychologically balanced guidance in oral cancer-related scenarios. Unlike prior studies, this research integrates:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eObjective Turkish-language readability metrics (Ateşman and Bezirci\u0026ndash;Yılmaz indices (23, 24)),\u003c/li\u003e\n \u003cli\u003eInter-model agreement analysis using Intraclass Correlation Coefficient,\u003c/li\u003e\n \u003cli\u003eTemporal reliability assessment using Cronbach\u0026rsquo;s alpha,\u003c/li\u003e\n \u003cli\u003eA multi-threshold clinical validity framework,\u003c/li\u003e\n \u003cli\u003eAnd referral safety evaluation as a patient-protection indicator.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBy combining linguistic analysis with clinical safety thresholds, this study aims to determine whether contemporary LLMs can provide stable, safe, and population-appropriate guidance in the context of oral oncology.\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cp\u003eThis study used a prospective longitudinal comparative design to assess the performance, stability, and clinical safety of two advanced large language models: Google Gemini (Pro version) and xAI Grok (Grok-1).\u003c/p\u003e\n\u003cp\u003eTo assess temporal consistency and response reproducibility, the evaluation was conducted over a continuous 7-day period. Each model was prompted independently and repeatedly using standardized patient scenarios.\u003c/p\u003e\n\u003cp\u003eA total of 20 standardized Turkish-language questions were developed to simulate real-world patient concerns regarding oral cancer. The 20 standardized patient scenarios used in this study are presented in Table 1. The original questions were developed in Turkish.\u003c/p\u003e\n\u003cp\u003eThe scenarios included:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eBenign anatomical variations (e.g., torus palatinus, linea alba)\u003c/li\u003e\n \u003cli\u003eChronic trauma-related lesions\u003c/li\u003e\n \u003cli\u003ePotentially malignant symptoms (non-healing ulcer, unexplained bleeding)\u003c/li\u003e\n \u003cli\u003eSystemic signs (lymphadenopathy, unexplained mobility)\u003c/li\u003e\n \u003cli\u003eCommon oncological myths (e.g., \u0026ldquo;Does biopsy spread cancer?\u0026rdquo;)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe questions were reviewed and validated by two independent oral and maxillofacial radiologists to ensure clinical relevance and construct validity.\u0026nbsp;\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEach question was submitted to both Gemini and Grok:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eOnce daily for 7 consecutive days\u003c/li\u003e\n \u003cli\u003eIn newly initiated chat sessions\u003c/li\u003e\n \u003cli\u003eWith chat history cleared before each interaction\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis approach minimized contextual memory bias and ensured independent response generation.\u003c/p\u003e\n\u003cp\u003eAll responses were archived in a structured database for blinded evaluation.\u003c/p\u003e\n\u003cp\u003eScientific accuracy was assessed by two expert reviewers using a 5-point Likert scale. In cases of discrepancy, consensus was achieved through joint discussion before final score assignment. The consensus score was used for all statistical analyses.\u003c/p\u003e\n\u003cp\u003e1= Very poor\u003cbr\u003e\u0026nbsp;2 = Poor\u003cbr\u003e\u0026nbsp;3 = Acceptable\u003cbr\u003e\u0026nbsp;4 = Good\u003cbr\u003e\u0026nbsp;5 = Excellent\u003c/p\u003e\n\u003cp\u003eAlthough a 5-point Likert scale was used, no response received a score of 5 (\u0026ldquo;excellent\u0026rdquo;) in our dataset; observed scores ranged from 1 to 4. Disagreements were resolved by consensus through joint discussion, and consensus scores were used for statistical analyses.\u003c/p\u003e\n\u003cp\u003eEvaluation criteria included:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eAlignment with current oral oncology guidelines\u003c/li\u003e\n \u003cli\u003eAbsence of misinformation\u003c/li\u003e\n \u003cli\u003eCorrect interpretation of symptoms\u003c/li\u003e\n \u003cli\u003eClinical appropriateness of explanations\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eCompleteness was evaluated using the same 5-point Likert scale.\u003c/p\u003e\n\u003cp\u003eAssessment focused on:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eInclusion of differential diagnoses\u003c/li\u003e\n \u003cli\u003eMention of red-flag signs\u003c/li\u003e\n \u003cli\u003eDiscussion of risk factors\u003c/li\u003e\n \u003cli\u003eAdequacy of patient counseling information\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUnlike subjective readability scoring, linguistic complexity was measured using validated Turkish readability formulas:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAteşman Readability Formula\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA Turkish adaptation of the Flesch Reading Ease score (23).\u003c/p\u003e\n\u003cp\u003eFormula:\u003cbr\u003e\u0026nbsp;198.825 \u0026minus; (40.175 \u0026times; average syllables per word) \u0026minus; (2.610 \u0026times; average sentence length)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBezirci\u0026ndash;Yılmaz Readability Index\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA syllable-based Turkish readability metric estimating educational level (24).\u003c/p\u003e\n\u003cp\u003eFor each response, the following linguistic parameters were automatically computed:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eWord count\u003c/li\u003e\n \u003cli\u003eSentence count\u003c/li\u003e\n \u003cli\u003eMean sentence length\u003c/li\u003e\n \u003cli\u003eAverage syllables per word\u003c/li\u003e\n \u003cli\u003eAteşman score\u003c/li\u003e\n \u003cli\u003eBezirci\u0026ndash;Yılmaz index\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis approach ensured objective and reproducible readability assessment.\u003c/p\u003e\n\u003cp\u003eReferral safety was evaluated as a binary variable (Yes/No).\u003c/p\u003e\n\u003cp\u003eA response was considered \u0026ldquo;Referral Safe\u0026rdquo; if it explicitly recommended professional clinical evaluation in scenarios suggestive of potentially malignant or serious conditions.\u003c/p\u003e\n\u003cp\u003eAll statistical analyses were performed using SPSS 22 software (IBM Corporation, Armonk, NY, USA).\u003c/p\u003e\n\u003cp\u003eTemporal stability was assessed using Cronbach\u0026rsquo;s alpha (\u0026alpha;).\u003c/p\u003e\n\u003cp\u003eInternal consistency across the 7-day period was calculated separately for each model.\u003c/p\u003e\n\u003cp\u003eAgreement between Gemini and Grok was evaluated using the ICC.\u003c/p\u003e\n\u003cp\u003eA two-way mixed-effects model with absolute agreement ICC(2,1) was applied.\u003c/p\u003e\n\u003cp\u003eThe Friedman test was used to assess score variation over time.\u003c/p\u003e\n\u003cp\u003eThe Wilcoxon signed-rank test was applied to compare Likert-based domains between models.\u003c/p\u003e\n\u003cp\u003eMean readability metrics were compared descriptively and inferentially between models.\u003c/p\u003e\n\u003cp\u003eObjective readability metrics between Gemini and Grok were compared using independent samples t-tests following normality assessment with the Shapiro\u0026ndash;Wilk test.\u003c/p\u003e\n\u003cp\u003eMean sentence length, Ateşman score, and Bezirci\u0026ndash;Yılmaz index were analyzed separately (23, 24). Statistical significance was set at p \u0026lt; 0.05.\u003c/p\u003e\n\u003cp\u003eClinical utility was assessed at three predefined thresholds:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eLow validity: \u0026ge;2 points\u003c/li\u003e\n \u003cli\u003eModerate validity: \u0026ge;3 points\u003c/li\u003e\n \u003cli\u003eHigh validity: \u0026ge;4 points\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eA p-value \u0026lt; 0.05 was considered statistically significant.\u003c/p\u003e"},{"header":"Results","content":"\u003ch2\u003eDescriptive Overview\u003c/h2\u003e\n\u003cp\u003eA total of 20 standardized patient scenarios were submitted daily to each AI system over a 7-day period, yielding 140 responses per model and 280 responses overall.\u003c/p\u003e\n\u003ch2\u003eObjective Readability\u003c/h2\u003e\n\u003cp\u003eOverall readability metrics are presented in Table 2.\u003c/p\u003e\n\u003cp\u003eMean sentence length was significantly higher in Grok responses (14.83 \u0026plusmn; 1.16) compared to Gemini (12.61 \u0026plusmn; 0.49) (p = 0.0005).\u003c/p\u003e\n\u003cp\u003eNo statistically significant differences were observed between models in Ateşman scores (59.23 \u0026plusmn; 1.33 vs. 57.67 \u0026plusmn; 3.16; p = 0.254) or Bezirci\u0026ndash;Yılmaz index values (9.23 \u0026plusmn; 0.38 vs. 9.15 \u0026plusmn; 0.88; p = 0.814).\u003c/p\u003e\n\u003ch2\u003eScientific Accuracy\u003c/h2\u003e\n\u003cp\u003eThe overall mean scientific accuracy score was 3.52 \u0026plusmn; 0.57 for Gemini and 3.39 \u0026plusmn; 0.68 for Grok (Table 3).\u003c/p\u003e\n\u003cp\u003eThe difference between models was not statistically significant (Wilcoxon signed-rank test, p = 0.072).\u003c/p\u003e\n\u003ch2\u003eCompleteness\u003c/h2\u003e\n\u003cp\u003eThe overall completeness score was 3.40 \u0026plusmn; 0.70 for Gemini and 3.25 \u0026plusmn; 0.78 for Grok (Table 3).\u003c/p\u003e\n\u003cp\u003eNo statistically significant difference was observed between models (Wilcoxon signed-rank test, p = 0.091).\u003c/p\u003e\n\u003ch2\u003eReliability\u003c/h2\u003e\n\u003cp\u003eTemporal reliability analysis demonstrated high internal consistency for both models (Gemini \u0026alpha; = 0.942; Grok \u0026alpha; = 0.886) (Table 4).\u003c/p\u003e\n\u003cp\u003eInter-model agreement was evaluated using a two-way mixed-effects absolute agreement model ICC(2,1). ICC values were interpreted according to the criteria proposed by Koo and Li (14).\u003c/p\u003e\n\u003cp\u003eOverall ICC values were:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eScientific accuracy: 0.58 (95% CI: 0.21\u0026ndash;0.81)\u003c/li\u003e\n \u003cli\u003eCompleteness: 0.50 (95% CI: 0.09\u0026ndash;0.77)\u003c/li\u003e\n \u003cli\u003eReadability: 0.44 (95% CI: 0.00\u0026ndash;0.73)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese findings indicate moderate agreement for accuracy and completeness, and low-to-moderate agreement for readability (Table 5).\u003c/p\u003e\n\u003ch2\u003eReferral Safety\u003c/h2\u003e\n\u003cp\u003eReferral-safe responses were observed in 90.0% of Gemini outputs and 92.1% of Grok outputs (Table 6).\u003c/p\u003e\n\u003cp\u003eThe difference between models was not statistically significant (p = 0.536).\u003c/p\u003e\n\u003ch2\u003eThreshold Analysis\u003c/h2\u003e\n\u003cp\u003eAt the \u0026ge;2 threshold, 100% of Gemini and 99.3% of Grok responses met basic validity criteria.\u003c/p\u003e\n\u003cp\u003eAt the \u0026ge;3 threshold, 89.2% of Gemini and 84.5% of Grok responses met moderate validity criteria.\u003c/p\u003e\n\u003cp\u003eAt the \u0026ge;4 threshold, 55.4% of Gemini and 48.2% of Grok responses achieved high validity (Table 7).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe growing use of large language models in healthcare communication has created both opportunities and uncertainties, particularly in the context of oncology-related patient education\u0026nbsp;(8, 15). As individuals increasingly rely on AI systems for symptom interpretation and risk assessment, the potential for both beneficial triage support and unintended anxiety amplification warrants systematic evaluation (10, 21, 22). Despite growing literature on AI performance in medicine, evidence regarding longitudinal stability and referral safety in oral oncology remains limited.\u003c/p\u003e\n\u003cp\u003eIn oral oncology, these issues are especially important. Delayed diagnosis remains one of the most critical determinants of poor prognosis, especially in cases presenting with subtle or initially asymptomatic mucosal changes (1-3). Digital systems that underestimate persistent ulcers, unexplained bleeding, or lymphadenopathy could inadvertently contribute to diagnostic delay. Conversely, overly alarming communication may intensify cancer-related anxiety and influence health-seeking behavior in unpredictable ways. Despite increasing evaluation of AI systems in general medical domains, longitudinal data examining clinical safety, referral behavior, and linguistic balance in oral cancer contexts remain limited.\u003c/p\u003e\n\u003cp\u003eIn the present study, we systematically evaluated whether contemporary LLMs can provide clinically safe and psychologically balanced guidance in oral cancer\u0026ndash;related patient scenarios. By integrating scientific accuracy, completeness, referral safety, Turkish readability metrics, and inter-model agreement across a 7-day period, this study offers a multidimensional evaluation of AI performance in oral oncology communication.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Accuracy and Referral Safety\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBoth Gemini and Grok demonstrated moderate-to-high scientific accuracy and completeness, with no statistically significant differences between models. These findings are consistent with recent oncology-focused evaluations of AI chatbots, which report generally acceptable factual alignment but variability in depth and framing (10, 15).\u003c/p\u003e\n\u003cp\u003eImportantly, referral-safe responses were observed in more than 90% of high-risk scenarios in both systems. In oral oncology, delayed diagnosis remains a critical contributor to poor prognosis (1-3). Digital tools that incorrectly minimize red-flag symptoms such as persistent ulcers or unexplained bleeding could potentially delay care. The high referral rates observed in this study suggest that both models adopt a precautionary stance rather than providing false reassurance, which is a clinically desirable pattern.\u003c/p\u003e\n\u003cp\u003eHowever, inter-model agreement for scientific accuracy (ICC = 0.58) and completeness (ICC = 0.50) was only moderate. This indicates that while both systems generally provide acceptable information, the structure, emphasis, and explanatory depth vary between models. Such variability may influence patient interpretation and perceived urgency.\u003c/p\u003e\n\u003cp\u003eBeyond accuracy alone, another critical dimension of AI-based medical communication is the reliability of information provided to patients outside the clinical setting. Previous studies have shown that individuals increasingly rely on online platforms and AI-driven tools when interpreting symptoms or seeking preliminary health advice (21,22). In oncology-related contexts, this behavior can influence the timing of professional consultation and may shape patient perceptions of disease severity. Recent investigations evaluating AI chatbots in cancer-related queries have demonstrated that these systems are often able to provide broadly accurate medical information, although variability in response depth and contextual interpretation remains a concern (10,15). In line with these findings, the present study observed generally acceptable levels of scientific accuracy and completeness in responses generated by both LLMs. However, moderate inter-model agreement suggests that different AI architectures may frame similar clinical scenarios in distinct ways, which could potentially influence patient interpretation and decision-making.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReadability and Patient Comprehension\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOne strength of this study is the use of validated Turkish readability indices\u0026nbsp;(Ateşman and Bezirci\u0026ndash;Yılmaz), addressing a gap in non-English AI evaluation (18,20, 24, 25). While overall readability scores were comparable between models, Grok generated significantly longer sentences.\u003c/p\u003e\n\u003cp\u003eIn health communication, linguistic complexity is associated with increased cognitive load and potential misunderstanding, particularly in populations with limited health literacy (19). In the context of cancer-related concerns, even subtle differences in sentence structure may affect how patients perceive severity and risk. For this reason, readability assessment should extend beyond formula-based scores and consider structural characteristics that influence comprehension.\u003c/p\u003e\n\u003cp\u003eAnother dimension that deserves attention is the potential impact of AI-generated information on patient behavior and emotional response. Online health information has been shown to significantly affect health-related decision making, particularly among individuals experiencing uncertainty or anxiety about possible disease symptoms (21). In oncology contexts, ambiguous or overly complex information may increase anxiety, while overly reassuring responses may delay professional evaluation. The findings of the present study suggest that both LLMs generally adopted a precautionary communication style by recommending professional consultation in the majority of high-risk scenarios. Such referral-oriented responses may help encourage timely clinical assessment, which is essential in conditions such as oral cancer where early diagnosis substantially improves prognosis (1-3). Nevertheless, the variability observed in linguistic structure and explanatory detail highlights the importance of continued clinical oversight and\u0026nbsp;further research exploring how patients interpret and act upon AI-generated medical guidance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAnxiety Amplification Versus Clinical Caution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study also explored whether AI-generated responses might contribute to unnecessary anxiety or, conversely, provide inappropriate reassurance. Although patient anxiety was not directly measured, referral behavior and linguistic features provide indirect insight.\u003c/p\u003e\n\u003cp\u003eThe high referral rates indicate that both models avoid minimizing potentially malignant symptoms, which is essential for patient safety. Nevertheless, variability in framing and sentence complexity may shape emotional interpretation. Prior research has shown that online health information can influence anxiety levels depending on tone and clarity (21,22).\u003c/p\u003e\n\u003cp\u003eAlthough current LLMs appear to adopt a cautious approach rather than providing false reassurance, the psychological impact of repeated AI exposure in oncology-related contexts still requires further investigation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTemporal Stability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUnlike many cross-sectional AI evaluations, this study incorporated a 7-day longitudinal design. Cronbach\u0026rsquo;s alpha values demonstrated high internal consistency for both models (Gemini \u0026alpha; = 0.942; Grok \u0026alpha; = 0.886), indicating stable performance over time. This is particularly relevant, as patients may consult AI tools repeatedly before seeking professional care.\u003c/p\u003e\n\u003cp\u003eNevertheless, moderate ICC values demonstrate that consistency within a model does not equate to agreement between models, underscoring the absence of standardized oncologic reasoning across AI architectures.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Implications\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFrom an oral oncology perspective, early recognition and timely referral remain essential to improving outcomes (1-3). The findings suggest that contemporary LLMs may serve as informational adjuncts that encourage professional evaluation rather than replace it (10, 15, 17).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eHowever, clinicians should remain aware that AI-generated information varies in framing and depth. While LLMs may support awareness and triage, they should not substitute clinical examination, biopsy, or specialist assessment in suspected malignancies.\u003c/p\u003e\n\u003cp\u003eAs AI systems become increasingly integrated into patient decision-making pathways, ensuring that responses remain both clinically safe and psychologically balanced will be critical.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLimitations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study evaluated only two LLM systems and one language (Turkish), which may limit generalizability. Anxiety amplification was inferred indirectly rather than measured using validated psychological instruments. Future research should incorporate patient-reported outcomes and qualitative assessment to better understand emotional impact. Additionally, real patient behavioral outcomes were not evaluated.\u0026nbsp;\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eContemporary large language models showed moderate-to-high scientific accuracy and generally strong referral safety. They appear to adopt a precautionary communication style rather than providing potentially misleading reassurance. However, inter-model variability and linguistic differences highlight the need for clinician oversight and continued patient-centered research as AI tools become more prominent in oncology-related health communication. Clinicians should remain aware that while LLMs may encourage appropriate referrals, they should not replace professional evaluation in suspected oral malignancies.\u003c/p\u003e\n"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBYK: Conceptualization, Methodology, Investigation, Writing \u0026ndash; Original Draft, Writing \u0026ndash; Review \u0026amp; Editing.\u003c/p\u003e\n\u003cp\u003eTC: Conceptualization, Methodology, Investigation, Writing \u0026ndash; Original Draft, Writing \u0026ndash; Review \u0026amp; Editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe author(s) received no financial support for the research, authorship, and/or publication of this article.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics Approval\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eand consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study did not involve human participants, patient data, or clinical records. The analysis was conducted solely on AI-generated responses. Therefore, institutional ethics committee approval was not required in accordance with local regulations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe author(s) declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical trial number:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eSung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(3):209-249. doi:10.3322/caac.21660.\u003c/li\u003e\n\u003cli\u003eAbati S, Bramati C, Bondi S, Lissoni A, Trimarchi M. Oral cancer and precancer: a narrative review on the relevance of early diagnosis. Int J Environ Res Public Health. 2020;17(24):9160. doi:10.3390/ijerph17249160.\u003c/li\u003e\n\u003cli\u003eDhanuthai K, Rojanawatsirivej S, Thosaporn W, et al. Oral cancer: a multicenter study. Med Oral Patol Oral Cir Bucal. 2018;23(1):e23. doi:10.4317/medoral.21999.\u003c/li\u003e\n\u003cli\u003e\u0026Uuml;nal Erzurumlu Z, G\u0026ouml;kkurt Yılmaz BN, Şahiner SP, Sadık E. Bir grup hastanın ağız kanseri ve risk fakt\u0026ouml;rlerinin farkındalığının değerlendirilmesi. Selcuk Dent J. 2025;12(3):372-376.\u003c/li\u003e\n\u003cli\u003eAlqahtani WS, Almufareh NA, Al-Johani HA, Alotaibi RK, Juliana CI, Aljarba NH, et al. Oral and oropharyngeal cancers and possible risk factors across Gulf Cooperation Council countries: a systematic review. World J Oncol. 2020;11(4):173-181.\u003c/li\u003e\n\u003cli\u003eKeser G, \u0026Ouml;zt\u0026uuml;rk M, Namdar Pekiner F. Assessment of awareness and knowledge of oral cancer among tobacco-using dental patients. Clin Exp Health Sci. 2021;11(2):279-284.\u003c/li\u003e\n\u003cli\u003eDergaa I, Fekih-Romdhane F, Hallit S, Loch AA, Glenn JM, Fessi MS, et al. ChatGPT is not ready yet for use in providing mental health assessment and interventions. Front Psychiatry. 2024;14:1277756.\u003c/li\u003e\n\u003cli\u003eSun G, Zhou YH. AI in healthcare: navigating opportunities and challenges in digital communication. Front Digit Health. 2023;5:1291132.\u003c/li\u003e\n\u003cli\u003eDaniel T, de Chevigny A, Champrigaud A, Valette J, Sitbon M, Jardin M, et al. Answering hospital caregivers\u0026rsquo; questions at any time: proof-of-concept study of an artificial intelligence\u0026ndash;based chatbot in a French hospital. JMIR Hum Factors. 2022;9(4):e39102.\u003c/li\u003e\n\u003cli\u003ePan A, Musheyev D, Bockelman D, Loeb S, Kabarriti AE. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol. 2023;9(10):1437-1440. doi:10.1001/jamaoncol.2023.2947.\u003c/li\u003e\n\u003cli\u003eErden Y, Temel MH, Bağcıer F. Artificial intelligence insights into osteoporosis: assessing ChatGPT\u0026rsquo;s information quality and readability. Arch Osteoporos. 2024;19(1):17.\u003c/li\u003e\n\u003cli\u003eNian PP, Saleet J, Magruder M, Wellington IJ, Choueka J, Houten JK, et al. ChatGPT as a source of patient information for lumbar spinal fusion and laminectomy: a comparative analysis against Google web search. Clin Spine Surg. 2024;37(10):E394-E403.\u003c/li\u003e\n\u003cli\u003eMusheyev D, Pan A, Kabarriti AE, Loeb S, Borin JF. Quality of information about kidney stones from artificial intelligence chatbots. J Endourol. 2024;38(10):1056-1061.\u003c/li\u003e\n\u003cli\u003eKoo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012.\u003c/li\u003e\n\u003cli\u003eChen D, Parsa R, Swanson K, Nunez JJ, Critch A, Bitterman DS, et al. Large language models in oncology: a review. BMJ Oncol. 2025;4(1):e000759. doi:10.1136/bmjonc-2025-000759.\u003c/li\u003e\n\u003cli\u003eTeplinsky E, Ponce SB, Drake EK, Garcia AM, Loeb S, van Londen GJ, et al. Online medical misinformation in cancer: distinguishing fact from fiction. JCO Oncol Pract. 2022;18(8):584-589. doi:10.1200/OP.21.00764.\u003c/li\u003e\n\u003cli\u003eTopol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44-56. doi:10.1038/s41591-018-0300-7.\u003c/li\u003e\n\u003cli\u003eAbreu AA, Murimwa GZ, Farah E, Stewart JW, Zhang L, Rodriguez J, et al. Enhancing readability of online patient-facing content: the role of AI chatbots in improving cancer information accessibility. J Natl Compr Canc Netw. 2024;22(2):e237334. doi:10.6004/jnccn.2023.7334.\u003c/li\u003e\n\u003cli\u003evan Merri\u0026euml;nboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44(1):85-93. doi:10.1111/j.1365-2923.2009.03498.x.\u003c/li\u003e\n\u003cli\u003eSzab\u0026oacute; P, B\u0026iacute;r\u0026oacute; \u0026Eacute;, K\u0026oacute;sa K. Readability and comprehension of printed patient education materials. Front Public Health. 2021;9:725840. doi:10.3389/fpubh.2021.725840.\u003c/li\u003e\n\u003cli\u003eBujnowska-Fedak MM, Węgierek P. The impact of online health information on patient health behaviours and making decisions concerning health. Int J Environ Res Public Health. 2020;17(3):880. doi:10.3390/ijerph17030880.\u003c/li\u003e\n\u003cli\u003eLink E, Baumann E, Klimmt C. Explaining online information seeking behaviors in people with different health statuses: German representative cross-sectional survey. J Med Internet Res. 2021;23(12):e25963.\u003c/li\u003e\n\u003cli\u003eAteşman E. T\u0026uuml;rk\u0026ccedil;ede okunabilirliğin \u0026ouml;l\u0026ccedil;\u0026uuml;lmesi. Dil Dergisi. 1997;58:71-74.\u003c/li\u003e\n\u003cli\u003eBezirci B, Yılmaz AE. T\u0026uuml;rk\u0026ccedil;e i\u0026ccedil;in yeni bir okunabilirlik \u0026ouml;l\u0026ccedil;\u0026uuml;t\u0026uuml;: Bezirci-Yılmaz form\u0026uuml;l\u0026uuml;. Gazi Univ J Sci. 2010;23(3):353-358.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables are available in the Supplementary Files section.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-oral-health","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ohea","sideBox":"Learn more about [BMC Oral Health](http://bmcoralhealth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ohea/default.aspx","title":"BMC Oral Health","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"oral cancer, artificial intelligence, large language models, patient communication, readability","lastPublishedDoi":"10.21203/rs.3.rs-9030646/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9030646/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground: Oral cancer is still a major global health problem. It is among the most common malignancies worldwide and continues to cause significant morbidity and mortality. As patients increasingly consult large language models (LLMs) for health information prior to professional evaluation, assessing the clinical safety and reliability of AI-generated responses in oral oncology has become essential.\u003c/p\u003e\n\u003cp\u003eMethods: This prospective longitudinal comparative study evaluated two advanced LLMs (Google Gemini Pro and xAI Grok-1) over a 7-day period. Twenty standardized Turkish-language oral cancer–related patient scenarios were submitted daily to each model, yielding 280 total responses. Scientific accuracy and completeness were assessed using a 5-point Likert scale by two independent oral and maxillofacial radiologists. Objective readability was measured using validated Turkish formulas (Ateşman and Bezirci–Yılmaz). Referral safety was evaluated as a binary outcome. Temporal stability was assessed using Cronbach’s alpha, and inter-model agreement was analyzed using intraclass correlation coefficients (ICC(2,1)).\u003c/p\u003e\n\u003cp\u003eResults: Mean scientific accuracy scores were 3.52 ± 0.57 for Gemini and 3.39 ± 0.68 for Grok (p = 0.072). Completeness scores were 3.40 ± 0.70 and 3.25 ± 0.78, respectively (p = 0.091). Grok generated significantly longer sentences (14.83 ± 1.16 vs. 12.61 ± 0.49; p = 0.0005), although overall readability indices did not differ significantly. Referral-safe responses were observed in 90.0% of Gemini and 92.1% of Grok outputs (p = 0.536). Temporal reliability was high (Gemini α = 0.942; Grok α = 0.886). Inter-model agreement was moderate for scientific accuracy (ICC = 0.58) and completeness (ICC = 0.50).\u003c/p\u003e\n\u003cp\u003eConclusions: Contemporary LLMs demonstrated moderate-to-high scientific accuracy and strong referral safety in oral cancer–related scenarios. While they appear to favor clinical caution over false reassurance, variability in linguistic structure and inter-model agreement highlights the need for clinician oversight. LLMs may serve as informational adjuncts but should not replace professional evaluation in suspected oral malignancies.\u003c/p\u003e","manuscriptTitle":"Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-16 08:28:12","doi":"10.21203/rs.3.rs-9030646/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-07T09:57:55+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-28T11:38:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"252871235012440321136429867398928533108","date":"2026-03-23T18:54:18+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-18T07:06:22+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"72720429653564333707035186098898160963","date":"2026-03-15T14:07:05+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-13T12:43:20+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-05T11:57:01+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-05T06:29:12+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-05T06:27:32+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Oral Health","date":"2026-03-04T13:01:39+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-oral-health","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ohea","sideBox":"Learn more about [BMC Oral Health](http://bmcoralhealth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ohea/default.aspx","title":"BMC Oral Health","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"56f96c76-5124-468c-8ff8-146516723ebc","owner":[],"postedDate":"March 16th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-29T09:39:51+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-16 08:28:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9030646","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9030646","identity":"rs-9030646","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}