Quality Assessment of Patient-Facing Urologic Telesurgery Content Using Validated Tools

Quality Assessment of Patient-Facing Urologic Telesurgery Content Using Validated Tools | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Quality Assessment of Patient-Facing Urologic Telesurgery Content Using Validated Tools Tarak Davuluri, Paul Gabriel, Matthew Wainstein, Obi Ekwenna This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7527866/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 14 Oct, 2025 Read the published version in Journal of Robotic Surgery → Version 1 posted 10 You are reading this latest preprint version Abstract Introduction : With increasing accessibility to Artificial Intelligence (AI) chatbots, the precision and clarity of medical information provided requires rigorous assessment. Urologic telesurgery represents a complex concept that patients will investigate using AI. We compared ChatGPT and Google Gemini in providing patient-facing information on urologic telesurgical procedures. Methods : 19 questions related to urologic telesurgery were generated using general information from the American Urologic Association (AUA) and European Robotic Urology Section (ERUS). Questions were organized into 4 categories (Prospective, Technical, Recovery, Other) and directly typed into ChatGPT 4o and Google Gemini 2.5 (non-paid versions). For each question, a new chat was started to prevent any continuation of answers. Three reviewers independently reviewed the responses using two validated healthcare tools: DISCERN (quality) and Patient Education Material Assessment Tool (understandability and actionability). Results : Mean DISCERN scores (out of 80) were higher for Gemini than ChatGPT in all domains except “Other”. Prospective 49.2 vs 39.1; technical 52.3 vs 44.3; recovery 53.7 vs 45.4; other 54.3 vs 56.5; overall 52.4 vs 45.8 (Figure 1). PEMAT-P understandability uniformly exceeded 70% for both platforms: prospective 80.0% vs 71.7%; technical 80.1% vs 79.8%; recovery 79.2% vs 80.1%; other 79.2% vs 81.3%; overall 79.7% vs 78.1% (Figure 2). Actionability was uniformly low; only Gemini met 70% threshold in the prospective domain (Figure 3). Conclusion : ChatGPT and Gemini deliver relevant and understandable information related to urologic telesurgery, with Gemini more consistently providing sources. However, neither chatbot reliably offers actionable responses, limiting their utility as a standalone gateway for patient decision-making. AI Chatbots Urologic Telesurgery DISCERN PEMAT Figures Figure 1 Figure 2 Figure 3 Introduction Traditionally, surgery has been performed with the surgeon and their team physically present in the operating room. Advancements in technology have led to the rise of laparoscopic surgery, followed by robotic-assisted procedures, and most recently, telesurgery. Telesurgery is defined as a surgeon performing robotic procedures using a robotic system that is controlled remotely, often in a different city or country [ 1 ]. With continuous advancements in communication networks and robotic technology, the increasing success of robotic-assisted surgery in urology has laid the foundation for urologic telesurgery. A 2021 study, lasting 8 months, measured the feasibility of urologic telesurgery and tracked 29 successful telesurgical radical nephrectomies with a median distance of 116 miles between the patient and surgeon [ 2 ]. This study, among others [ 3 – 5 ], shows the increasing success and new innovations of telesurgery, specifically within the field of urology. As the popularity of telesurgery grows, patients are more likely to seek information due to the complexity of the concept. A 2018–2020 study found that 70–81% of Americans have looked up medical and health information online [ 6 ]. In line with the rising prevalence of AI chatbots, a cross-sectional study of 697 participants found that 78% of the respondents stated they would be willing to use ChatGPT for self-diagnosis [ 7 ]. ChatGPT and Google Gemini are two of the leading AI chatbot sources, with 600 and 350 million monthly active users, respectively [ 8 ]. With the very prevalent usage of these AI chatbots coupled with the growing use of telesurgical procedures, it is reasonable to assume that prospective patients may turn to AI chatbots for medical information. As a result, the precision and clarity of the medical information provided requires rigorous assessments. There are many studies that have evaluated the medical information provided by AI chatbots in general and regarding more specific topics, such as head and neck cancer treatment [ 9 – 11 ]. However, there are no studies specifically evaluating the information provided by AI chatbots in reference to urologic telesurgery. This study aims to analyze and compare the quality information provided by ChatGPT and Google Gemini regarding urologic telesurgery. Methods Search Strategy and Eligibility Screening: A set of 19 questions related to urologic telesurgery were generated using general information from the American Urologic Association (AUA) and European Robotic Urology Section (ERUS). The questions were formatted to best mimic the language and vocabular of an actual patient, and were then placed in one of 4 categories: Prospective, Technical, Recovery/Complications/Risks (labeled Recovery), Other. “Prospective” questions were about patient eligibility and participation (e.g., Am I a candidate for telesurgery for my urologic condition?). “Technical questions” were about details of the procedure (e.g., What happens if there is a technical problem during telesurgery?). “Recovery” questions were about post-procedures workings (e.g., Is pain better managed after robotic telesurgery?). “Other” questions were considered miscellaneous and referred to questions that did not fit into any of the other categories (e.g., What if I change my mind and want traditional surgery instead?). The questions were then typed directly into ChatGPT (OpenAI, San Francisco, CA) and Bard (Google, Mountain View, CA) [ 12 , 13 ]. ChatGPT 4o and Gemini 2.5 Flash were used. ChatGPT and Gemini are known to tailor their responses based on your conversation with them to provide more congruent answers. To avoid this confounding variable, for each question, a new conversation was generated to best replicate the experience a patient would have when they ask either of these chatbots a question. Responses were excluded if they were duplicates or were irrelevant to urologic telesurgery within reason. The remaining responses were considered for evaluation and scoring. Response Review: Each individual response was evaluated by three medical students (TD, PG, and MW). Responses were evaluated using the DISCERN and the Patient Education Materials Assessment Tool for Printed materials (PEMAT-P) tools. Both tools are validated for evaluating health information. DISCERN is a tool used to assess the quality of health information, with studies showing that both untrained individuals and healthcare professionals, alike, can effectively apply DISCERN to identify biases, gaps, and inaccuracies in medical information [ 14 , 15 ]. The DISCERN survey includes 16 questions. The first 8 evaluate the reliability and sourcing of the information provided, while the next 7 focus on the clarity and completeness of details regarding treatment options. The final question asks the rater to give an overall assessment of the material’s quality. Each item is scored on a scale from 1 (definite no) to 5 (definite yes), with intermediate scores (2–4) reflecting partial fulfillment of the criteria. Individual question score interpretation represented as follows: 4.01–5 = excellent, 3.01–4 = good, 2.01–3 = fair, 1.01–2 = poor, and 0–1 = very poor. Category scores are then summed, producing a total score that ranges from 16 to 80. Total scores are interpreted as follows: 68–80 = excellent quality, 55–67 = good quality, 42–54 = fair quality, 31–41 = poor quality, and below 30 = very poor quality. While DISCERN evaluated the quality of the information provided, PEMAT-P is a standardized tool that focuses on the actionability and understandability of health information [ 16 , 17 ]. PEMAT-P is comprised of 24 questions, with the first 17 measuring understandability and the remaining 7 measuring actionability [ 16 , 17 ]. Of the 24 questions, 7 were excluded as they were not applicable to the method we were using. Of the 7 excluded questions, 5 were excluded due to their emphasis on visual aids, 1 was excluded due to limited need for calculations, and the final one was excluded as it referred to word counts typically associated with research papers and not AI generated responses. Results General Characteristics: All 38 responses generated by both chatbots were deemed to be relevant to urologic telesurgery and were included. The average word count for responses were 353 and 579 words for ChatGPT and Gemini, respectively. Gemini more consistently provided sources for its answers (16/19 responses), while ChatGPT was more limited (4/19 responses). Common sources included the NIH, Johns Hopkins, and Mayo Clinic. Notably, neither chatbot utilized visual aids for as on average, only 3.51% of responses provided a visual aid for information clarity (Figure 2). Discern: Figure 1 represents the mean total DISCERN scores (out of 80) along with standard deviations for each category (Prospective, Technical, Recovery, Other, and Overall). No significant differences were noted between ChatGPT and Gemini scores, however, across all categories except “Other”, Gemini outperformed ChatGPT. DISCERN contained 16 questions with the average score for each question represented in Figure 1. ChatGPT had 4 excellent, 4 good, 4 fair, and 4 poor average scores (Figure 1). Gemini had 4 excellent, 6 good, 4 fair, and 2 poor scores (Figure 1). The lowest performing question for both chatbots was “Does it describe what happens if no treatment is used?” ChatGPT scored an average of 1.26 on this question, while Gemini scored a 1.14. ChatGPT had a high score a 4.35 for two questions: “Is it relevant?” and “Is it balanced and unbiased?” Gemini’s highest scoring question was in “Is it balanced and unbiased?”, scoring 4.58. Regarding cumulative scores, ChatGPT’s “Prospective” score was 39 (poor) and its “Other” score was 57 (good). Otherwise, the remaining 8 cumulative ChatGPT and Gemini DISCERN scores fell into the fair category. PEMAT-P Understandability and Actionability: Figure 2 represents the mean total PEMAT-P understandability scores for each category along with the mean score for each PEMAT-P understandability question. All categories met the 70% threshold to be considered understandable. The lowest performing PEMAT-P question was “The material used visual aids whenever they could make content more easily understood” with both chatbots averaging a 3.51% for this question. No significant differences were noted by the two chatbots overall, but Gemini did perform better in “Prospective” questions compared to ChatGPT (80% vs 72%). Both chatbots scored 100% in many categories including breaking up the information into sections, using concise language, and using an active voice among others. The information provided by both Chatbots was relevant and understandable. Figure 3 represents the mean total PEMAT-P actionability scores for each category along with the mean score for each PEMAT-P actionability question. Performance here was uniformly bad among the chatbots, with only Gemini’s score in the “Prospective” question category meeting the 70% threshold of being actionable. The worse scoring PEMAT-P actionability question was “using visual aids” which they scored 3.51% and 0.00% in for ChatGPT and Gemini, respectively. While Gemini had the highest score in “Prospective” questions, it also scored a 0% in “Technical” questions. Limited scores in this category show that while the information provided by these chatbots has been deemed understandable and relevant due to previous scores, the actionability they provide remains limited. Discussion With the increasing popularity of AI, their integration into medicine continues to grow. Currently, AI is being used in a variety of medical settings, including helping take patient histories and assisting with administrative tasks [ 18 , 19 ]. As physician use of AI continues to grow and evolve, it is safe to assume that patients will also turn to AI. As a result, it is important to evaluate the information provided by AI, specifically for medical questions in which inappropriate information can have direct consequences upon the patient. Other papers have compared information provided by ChatGPT and Gemini regarding specific procedures, including laparoscopic donor nephrectomies [ 20 ]. However, to our knowledge, this is the first study to analyze and compare the quality of information provided by ChatGPT and Gemini regarding urologic telesurgery. Urologic telesurgery is a field that is going to grow with continuous advancements in technology. The idea of a surgeon being able to perform on a patient while not being in the same room, let alone country can be hard for patients to grasp. Hence, they may turn to AI chatbots for information. In our analysis, most of the cumulative scores for DISCERN fell into the fair category (6/8) with ChatGPT’s “Prospective” and “Other” questions landing in poor and good, respectively (Fig. 1 ). While there were no statistical differences among mean DISCERN scores and in individual questions, the one metric in which Gemini performed notably better than ChatGPT was in providing sources for information. Regarding this, ChatGPT scored a 1.71 on average while Gemini scored a 3.11. This is supported by a similar study looking at urogenital cancer information provided by AI chatbot which had Gemini outperforming ChatGPT in the same category 4 to 1, respectively [ 21 ]. However, while the sources provided differed between the two chatbots, the relevancy, clarity, and appropriate nature of information provided was similar (Fig. 1 ). All PEMAT-P mean understandability scores met the minimum 70% threshold for being deemed understandable (Fig. 2 ). The threshold of 70% was consistently met despite neither AI chatbot providing a visual aid more than a handful of times (3.51% for both) (Fig. 2 ). The lack of visual aids was counteracted by the consistent use of everyday language and breaking up of the material into sections for easier digestibility. While no statistical differences were present between ChatGPT and Gemini in terms of understandability, Gemini provided summaries for their answers more consistently, scoring 86% vs the 61.4% of ChatGPT (Fig. 2 ). This could contribute to the notably larger average word count provided by Gemini of 579 words vs ChatGPT’s 353 words. While our study holds this notable word count difference, other studies have conflicting results, having ChatGPT having higher word counts than Gemini [ 21 , 22 ]. One of these studies utilized ChatGPT v3.5, while the other utilized ChatGPT 4o, same as this study. While AI chatbot performance in for PEMAT-P understandability was sufficient, the opposite was found for PEMAT-P actionability. Only Gemini in the “Prospective” question category met the 70% minimum threshold for responses deemed “actionable” (Fig. 3 ). Otherwise, all mean responses did not meet the minimum 70% threshold. Due to the variability of AI chatbots, there are conflicting results on actionability. Some studies show similar results in which both AI chatbots scored uniformly poor on actionability [ 23 , 24 ], while others have great results in actionability, meeting the 70% threshold [ 25 , 26 ]. Some limitations within the actionability aspect of this study is that sometimes a question may not have a direct or clear actionable response. For example, if you were to ask an AI chatbot “what is urologic telesurgery”, it can provide a very relevant and understandable answer without necessarily providing any actionable information. However, all responses were still included in actionability grading as the chatbots would occasionally responds with phrases included “please consult with a urologist” or “check nearby hospitals and reference their websites”. One potential application of AI chatbots based on our findings, along with other studies would be the development of healthcare-specific AI chatbots to address common patient inquiries. Current AI chatbots use a general-purpose language model and adjusting that to a healthcare-specific language model can be more appropriate for healthcare. Patients, rightfully so, will have questions about prospective procedures and appointments, and their best way to get accurate information is to call their provider. However, there are studies showing that unexpected telephone calls can result in a significant, unpredictable demand on workload for nurses and physicians [ 27 , 28 ]. To help relieve burden on the nursing staff and improve ease of access to patients, AI chatbots could be a solution. These chatbots could be programmed with accurate, up-to-date information tailored to the clinic’s services, policies, and patient population, allowing patients to obtain answers to frequently asked questions without calling the office, saving time for both the nurses and the patient. However, these chatbots must have strict safeguards that encourages the patient to call the office with any additional questions or confusion. While members of the public have trust in AI chatbots, there are existing risk concerns [ 29 ], which could be addressed with specific tailoring of these chatbots to their respective clinics. There has been success in developing healthcare-specific large language models with Google’s Med-PaLM, and the University of Florida’s GatorTron. Med-PaLM, now Med-PaLM 2, was designed for consumer health queries and became the first large language model to achieve a passing score on United States Medical Licensing Exam questions [ 30 ]. GatorTron is being developed by the University of Florida and was built on a large database of de-identified clinical data. The strengths of this model is in its ability to answer clinical questions and generate synthetic clinical text [ 31 ]. While both large language models have shown remarkable success in the clinical setting their application is currently not available outside of research collaborations or institutional oversight. However, just the development of these models shows the practicality they may be able to have if accessible to the public. With appropriate safeguards and criteria meeting the clinical standard, if employed by clinics, healthcare-specific language models can be both beneficial to patients and their providers. This study evaluated the information provided by ChatGPT 4o and Gemini 2.5, however, upon conclusion of this paper, ChatGPT 5 was released on August 7th, 2025. Considering ChatGPT 5 being the “next step up from 4o,” another study should be performed comparing the medical information provided by GPT 5 against 4o and Gemini 2.5 to see if there is truly a significant difference. Conclusion AI chatbots, ChatGPT 4o and Gemini 2.5, deliver relevant and understandable information related to urologic telesurgery. Of note, Gemini more consistently provides sources for its responses compared to ChatGPT, while neither chatbot consistently provided visual aids other than an occasional table. Additionally, neither chatbot reliably offers actionable responses. Their poor performance amongst PEMAT-P actionability criteria limits their utility as a standalone gateway for patient decision-making. This highlights the important role physicians have and will continue have in patient care. Physicians should still be consulted as AI should not be used as a standalone measure for patients. Overall, the use of AI chatbots is effective to garner relevant information, but to acquire specific, actionable information, more detailed prompts/queries along with further discussion with their physician is required. Declarations - The authors declare that no funds, grants, or other support were received during the preparation of this manuscript. - The authors have no relevant financial or non-financial interests to disclose. - All authors contributed to the study conception and design. Material preparation and data collection were performed by Tarak Davuluri, Paul Gabriel, and Matthew Wainstein. Analysis performed by Tarak Davuluri. The first draft of the manuscript was written by Tarak Davuluri and all authors commented on previous versions of the manuscript. Obi Ekwenna led in conceptualization, methodology, supervision, and contributed thoroughly to editing and review process as well. All authors read and approved the final manuscript. Author Contribution All authors contributed to the study conception and design. Material preparation and data collection were performed by Tarak Davuluri, Paul Gabriel, and Matthew Wainstein. Analysis performed by Tarak Davuluri. Figures prepared by Tarak Davuluri. The first draft of the manuscript was written by Tarak Davuluri and all authors commented on previous versions of the manuscript. Obi Ekwenna led in conceptualization, methodology, supervision, and contributed thoroughly to editing and review process as well. All authors read and approved the final manuscript. References Motiwala, Z.Y., et al. (2025). Telesurgery: current status and strategies for latency reduction. J Robot Surg, 19(1), 153. https://doi.org/10.1007/s11701-025-02333-1 Li, J., et al. (2023). Application of Improved Robot-assisted Laparoscopic Telesurgery with 5G Technology in Urology. Eur Urol, 83(1), 41-44. https://doi.org/10.1016/j.eururo.2022.06.018 Aldousari, S., et al. (2025). The era of telesurgery: insights from ultra-long-distance Asia to Middle East human telesurgery robotic assisted radical prostatectomy. J Robot Surg, 19(1), 108. https://doi.org/10.1007/s11701-025-02274-9 Zhou, F., et al. (2025). Application of 5G Remote Robotic-assisted Laparoscopy in Urological Surgery: A Small Sample Analysis. Urology, 197, 110-114. https://doi.org/10.1016/j.urology.2024.11.019 Ferreira, S.V., et al. (2025). Feasibility and Initial Outcomes of Telesurgery in Urology: a Systematic Review of the Literature. Int Braz J Urol, 51(3). https://doi.org/10.1590/S1677-5538.IBJU.2024.0494 Alma Taya, D. and Y.C. Chuang. (2025). Internet use for health information, health service utilization, and quality of care in the U.S. BMC Health Serv Res, 25(1), 659. https://doi.org/10.1186/s12913-025-12807-5 Shahsavar, Y. and A. Choudhury. (2023). User Intentions to Use ChatGPT for Self-Diagnosis and Health-Related Purposes: Cross-sectional Survey Study. JMIR Hum Factors, 10, e47564. https://doi.org/10.2196/47564 Musumeci, N. Google’s Gemini usage is surging, but rivals still dominating. 2025; Available from: https://www.businessinsider.com/google-gemini-usage-surging-rivals-chatgpt-meta-dominating-2025-4. Kuscu, O., et al. (2023). Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front Oncol, 13, 1256459. https://doi.org/10.3389/fonc.2023.1256459 Johnson, D., et al. (2023). Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Res Sq. https://doi.org/10.21203/rs.3.rs-2566942/v1 Fattah, F.H., et al. (2025). Comparative analysis of ChatGPT and Gemini (Bard) in medical inquiry: a scoping review. Front Digit Health, 7, 1482712. https://doi.org/10.3389/fdgth.2025.1482712 OpenAi, ChatGPT (June 2024) [Large language model] . 2024, OpenAI. DeepMind, G., ChatGPT (June 2024) [Large language model] . 2024, Google. Charnock, D., et al. (1999). DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health, 53(2), 105-11. https://doi.org/10.1136/jech.53.2.105 Charnock, D. and S. Shepperd. (2004). Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res, 19(4), 440-6. https://doi.org/10.1093/her/cyg046 Shoemaker, S.J., M.S. Wolf, and C. Brach. (2014). Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns, 96(3), 395-403. https://doi.org/10.1016/j.pec.2014.05.027 Furukawa, E., et al. (2025). Evaluating Online and Offline Health Information With the Patient Education Materials Assessment Tool: Protocol for a Systematic Review. JMIR Res Protoc, 14, e63489. https://doi.org/10.2196/63489 Hindelang, M., S. Sitaru, and A. Zink. (2024). Transforming Health Care Through Chatbots for Medical History-Taking and Future Directions: Comprehensive Systematic Review. JMIR Med Inform, 12, e56628. https://doi.org/10.2196/56628 Qin, S., et al. (2024). ChatGPT and generative AI in urology and surgery-A narrative review. BJUI Compass, 5(9), 813-821. https://doi.org/10.1002/bco2.390 Matthew Wainstein, I.D., Stephen Hong, Mehdi Nayebpour, Naoru Koizumi, Obi Ekwenna. (2024). A Quality Analysis of Laparoscopic Donor Nephrectomy-related Information Disseminated by Artificial Intelligence Chatbots using Validated Tools. Journal of Community Medicine & Health Education, 14(03). Erkan, A., et al. (2024). Can Patients With Urogenital Cancer Rely on Artificial Intelligence Chatbots for Treatment Decisions? Clin Genitourin Cancer, 22(6), 102206. https://doi.org/10.1016/j.clgc.2024.102206 Ozcan, S.G.G. and M. Erkan. (2024). Reliability and quality of information provided by artificial intelligence chatbots on post-contrast acute kidney injury: an evaluation of diagnostic, preventive, and treatment guidance. Rev Assoc Med Bras (1992), 70(11), e20240891. https://doi.org/10.1590/1806-9282.20240891 Kolac, U.C., et al. (2025). Can popular AI large language models provide reliable answers to frequently asked questions about rotator cuff tears? JSES Int, 9(2), 390-397. https://doi.org/10.1016/j.jseint.2024.11.012 Delsoz, M., et al. (2025). Large Language Models: Pioneering New Educational Frontiers in Childhood Myopia. Ophthalmol Ther, 14(6), 1281-1295. https://doi.org/10.1007/s40123-025-01142-x Behers, B.J., et al. (2024). Assessing the Quality of Patient Education Materials on Cardiac Catheterization From Artificial Intelligence Chatbots: An Observational Cross-Sectional Study. Cureus, 16(9), e69996. https://doi.org/10.7759/cureus.69996 Ito, S., et al. (2025). Leveraging artificial intelligence chatbots for anemia prevention: A comparative study of ChatGPT-3.5, copilot, and Gemini outputs against Google Search results. PEC Innov, 6, 100390. https://doi.org/10.1016/j.pecinn.2025.100390 Burnet, E., et al. (2018). A prospective analysis of unplanned patient-initiated contacts in an adult cystic fibrosis centre. J Cyst Fibros, 17(5), 636-642. https://doi.org/10.1016/j.jcf.2018.04.006 Flannery, M., S.M. Phillips, and C.A. Lyons. (2009). Examining telephone calls in ambulatory oncology. J Oncol Pract, 5(2), 57-60. https://doi.org/10.1200/JOP.0922002 Chen, S.Y., H.Y. Kuo, and S.H. Chang. (2024). Perceptions of ChatGPT in healthcare: usefulness, trust, and risk. Front Public Health, 12, 1457131. https://doi.org/10.3389/fpubh.2024.1457131 Singhal, K., et al. (2025). Toward expert-level medical question answering with large language models. Nat Med, 31(3), 943-950. https://doi.org/10.1038/s41591-024-03423-7 Peng, C., et al. (2023). A study of generative large language model for medical research and healthcare. NPJ Digit Med, 6(1), 210. https://doi.org/10.1038/s41746-023-00958-w Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 14 Oct, 2025 Read the published version in Journal of Robotic Surgery → Version 1 posted Editorial decision: Revision requested 18 Sep, 2025 Reviews received at journal 17 Sep, 2025 Reviewers agreed at journal 17 Sep, 2025 Reviews received at journal 15 Sep, 2025 Reviewers agreed at journal 09 Sep, 2025 Reviewers agreed at journal 08 Sep, 2025 Reviewers invited by journal 06 Sep, 2025 Editor assigned by journal 06 Sep, 2025 Submission checks completed at journal 05 Sep, 2025 First submitted to journal 03 Sep, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7527866","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":512267582,"identity":"901addfa-950b-4a41-bab2-2af57632d63a","order_by":0,"name":"Tarak Davuluri","email":"","orcid":"","institution":"University of Toledo","correspondingAuthor":false,"prefix":"","firstName":"Tarak","middleName":"","lastName":"Davuluri","suffix":""},{"id":512267586,"identity":"3a587d9d-e12f-4b7f-ba21-9e6939812910","order_by":1,"name":"Paul Gabriel","email":"","orcid":"","institution":"University of Toledo","correspondingAuthor":false,"prefix":"","firstName":"Paul","middleName":"","lastName":"Gabriel","suffix":""},{"id":512267587,"identity":"6064bd48-69a7-4b40-8ff9-fba978080171","order_by":2,"name":"Matthew Wainstein","email":"","orcid":"","institution":"University of Toledo","correspondingAuthor":false,"prefix":"","firstName":"Matthew","middleName":"","lastName":"Wainstein","suffix":""},{"id":512267590,"identity":"299e51bb-58f9-4cbf-acd9-1cd8675b3680","order_by":3,"name":"Obi Ekwenna","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIiWNgGAWjYBACfgnGBhBdLwHh2yTuP8DcwMADYh/ArkVyBkRLAlRLWmIbAyN+LQY3IDRMy2EitNxubntcUMOQIDkj+RiQcT6xjb2x8cGbCgY5vhsJ2LXcOdhuPOMYQ520RFo6kHE7sY3nYLPhnDMMxpK4tNxIbJPmYWNIkJPOMQMygFokgCK8bQyJG/Bq+QfSkv8NyDgH0tL+G6ilHq8WoIIEaekcNiDjANgWZpCIAQ4tknMOArX0SSRIzn9mBmQkG7MB/SI554yE4cwzD7Bq4ZdufybN882mXuLMYRDDTo6NvfnghzcVNvJ8x7HbAgUSRIiMglEwCkbBKCAeAACetmDZMH13kwAAAABJRU5ErkJggg==","orcid":"","institution":"University of Toledo","correspondingAuthor":true,"prefix":"","firstName":"Obi","middleName":"","lastName":"Ekwenna","suffix":""}],"badges":[],"createdAt":"2025-09-03 14:08:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7527866/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7527866/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11701-025-02871-8","type":"published","date":"2025-10-14T15:57:56+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":91195815,"identity":"95d28bf3-fc4b-4f94-9558-3b837f59c427","added_by":"auto","created_at":"2025-09-12 14:59:21","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":168368,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eA)\u003c/strong\u003e Mean DISCERN scores (out of 5) with standard deviations for each response \u003cstrong\u003eB)\u003c/strong\u003e Mean total DISCERN scores (out of 80) among each question category and overall.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7527866/v1/d55e156d0b09ff1e244c768f.png"},{"id":91196881,"identity":"178e90d8-1770-435c-baac-52a77590b7e2","added_by":"auto","created_at":"2025-09-12 15:07:21","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":192963,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eA)\u003c/strong\u003e Mean PEMAT-P Understandability scores with standard deviations for each response \u003cstrong\u003eB)\u003c/strong\u003eMean PEMAT-P Understandability scores among each question category and overall (70% minimum threshold for responses to be deemed “understandable”).\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-7527866/v1/37c17d5c881d5d41dd34da52.png"},{"id":91195813,"identity":"d3fb2595-709f-45e0-b6c7-0f63b63818d0","added_by":"auto","created_at":"2025-09-12 14:59:21","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":132515,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eA)\u003c/strong\u003e Mean PEMAT-P Actionability scores with standard errors for each response \u003cstrong\u003eB)\u003c/strong\u003e Mean PEMAT-P Actionability scores among each question category and overall (70% is minimum threshold for responses to be deemed “actionable”).\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-7527866/v1/7d4beb79afc9ffd304cc4bba.png"},{"id":93956053,"identity":"1464bae5-42e3-4e83-81b2-fbd89d8314bf","added_by":"auto","created_at":"2025-10-20 16:09:49","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":797604,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7527866/v1/fb72a938-3256-4aa3-a3f3-26c1d4c4a8e5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Quality Assessment of Patient-Facing Urologic Telesurgery Content Using Validated Tools","fulltext":[{"header":"Introduction","content":"\u003cp\u003eTraditionally, surgery has been performed with the surgeon and their team physically present in the operating room. Advancements in technology have led to the rise of laparoscopic surgery, followed by robotic-assisted procedures, and most recently, telesurgery. Telesurgery is defined as a surgeon performing robotic procedures using a robotic system that is controlled remotely, often in a different city or country [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. With continuous advancements in communication networks and robotic technology, the increasing success of robotic-assisted surgery in urology has laid the foundation for urologic telesurgery. A 2021 study, lasting 8 months, measured the feasibility of urologic telesurgery and tracked 29 successful telesurgical radical nephrectomies with a median distance of 116 miles between the patient and surgeon [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. This study, among others [\u003cspan additionalcitationids=\"CR4\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], shows the increasing success and new innovations of telesurgery, specifically within the field of urology.\u003c/p\u003e\u003cp\u003eAs the popularity of telesurgery grows, patients are more likely to seek information due to the complexity of the concept. A 2018\u0026ndash;2020 study found that 70\u0026ndash;81% of Americans have looked up medical and health information online [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. In line with the rising prevalence of AI chatbots, a cross-sectional study of 697 participants found that 78% of the respondents stated they would be willing to use ChatGPT for self-diagnosis [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. ChatGPT and Google Gemini are two of the leading AI chatbot sources, with 600 and 350\u0026nbsp;million monthly active users, respectively [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eWith the very prevalent usage of these AI chatbots coupled with the growing use of telesurgical procedures, it is reasonable to assume that prospective patients may turn to AI chatbots for medical information. As a result, the precision and clarity of the medical information provided requires rigorous assessments. There are many studies that have evaluated the medical information provided by AI chatbots in general and regarding more specific topics, such as head and neck cancer treatment [\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. However, there are no studies specifically evaluating the information provided by AI chatbots in reference to urologic telesurgery. This study aims to analyze and compare the quality information provided by ChatGPT and Google Gemini regarding urologic telesurgery.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eSearch Strategy and Eligibility Screening:\u003c/h2\u003e\u003cp\u003eA set of 19 questions related to urologic telesurgery were generated using general information from the American Urologic Association (AUA) and European Robotic Urology Section (ERUS). The questions were formatted to best mimic the language and vocabular of an actual patient, and were then placed in one of 4 categories: Prospective, Technical, Recovery/Complications/Risks (labeled Recovery), Other. \u0026ldquo;Prospective\u0026rdquo; questions were about patient eligibility and participation (e.g., Am I a candidate for telesurgery for my urologic condition?). \u0026ldquo;Technical questions\u0026rdquo; were about details of the procedure (e.g., What happens if there is a technical problem during telesurgery?). \u0026ldquo;Recovery\u0026rdquo; questions were about post-procedures workings (e.g., Is pain better managed after robotic telesurgery?). \u0026ldquo;Other\u0026rdquo; questions were considered miscellaneous and referred to questions that did not fit into any of the other categories (e.g., What if I change my mind and want traditional surgery instead?). The questions were then typed directly into ChatGPT (OpenAI, San Francisco, CA) and Bard (Google, Mountain View, CA) [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. ChatGPT 4o and Gemini 2.5 Flash were used. ChatGPT and Gemini are known to tailor their responses based on your conversation with them to provide more congruent answers. To avoid this confounding variable, for each question, a new conversation was generated to best replicate the experience a patient would have when they ask either of these chatbots a question. Responses were excluded if they were duplicates or were irrelevant to urologic telesurgery within reason. The remaining responses were considered for evaluation and scoring.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eResponse Review:\u003c/h3\u003e\n\u003cp\u003eEach individual response was evaluated by three medical students (TD, PG, and MW). Responses were evaluated using the DISCERN and the Patient Education Materials Assessment Tool for Printed materials (PEMAT-P) tools. Both tools are validated for evaluating health information. DISCERN is a tool used to assess the quality of health information, with studies showing that both untrained individuals and healthcare professionals, alike, can effectively apply DISCERN to identify biases, gaps, and inaccuracies in medical information [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. The DISCERN survey includes 16 questions. The first 8 evaluate the reliability and sourcing of the information provided, while the next 7 focus on the clarity and completeness of details regarding treatment options. The final question asks the rater to give an overall assessment of the material\u0026rsquo;s quality. Each item is scored on a scale from 1 (definite no) to 5 (definite yes), with intermediate scores (2\u0026ndash;4) reflecting partial fulfillment of the criteria. Individual question score interpretation represented as follows: 4.01\u0026ndash;5\u0026thinsp;=\u0026thinsp;excellent, 3.01\u0026ndash;4\u0026thinsp;=\u0026thinsp;good, 2.01\u0026ndash;3\u0026thinsp;=\u0026thinsp;fair, 1.01\u0026ndash;2\u0026thinsp;=\u0026thinsp;poor, and 0\u0026ndash;1\u0026thinsp;=\u0026thinsp;very poor. Category scores are then summed, producing a total score that ranges from 16 to 80. Total scores are interpreted as follows: 68\u0026ndash;80\u0026thinsp;=\u0026thinsp;excellent quality, 55\u0026ndash;67\u0026thinsp;=\u0026thinsp;good quality, 42\u0026ndash;54\u0026thinsp;=\u0026thinsp;fair quality, 31\u0026ndash;41\u0026thinsp;=\u0026thinsp;poor quality, and below 30\u0026thinsp;=\u0026thinsp;very poor quality.\u003c/p\u003e\u003cp\u003eWhile DISCERN evaluated the quality of the information provided, PEMAT-P is a standardized tool that focuses on the actionability and understandability of health information [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. PEMAT-P is comprised of 24 questions, with the first 17 measuring understandability and the remaining 7 measuring actionability [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Of the 24 questions, 7 were excluded as they were not applicable to the method we were using. Of the 7 excluded questions, 5 were excluded due to their emphasis on visual aids, 1 was excluded due to limited need for calculations, and the final one was excluded as it referred to word counts typically associated with research papers and not AI generated responses.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cem\u003eGeneral Characteristics:\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAll 38 responses generated by both chatbots were deemed to be relevant to urologic telesurgery and were included. The average word count for responses were 353 and 579 words for ChatGPT and Gemini, respectively. Gemini more consistently provided sources for its answers (16/19 responses), while ChatGPT was more limited (4/19 responses). Common sources included the NIH, Johns Hopkins, and Mayo Clinic. Notably, neither chatbot utilized visual aids for as on average, only 3.51% of responses provided a visual aid for information clarity (Figure 2).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eDiscern:\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFigure 1 represents the mean total DISCERN scores (out of 80) along with standard deviations for each category (Prospective, Technical, Recovery, Other, and Overall). No significant differences were noted between ChatGPT and Gemini scores, however, across all categories except “Other”, Gemini outperformed ChatGPT. DISCERN contained 16 questions with the average score for each question represented in Figure 1. ChatGPT had 4 excellent, 4 good, 4 fair, and 4 poor average scores (Figure 1). Gemini had 4 excellent, 6 good, 4 fair, and 2 poor scores (Figure 1). The lowest performing question for both chatbots was “Does it describe what happens if no treatment is used?” ChatGPT scored an average of 1.26 on this question, while Gemini scored a 1.14. ChatGPT had a high score a 4.35 for two questions: “Is it relevant?” and “Is it balanced and unbiased?” Gemini’s highest scoring question was in “Is it balanced and unbiased?”, scoring 4.58. Regarding cumulative scores, ChatGPT’s “Prospective” score was 39 (poor) and its “Other” score was 57 (good). Otherwise, the remaining 8 cumulative ChatGPT and Gemini DISCERN scores fell into the fair category.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePEMAT-P Understandability and Actionability:\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFigure 2 represents the mean total PEMAT-P understandability scores for each category along with the mean score for each PEMAT-P understandability question. All categories met the 70% threshold to be considered understandable. The lowest performing PEMAT-P question was “The material used visual aids whenever they could make content more easily understood” with both chatbots averaging a 3.51% for this question. No significant differences were noted by the two chatbots overall, but Gemini did perform better in “Prospective” questions compared to ChatGPT (80% vs 72%). Both chatbots scored 100% in many categories including breaking up the information into sections, using concise language, and using an active voice among others. The information provided by both Chatbots was relevant and understandable.\u003c/p\u003e\n\u003cp\u003eFigure 3 represents the mean total PEMAT-P actionability scores for each category along with the mean score for each PEMAT-P actionability question. Performance here was uniformly bad among the chatbots, with only Gemini’s score in the “Prospective” question category meeting the 70% threshold of being actionable. The worse scoring PEMAT-P actionability question was “using visual aids” which they scored 3.51% and 0.00% in for ChatGPT and Gemini, respectively. While Gemini had the highest score in “Prospective” questions, it also scored a 0% in “Technical” questions. Limited scores in this category show that while the information provided by these chatbots has been deemed understandable and relevant due to previous scores, the actionability they provide remains limited.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWith the increasing popularity of AI, their integration into medicine continues to grow. Currently, AI is being used in a variety of medical settings, including helping take patient histories and assisting with administrative tasks [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. As physician use of AI continues to grow and evolve, it is safe to assume that patients will also turn to AI. As a result, it is important to evaluate the information provided by AI, specifically for medical questions in which inappropriate information can have direct consequences upon the patient. Other papers have compared information provided by ChatGPT and Gemini regarding specific procedures, including laparoscopic donor nephrectomies [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. However, to our knowledge, this is the first study to analyze and compare the quality of information provided by ChatGPT and Gemini regarding urologic telesurgery. Urologic telesurgery is a field that is going to grow with continuous advancements in technology. The idea of a surgeon being able to perform on a patient while not being in the same room, let alone country can be hard for patients to grasp. Hence, they may turn to AI chatbots for information.\u003c/p\u003e\u003cp\u003eIn our analysis, most of the cumulative scores for DISCERN fell into the fair category (6/8) with ChatGPT\u0026rsquo;s \u0026ldquo;Prospective\u0026rdquo; and \u0026ldquo;Other\u0026rdquo; questions landing in poor and good, respectively (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). While there were no statistical differences among mean DISCERN scores and in individual questions, the one metric in which Gemini performed notably better than ChatGPT was in providing sources for information. Regarding this, ChatGPT scored a 1.71 on average while Gemini scored a 3.11. This is supported by a similar study looking at urogenital cancer information provided by AI chatbot which had Gemini outperforming ChatGPT in the same category 4 to 1, respectively [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. However, while the sources provided differed between the two chatbots, the relevancy, clarity, and appropriate nature of information provided was similar (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eAll PEMAT-P mean understandability scores met the minimum 70% threshold for being deemed understandable (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). The threshold of 70% was consistently met despite neither AI chatbot providing a visual aid more than a handful of times (3.51% for both) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). The lack of visual aids was counteracted by the consistent use of everyday language and breaking up of the material into sections for easier digestibility. While no statistical differences were present between ChatGPT and Gemini in terms of understandability, Gemini provided summaries for their answers more consistently, scoring 86% vs the 61.4% of ChatGPT (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). This could contribute to the notably larger average word count provided by Gemini of 579 words vs ChatGPT\u0026rsquo;s 353 words. While our study holds this notable word count difference, other studies have conflicting results, having ChatGPT having higher word counts than Gemini [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. One of these studies utilized ChatGPT v3.5, while the other utilized ChatGPT 4o, same as this study.\u003c/p\u003e\u003cp\u003eWhile AI chatbot performance in for PEMAT-P understandability was sufficient, the opposite was found for PEMAT-P actionability. Only Gemini in the \u0026ldquo;Prospective\u0026rdquo; question category met the 70% minimum threshold for responses deemed \u0026ldquo;actionable\u0026rdquo; (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Otherwise, all mean responses did not meet the minimum 70% threshold. Due to the variability of AI chatbots, there are conflicting results on actionability. Some studies show similar results in which both AI chatbots scored uniformly poor on actionability [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], while others have great results in actionability, meeting the 70% threshold [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Some limitations within the actionability aspect of this study is that sometimes a question may not have a direct or clear actionable response. For example, if you were to ask an AI chatbot \u0026ldquo;what is urologic telesurgery\u0026rdquo;, it can provide a very relevant and understandable answer without necessarily providing any actionable information. However, all responses were still included in actionability grading as the chatbots would occasionally responds with phrases included \u0026ldquo;please consult with a urologist\u0026rdquo; or \u0026ldquo;check nearby hospitals and reference their websites\u0026rdquo;.\u003c/p\u003e\u003cp\u003eOne potential application of AI chatbots based on our findings, along with other studies would be the development of healthcare-specific AI chatbots to address common patient inquiries. Current AI chatbots use a general-purpose language model and adjusting that to a healthcare-specific language model can be more appropriate for healthcare. Patients, rightfully so, will have questions about prospective procedures and appointments, and their best way to get accurate information is to call their provider. However, there are studies showing that unexpected telephone calls can result in a significant, unpredictable demand on workload for nurses and physicians [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. To help relieve burden on the nursing staff and improve ease of access to patients, AI chatbots could be a solution. These chatbots could be programmed with accurate, up-to-date information tailored to the clinic\u0026rsquo;s services, policies, and patient population, allowing patients to obtain answers to frequently asked questions without calling the office, saving time for both the nurses and the patient. However, these chatbots must have strict safeguards that encourages the patient to call the office with any additional questions or confusion. While members of the public have trust in AI chatbots, there are existing risk concerns [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e], which could be addressed with specific tailoring of these chatbots to their respective clinics.\u003c/p\u003e\u003cp\u003eThere has been success in developing healthcare-specific large language models with Google\u0026rsquo;s Med-PaLM, and the University of Florida\u0026rsquo;s GatorTron. Med-PaLM, now Med-PaLM 2, was designed for consumer health queries and became the first large language model to achieve a passing score on United States Medical Licensing Exam questions [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. GatorTron is being developed by the University of Florida and was built on a large database of de-identified clinical data. The strengths of this model is in its ability to answer clinical questions and generate synthetic clinical text [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. While both large language models have shown remarkable success in the clinical setting their application is currently not available outside of research collaborations or institutional oversight. However, just the development of these models shows the practicality they may be able to have if accessible to the public. With appropriate safeguards and criteria meeting the clinical standard, if employed by clinics, healthcare-specific language models can be both beneficial to patients and their providers.\u003c/p\u003e\u003cp\u003eThis study evaluated the information provided by ChatGPT 4o and Gemini 2.5, however, upon conclusion of this paper, ChatGPT 5 was released on August 7th, 2025. Considering ChatGPT 5 being the \u0026ldquo;next step up from 4o,\u0026rdquo; another study should be performed comparing the medical information provided by GPT 5 against 4o and Gemini 2.5 to see if there is truly a significant difference.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eAI chatbots, ChatGPT 4o and Gemini 2.5, deliver relevant and understandable information related to urologic telesurgery. Of note, Gemini more consistently provides sources for its responses compared to ChatGPT, while neither chatbot consistently provided visual aids other than an occasional table. Additionally, neither chatbot reliably offers actionable responses. Their poor performance amongst PEMAT-P actionability criteria limits their utility as a standalone gateway for patient decision-making. This highlights the important role physicians have and will continue have in patient care. Physicians should still be consulted as AI should not be used as a standalone measure for patients. Overall, the use of AI chatbots is effective to garner relevant information, but to acquire specific, actionable information, more detailed prompts/queries along with further discussion with their physician is required.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e- The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.\u003c/p\u003e\u003cp\u003e- The authors have no relevant financial or non-financial interests to disclose.\u003c/p\u003e\u003cp\u003e- All authors contributed to the study conception and design. Material preparation and data collection were performed by Tarak Davuluri, Paul Gabriel, and Matthew Wainstein. Analysis performed by Tarak Davuluri. The first draft of the manuscript was written by Tarak Davuluri and all authors commented on previous versions of the manuscript. Obi Ekwenna led in conceptualization, methodology, supervision, and contributed thoroughly to editing and review process as well. All authors read and approved the final manuscript.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAll authors contributed to the study conception and design. Material preparation and data collection were performed by Tarak Davuluri, Paul Gabriel, and Matthew Wainstein. Analysis performed by Tarak Davuluri. Figures prepared by Tarak Davuluri. The first draft of the manuscript was written by Tarak Davuluri and all authors commented on previous versions of the manuscript. Obi Ekwenna led in conceptualization, methodology, supervision, and contributed thoroughly to editing and review process as well. All authors read and approved the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eMotiwala, Z.Y., et al. (2025). Telesurgery: current status and strategies for latency reduction. J Robot Surg, 19(1), 153. https://doi.org/10.1007/s11701-025-02333-1 \u003c/li\u003e\n\u003cli\u003eLi, J., et al. (2023). Application of Improved Robot-assisted Laparoscopic Telesurgery with 5G Technology in Urology. Eur Urol, 83(1), 41-44. https://doi.org/10.1016/j.eururo.2022.06.018 \u003c/li\u003e\n\u003cli\u003eAldousari, S., et al. (2025). The era of telesurgery: insights from ultra-long-distance Asia to Middle East human telesurgery robotic assisted radical prostatectomy. J Robot Surg, 19(1), 108. https://doi.org/10.1007/s11701-025-02274-9 \u003c/li\u003e\n\u003cli\u003eZhou, F., et al. (2025). Application of 5G Remote Robotic-assisted Laparoscopy in Urological Surgery: A Small Sample Analysis. Urology, 197, 110-114. https://doi.org/10.1016/j.urology.2024.11.019 \u003c/li\u003e\n\u003cli\u003eFerreira, S.V., et al. (2025). Feasibility and Initial Outcomes of Telesurgery in Urology: a Systematic Review of the Literature. Int Braz J Urol, 51(3). https://doi.org/10.1590/S1677-5538.IBJU.2024.0494 \u003c/li\u003e\n\u003cli\u003eAlma Taya, D. and Y.C. Chuang. (2025). Internet use for health information, health service utilization, and quality of care in the U.S. BMC Health Serv Res, 25(1), 659. https://doi.org/10.1186/s12913-025-12807-5 \u003c/li\u003e\n\u003cli\u003eShahsavar, Y. and A. Choudhury. (2023). User Intentions to Use ChatGPT for Self-Diagnosis and Health-Related Purposes: Cross-sectional Survey Study. JMIR Hum Factors, 10, e47564. https://doi.org/10.2196/47564 \u003c/li\u003e\n\u003cli\u003eMusumeci, N. \u003cem\u003eGoogle\u0026rsquo;s Gemini usage is surging, but rivals still dominating.\u003c/em\u003e 2025; Available from: https://www.businessinsider.com/google-gemini-usage-surging-rivals-chatgpt-meta-dominating-2025-4.\u003c/li\u003e\n\u003cli\u003eKuscu, O., et al. (2023). Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front Oncol, 13, 1256459. https://doi.org/10.3389/fonc.2023.1256459 \u003c/li\u003e\n\u003cli\u003eJohnson, D., et al. (2023). Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Res Sq. https://doi.org/10.21203/rs.3.rs-2566942/v1 \u003c/li\u003e\n\u003cli\u003eFattah, F.H., et al. (2025). Comparative analysis of ChatGPT and Gemini (Bard) in medical inquiry: a scoping review. Front Digit Health, 7, 1482712. https://doi.org/10.3389/fdgth.2025.1482712 \u003c/li\u003e\n\u003cli\u003eOpenAi, \u003cem\u003eChatGPT (June 2024) [Large language model]\u003c/em\u003e. 2024, OpenAI.\u003c/li\u003e\n\u003cli\u003eDeepMind, G., \u003cem\u003eChatGPT (June 2024) [Large language model]\u003c/em\u003e. 2024, Google.\u003c/li\u003e\n\u003cli\u003eCharnock, D., et al. (1999). DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health, 53(2), 105-11. https://doi.org/10.1136/jech.53.2.105 \u003c/li\u003e\n\u003cli\u003eCharnock, D. and S. Shepperd. (2004). Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res, 19(4), 440-6. https://doi.org/10.1093/her/cyg046 \u003c/li\u003e\n\u003cli\u003eShoemaker, S.J., M.S. Wolf, and C. Brach. (2014). Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns, 96(3), 395-403. https://doi.org/10.1016/j.pec.2014.05.027 \u003c/li\u003e\n\u003cli\u003eFurukawa, E., et al. (2025). Evaluating Online and Offline Health Information With the Patient Education Materials Assessment Tool: Protocol for a Systematic Review. JMIR Res Protoc, 14, e63489. https://doi.org/10.2196/63489 \u003c/li\u003e\n\u003cli\u003eHindelang, M., S. Sitaru, and A. Zink. (2024). Transforming Health Care Through Chatbots for Medical History-Taking and Future Directions: Comprehensive Systematic Review. JMIR Med Inform, 12, e56628. https://doi.org/10.2196/56628 \u003c/li\u003e\n\u003cli\u003eQin, S., et al. (2024). ChatGPT and generative AI in urology and surgery-A narrative review. BJUI Compass, 5(9), 813-821. https://doi.org/10.1002/bco2.390 \u003c/li\u003e\n\u003cli\u003eMatthew Wainstein, I.D., Stephen Hong, Mehdi Nayebpour, Naoru Koizumi, Obi Ekwenna. (2024). A Quality Analysis of Laparoscopic Donor Nephrectomy-related Information Disseminated by Artificial Intelligence Chatbots using Validated Tools. Journal of Community Medicine \u0026amp; Health Education, 14(03). \u003c/li\u003e\n\u003cli\u003eErkan, A., et al. (2024). Can Patients With Urogenital Cancer Rely on Artificial Intelligence Chatbots for Treatment Decisions? Clin Genitourin Cancer, 22(6), 102206. https://doi.org/10.1016/j.clgc.2024.102206 \u003c/li\u003e\n\u003cli\u003eOzcan, S.G.G. and M. Erkan. (2024). Reliability and quality of information provided by artificial intelligence chatbots on post-contrast acute kidney injury: an evaluation of diagnostic, preventive, and treatment guidance. Rev Assoc Med Bras (1992), 70(11), e20240891. https://doi.org/10.1590/1806-9282.20240891 \u003c/li\u003e\n\u003cli\u003eKolac, U.C., et al. (2025). Can popular AI large language models provide reliable answers to frequently asked questions about rotator cuff tears? JSES Int, 9(2), 390-397. https://doi.org/10.1016/j.jseint.2024.11.012 \u003c/li\u003e\n\u003cli\u003eDelsoz, M., et al. (2025). Large Language Models: Pioneering New Educational Frontiers in Childhood Myopia. Ophthalmol Ther, 14(6), 1281-1295. https://doi.org/10.1007/s40123-025-01142-x \u003c/li\u003e\n\u003cli\u003eBehers, B.J., et al. (2024). Assessing the Quality of Patient Education Materials on Cardiac Catheterization From Artificial Intelligence Chatbots: An Observational Cross-Sectional Study. Cureus, 16(9), e69996. https://doi.org/10.7759/cureus.69996 \u003c/li\u003e\n\u003cli\u003eIto, S., et al. (2025). Leveraging artificial intelligence chatbots for anemia prevention: A comparative study of ChatGPT-3.5, copilot, and Gemini outputs against Google Search results. PEC Innov, 6, 100390. https://doi.org/10.1016/j.pecinn.2025.100390 \u003c/li\u003e\n\u003cli\u003eBurnet, E., et al. (2018). A prospective analysis of unplanned patient-initiated contacts in an adult cystic fibrosis centre. J Cyst Fibros, 17(5), 636-642. https://doi.org/10.1016/j.jcf.2018.04.006 \u003c/li\u003e\n\u003cli\u003eFlannery, M., S.M. Phillips, and C.A. Lyons. (2009). Examining telephone calls in ambulatory oncology. J Oncol Pract, 5(2), 57-60. https://doi.org/10.1200/JOP.0922002 \u003c/li\u003e\n\u003cli\u003eChen, S.Y., H.Y. Kuo, and S.H. Chang. (2024). Perceptions of ChatGPT in healthcare: usefulness, trust, and risk. Front Public Health, 12, 1457131. https://doi.org/10.3389/fpubh.2024.1457131 \u003c/li\u003e\n\u003cli\u003eSinghal, K., et al. (2025). Toward expert-level medical question answering with large language models. Nat Med, 31(3), 943-950. https://doi.org/10.1038/s41591-024-03423-7 \u003c/li\u003e\n\u003cli\u003ePeng, C., et al. (2023). A study of generative large language model for medical research and healthcare. NPJ Digit Med, 6(1), 210. https://doi.org/10.1038/s41746-023-00958-w \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-robotic-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jors","sideBox":"Learn more about [Journal of Robotic Surgery](http://link.springer.com/journal/11701)","snPcode":"11701","submissionUrl":"https://submission.nature.com/new-submission/11701/3","title":"Journal of Robotic Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"AI Chatbots, Urologic Telesurgery, DISCERN, PEMAT","lastPublishedDoi":"10.21203/rs.3.rs-7527866/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7527866/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eIntroduction\u003c/strong\u003e: With increasing accessibility to Artificial Intelligence (AI) chatbots, the precision and clarity of medical information provided requires rigorous assessment. Urologic telesurgery represents a complex concept that patients will investigate using AI. We compared ChatGPT and Google Gemini in providing patient-facing information on urologic telesurgical procedures.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: 19 questions related to urologic telesurgery were generated using general information from the American Urologic Association (AUA) and European Robotic Urology Section (ERUS). Questions were organized into 4 categories (Prospective, Technical, Recovery, Other) and directly typed into ChatGPT 4o and Google Gemini 2.5 (non-paid versions). For each question, a new chat was started to prevent any continuation of answers. Three reviewers independently reviewed the responses using two validated healthcare tools: DISCERN (quality) and Patient Education Material Assessment Tool (understandability and actionability).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Mean DISCERN scores (out of 80) were higher for Gemini than ChatGPT in all domains except “Other”. Prospective 49.2 vs 39.1; technical 52.3 vs 44.3; recovery 53.7 vs 45.4; other 54.3 vs 56.5; overall 52.4 vs 45.8 (Figure 1). PEMAT-P understandability uniformly exceeded 70% for both platforms: prospective 80.0% vs 71.7%; technical 80.1% vs 79.8%; recovery 79.2% vs 80.1%; other 79.2% vs 81.3%; overall 79.7% vs 78.1% (Figure 2). Actionability was uniformly low; only Gemini met 70% threshold in the prospective domain (Figure 3).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e: ChatGPT and Gemini deliver relevant and understandable information related to urologic telesurgery, with Gemini more consistently providing sources. However, neither chatbot reliably offers actionable responses, limiting their utility as a standalone gateway for patient decision-making.\u003c/p\u003e","manuscriptTitle":"Quality Assessment of Patient-Facing Urologic Telesurgery Content Using Validated Tools","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-12 14:59:17","doi":"10.21203/rs.3.rs-7527866/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-09-18T11:13:16+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-18T02:46:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"218783320639986576793499987869110147190","date":"2025-09-17T17:31:00+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-15T18:10:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"54047015210718659475539191656416255627","date":"2025-09-09T05:40:43+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"144008808130096572262004807020472070425","date":"2025-09-09T02:38:19+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-06T20:45:44+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-09-06T20:43:35+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-09-05T13:10:09+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Robotic Surgery","date":"2025-09-03T13:56:48+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-robotic-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jors","sideBox":"Learn more about [Journal of Robotic Surgery](http://link.springer.com/journal/11701)","snPcode":"11701","submissionUrl":"https://submission.nature.com/new-submission/11701/3","title":"Journal of Robotic Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"ec688fe7-a039-455c-a620-7c4569f0096a","owner":[],"postedDate":"September 12th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-10-20T16:02:50+00:00","versionOfRecord":{"articleIdentity":"rs-7527866","link":"https://doi.org/10.1007/s11701-025-02871-8","journal":{"identity":"journal-of-robotic-surgery","isVorOnly":false,"title":"Journal of Robotic Surgery"},"publishedOn":"2025-10-14 15:57:56","publishedOnDateReadable":"October 14th, 2025"},"versionCreatedAt":"2025-09-12 14:59:17","video":"","vorDoi":"10.1007/s11701-025-02871-8","vorDoiUrl":"https://doi.org/10.1007/s11701-025-02871-8","workflowStages":[]},"version":"v1","identity":"rs-7527866","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7527866","identity":"rs-7527866","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}