Expert Appraisal of AI-Generated Recommendations in Cardiovascular Surgical Decision-Making: A Multisurgeon Study

Expert Appraisal of AI-Generated Recommendations in Cardiovascular Surgical Decision-Making: A Multisurgeon Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Expert Appraisal of AI-Generated Recommendations in Cardiovascular Surgical Decision-Making: A Multisurgeon Study Bedirhan Bugra Bayici, Fatih Kizilyel, Mehmet Rum, Safa Ozcelik This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8793975/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 7 You are reading this latest preprint version Abstract Background Large Language Models (LLMs) have demonstrated strong performance in standardized medical examinations; however, their reliability in real-world surgical decision-making, particularly in complex and guideline-ambiguous clinical scenarios, remains uncertain. Cardiovascular surgery, characterized by high risk, complex comorbidities, and frequent reliance on individualized clinical judgment, represents a critical domain for examining how artificial intelligence (AI)–generated recommendations are appraised by clinicians operating in high-risk, judgment-intensive settings. Accordingly, this study aimed to present an expert appraisal of the perceived clinical appropriateness of GPT-5.2–generated recommendations as evaluated by cardiovascular surgeons, and to examine whether appraisal varies according to surgeon experience and scenario type, including guideline-ambiguous (“grey-zone”) decision-making contexts. Methods Fifty original cardiovascular surgical scenarios—covering coronary, valvular, aortic, and peripheral vascular diseases—were developed, including approximately 20% predefined guideline-ambiguous (“grey-zone”) cases. All scenarios were processed using GPT-5.2 with a standardized system instruction and deterministic settings (temperature 0.0; top_p 1.0), with web browsing enabled; each scenario was run in a separate session to minimize contextual carryover. Fourteen actively practicing cardiovascular surgeons independently evaluated the AI-generated recommendations using an anchored 5-point Likert scale capturing perceived clinical appropriateness rather than objective correctness or patient-level outcomes, with scores of 4–5 prespecified as indicating clinically appropriate management. Surgeons were stratified into Early-Career (1–9 years), Mid-Career (10–19 years), and Senior (≥ 20 years) groups. Differences in ratings were analyzed using linear mixed-effects models accounting for repeated measures and cross-classified clustering at both surgeon and scenario levels. Results Across 700 individual evaluations, AI-generated recommendations received a high overall mean rating of 4.31 ± 0.8, with 86% of responses classified as clinically appropriate (Likert score 4–5). Ratings did not differ significantly between Early- and Mid-Career surgeons (p = 0.750), whereas Senior surgeons assigned lower scores than the Mid-Career group (mean difference − 0.27, p = 0.001). Grey-zone scenarios received lower ratings overall (mean difference − 0.19, p = 0.023). Sensitivity analyses using ordinal proportional odds regression yielded directionally consistent results. Conclusions In this surgeon-based appraisal study, AI-generated recommendations demonstrated high perceived clinical appropriateness in cardiovascular surgical decision-making. Differences in ratings reflected an experience-dependent acceptability threshold, alongside an additive scenario-type effect, without evidence of Experience × ScenarioType interaction. These findings suggest that large language models may function as complementary decision-support resources in cardiovascular surgery, while underscoring the continued importance of expert judgment in complex and individualized clinical scenarios. Health sciences/Cardiology Health sciences/Diseases Health sciences/Health care Health sciences/Medical research Artificial intelligence Large language models Cardiovascular surgery Clinical decision-making Clinical appropriateness Figures Figure 1 Figure 2 Introduction Artificial intelligence (AI) has evolved from a purely theoretical concept into an increasingly integrated component of modern clinical practice. Large Language Models (LLMs) have demonstrated strong performance in standardized assessments such as the United States Medical Licensing Examination (USMLE) and specialty board examinations.¹˒² However, whether such performance translates into reliable and clinically acceptable decision-making in real-world surgical settings—where uncertainty, risk, and contextual judgment are central—remains unclear. Cardiovascular surgery represents a particularly demanding clinical domain, characterized by high procedural risk, complex patient comorbidities, and frequent reliance on advanced clinical judgment that extends beyond established guidelines. Surgical decision-making in this field often requires balancing population-based evidence with individualized patient factors, including frailty, anatomical variability, and cumulative operative risk. The distinction between “textbook correctness” and true clinical judgment is therefore critical when evaluating the role of AI in surgical care.³ The release of a next-generation large language model (GPT-5.2) in December 2025 marked a further step in the evolution of AI systems designed for complex reasoning and multistep inference. As such models continue to advance, their potential roles as health information resources for patients and as supportive tools for clinicians have attracted growing interest. However, independent clinical validation—particularly in unprompted, specialty-specific surgical decision-making contexts—remains limited. In complex and ambiguous clinical scenarios, AI-generated recommendations are often perceived as reflecting theoretical knowledge rather than fully capturing the nuances of individualized surgical judgment.⁴ Nevertheless, ongoing developments in AI architecture continue to raise the possibility that these limitations may be progressively addressed.⁵ Accordingly, this study was designed as an expert appraisal of AI-generated surgical recommendations rather than a validation of clinical correctness or patient-level outcomes. In addition, we sought to examine whether expert appraisal of AI-generated recommendations varies according to surgeon experience, with particular attention to guideline-ambiguous (“grey-zone”) clinical scenarios. By stratifying clinicians based on professional seniority, this study provides a focused assessment of how surgical experience influences the interpretation and acceptance of AI-supported clinical guidance. Results A total of 14 cardiovascular surgeons participated in the study, stratified into Early-Career (n = 5), Mid-Career (n = 4), and Senior (n = 5) groups. Across 700 individual surgeon evaluations, AI-generated recommendations received a high overall mean rating of 4.31 ± 0.8. Overall, 86% of recommendations were classified as clinically appropriate, defined as a Likert score of 4 or 5. Subgroup analysis demonstrated differences in Likert-scale ratings according to surgeon experience level. Using the Mid-Career group as the reference category in the linear mixed-effects model, no statistically significant difference was observed between Early-Career and Mid-Career surgeons (p = 0.750). In contrast, Senior surgeons assigned significantly lower scores compared with the Mid-Career group (estimated mean difference − 0.27, p = 0.001). Furthermore, guideline-ambiguous (“grey-zone”) scenarios were independently associated with lower appropriateness ratings (estimated mean difference − 0.19, p = 0.023). This experience-dependent pattern is illustrated in Fig. 1 , and the corresponding mixed-effects model estimates are summarized in Table 1 . Table 1 Fixed effects from the linear mixed-effects model assessing surgeon experience and scenario type Fixed Effect (Reference: Mid-Career / Standard) Estimate (β) 95% CI p-value Surgeon Experience Early-Career vs Mid-Career -0.03 -0.19 to + 0.14 0.750 Senior vs Mid-Career -0.27 -0.44 to -0.11 0.001 Scenario Type Grey-zone vs Standard -0.19 -0.36 to -0.03 0.023 Inter-rater reliability among surgeons was assessed using intraclass correlation coefficients with absolute agreement. The single-measure ICC (ICC[ 2 , 1 ]) was 0.25, reflecting expected variability in individual surgeons’ risk thresholds across complex clinical scenarios. In contrast, the average-measure ICC (ICC[ 2 , 14 ]) was 0.82, indicating excellent reliability of the aggregated expert ratings. Grey-Zone Scenarios High levels of agreement with AI-generated recommendations were observed across all experience groups in standard, guideline-based scenarios. Predefined guideline-ambiguous (“grey-zone”) scenarios were associated with lower appropriateness ratings overall (β = −0.19, 95% CI − 0.36 to − 0.03, p = 0.023) (Fig. 2 ), consistent with increased clinical uncertainty. Although descriptive patterns suggested lower ratings among more experienced surgeons, formal interaction testing within the adjusted mixed-effects model did not demonstrate scenario-specific effect modification, indicating additive rather than divergent effects across experience groups. Ordinal proportional odds sensitivity analyses yielded directionally consistent results, supporting the robustness of the primary mixed-effects findings. Discussion This study provides a surgeon-based appraisal of an advanced large language model in the context of cardiovascular surgical decision-making. Overall, the findings demonstrate a high level of perceived clinical appropriateness of AI-generated recommendations, consistent with prior reports evaluating large language models in medical decision-support settings. Importantly, the results highlight an experience-dependent difference in acceptability thresholds across clinical decision-making, without evidence of scenario-specific interaction effects.⁶˒⁷ The high level of agreement observed among early- and mid-career surgeons in guideline-based scenarios underscores the central role of established clinical guidelines in contemporary surgical decision-making. During these stages of professional development, clinical practice is often shaped by a strong emphasis on evidence-based frameworks, standardized recommendations, and accumulated theoretical knowledge. The substantial concordance between AI-generated recommendations and evaluations by early- and mid-career surgeons suggests that the model effectively reflects prevailing guideline-oriented standards of care in routine and moderately complex cardiovascular surgical scenarios.⁸ As surgical experience increases, clinical decision-making increasingly incorporates individualized judgment alongside guideline-based knowledge, influencing overall acceptability thresholds rather than scenario-specific evaluations. In this context, experienced surgeons may apply broader contextual considerations when evaluating management strategies, particularly in cases where evidence-based recommendations allow for flexibility. The group-level differences observed in this study reflect this shift toward experience-informed appraisal, highlighting how accumulated clinical exposure can influence the interpretation and acceptability of standardized treatment recommendations rather than indicating differences in objective clinical correctness.⁹ The substantial agreement observed between early- and mid-career surgeons carries important clinical implications. The absence of a significant difference between these groups suggests a shared evaluative framework grounded in contemporary evidence-based practice. The high level of concordance between AI-generated recommendations and assessments by these surgeons indicates that the model’s outputs align closely with prevailing guideline-oriented approaches commonly applied in routine cardiovascular surgical decision-making. The lower ratings assigned by senior surgeons reflect an experience-dependent acceptability threshold rather than diminished confidence in either clinical guidelines or AI-generated recommendations. This threshold effect represents a qualitative difference in evaluative perspective, whereby more experienced surgeons apply a higher bar for endorsing standardized management strategies across a range of clinical contexts. Consistent with the study’s results, agreement between AI-generated recommendations and expert appraisal remained high among early- and mid-career surgeons, whereas relatively lower ratings were observed among surgeons with ≥ 20 years of experience, consistent with this acceptability threshold. Importantly, this divergence reflects differences in acceptance rather than differences in objective clinical correctness.¹⁰˒¹¹ However, this pattern may also reflect differences in risk tolerance, expectations for justification/traceability, or baseline skepticism toward AI decision support, rather than differences in objective correctness. The inter-rater reliability findings further contextualize the observed experience-dependent variation in expert appraisal. While individual surgeon ratings demonstrated modest agreement, as reflected by a lower single-measure intraclass correlation coefficient, the aggregated expert assessment showed excellent reliability, with a high average-measure ICC. This pattern indicates that heterogeneity in individual evaluations represents meaningful differences in clinical judgment rather than measurement inconsistency. Importantly, the strong reliability of the consensus-based ratings supports the robustness of surgeon-derived appraisal as the primary outcome measure in this study. In this study, “grey-zone” scenarios were operationally defined a priori using prespecified criteria capturing guideline transitions, therapeutic equivalence, and evidence gaps (Methods). The lower appropriateness ratings observed in these scenarios likely reflect inherent clinical uncertainty and individualized decision-making trade-offs rather than deficiencies in guideline adherence or AI-generated recommendations. Consistent with this interpretation, no scenario-specific effect modification by surgeon experience was observed, underscoring the complementary roles of human expertise and artificial intelligence in managing complex clinical situations.¹²˒¹³ Taken together, these findings suggest that large language models may serve as complementary decision-support resources in cardiovascular surgery rather than autonomous decision-makers. AI-generated recommendations appear well suited to assist clinicians in routine and moderately complex scenarios by providing structured, guideline-consistent analytical input. However, in clinical situations characterized by uncertainty, competing risks, or individualized trade-offs, human expertise remains essential to contextualize recommendations and integrate factors that extend beyond algorithmic reasoning. Within this framework, artificial intelligence may enhance efficiency and consistency in surgical planning while preserving clinician responsibility for final decision-making. The experience-dependent differences observed in this study should be interpreted as differences in evaluative thresholds rather than evidence of context-specific variation in AI performance. While AI systems demonstrate substantial alignment with guideline-based practice, particularly in routine and moderately complex scenarios, grey-zone situations are likely to remain an inherent component of cardiovascular surgery. In such contexts, where clinical uncertainty and patient-specific risk are prominent, AI-generated recommendations alone may be insufficient without integration of expert clinical judgment.¹⁴˒¹⁵ Cardiovascular surgery has long been grounded in collaborative decision-making frameworks. Contemporary practice—shaped by increasing procedural complexity, heightened medicolegal considerations, and expanding patient longevity—has further reinforced the importance of interdisciplinary evaluation. Within this context, artificial intelligence may serve as an additional analytical resource that complements, rather than replaces, collective clinical expertise, supporting structured discussion while preserving the central role of human judgment in complex surgical decisions.⁵ Conclusions GPT-5.2–generated recommendations were perceived by cardiovascular surgeons as clinically appropriate and generally consistent with contemporary guideline-based practice in routine and moderately complex surgical scenarios. Surgeons with up to 19 years of experience demonstrated a high level of agreement regarding the acceptability of AI-generated decision support. In contrast, more experienced surgeons tended to apply a higher evaluative threshold when appraising AI-generated decision support, reflecting differences in clinical acceptability rather than objective correctness. Overall, these findings suggest that artificial intelligence may serve as a complementary decision-support resource in cardiovascular surgery by providing structured, guideline-consistent input while preserving the essential role of clinician-led, experience-based judgment in complex and individualized surgical decision-making. Limitations Several limitations should be acknowledged. Although the total number of scenarios and evaluations was substantial, the number of participating surgeons was relatively limited, and all were drawn from a single national healthcare setting, which may limit generalizability. The study relied on standardized, case-based scenarios rather than real-time clinical encounters; therefore, the findings reflect expert appraisal of structured decision-making rather than bedside performance. Surgeons were aware that the recommendations were generated by an artificial intelligence system, which may have influenced scoring behavior through expectation or skepticism effects, particularly among more experienced clinicians. However, this design choice reflects real-world clinical conditions, in which clinicians typically engage with AI decision-support tools with explicit awareness of their artificial origin. Clinical appropriateness was assessed using an anchored Likert scale, which inherently reflects subjective judgment; however, this subjectivity represents an integral component of real-world clinical decision-making. Consistent with this, inter-rater reliability analysis demonstrated excellent agreement at the aggregate level despite expected variability in individual ratings. In addition, this study did not include a direct baseline comparison with guideline excerpts or human-authored recommendations. This design choice was intentional, as the primary objective was to evaluate surgeon appraisal of AI-generated recommendations in isolation, reflecting how such systems are most likely to be encountered in real-world decision-support settings. Comparative evaluations against guideline text or expert consensus statements represent an important direction for future research, but were beyond the scope of the present study. Because web browsing was enabled, the external sources retrieved by the model may change over time; although we used deterministic sampling and separate sessions, exact reproducibility would require archiving the retrieved sources and access dates. Finally, the study focused on perceived clinical appropriateness rather than patient-level outcomes. The inclusion of predefined guideline-ambiguous (“grey-zone”) scenarios was intentional, reflecting clinically relevant situations in which guidelines are advisory rather than prescriptive. Methods Clinical Scenarios and Surgeon Appraisal Fifty original cardiovascular surgical scenarios were developed to represent a broad spectrum of contemporary clinical practice, including coronary artery disease, valvular heart disease, aortic pathology, and peripheral vascular disease. Scenarios were designed to reflect realistic decision-making contexts encountered in routine clinical practice. Approximately 20% of cases were predefined a priori as guideline-ambiguous (“grey-zone”) scenarios, characterized by limited or conflicting guideline recommendations, emerging evidence, or the need for individualized risk–benefit assessment. All clinical scenarios and corresponding AI-generated recommendations are provided in Supplementary Table S1 , with predefined guideline-ambiguous (“grey-zone”) scenarios explicitly indicated. To classify scenarios as guideline-ambiguous (“grey-zone”), predefined operational criteria were applied beyond low-level evidence classifications. Grey-zone designation was finalized a priori by consensus among three board-certified cardiovascular surgeons, based on the operational criteria below. A scenario was designated as grey-zone if it met at least one of the following conditions: (1) guideline transition, defined as recent updates with revised threshold values not yet uniformly adopted in clinical practice (e.g., updated left ventricular dimension cut-offs in aortic regurgitation); (2) therapeutic equivalence, involving multiple guideline-endorsed management strategies requiring individualized Heart Team–based decision-making (e.g., PCI versus CABG in low anatomic complexity); or (3) evidence gap, characterized by the absence of high-quality comparative randomized data necessitating reliance on expert consensus. The designation of scenarios as guideline-ambiguous (“grey-zone”) was intended as an operational classification rather than an assertion of universal clinical ambiguity. Scenarios were selected to reflect areas of contemporary practice in which guideline recommendations permit flexibility, competing strategies are acceptable, or thresholds are undergoing transition. Prompt Engineering and Model Configuration All scenarios were processed using the GPT-5.2 model (version gpt-5.2-turbo-2025-12-11). To ensure reproducibility and minimize stochastic variability, the model temperature was set to 0.0 with top_p set to 1.0. A standardized system instruction was used for all scenarios: “You are an expert cardiovascular surgeon acting as a senior consultant. Based on the provided clinical scenario, recommend the most appropriate management strategy in accordance with current clinical guidelines.” Web browsing functionality was enabled to allow access to up-to-date clinical evidence. To minimize contextual carryover effects, each scenario was submitted and evaluated in a separate session. AI-generated recommendations were recorded in a result-oriented format without supplementary explanations or justifications. Fourteen actively practicing cardiovascular surgeons independently reviewed the AI-generated recommendations. All participating surgeons were informed that the responses had been generated by an artificial intelligence system and were asked to evaluate each recommendation according to their own clinical judgment. Surgeons were stratified by years of independent practice into Early-Career (1–9 years), Mid-Career (10–19 years), and Senior (≥ 20 years) groups. Each AI-generated recommendation was assessed using an anchored 5-point Likert scale designed to capture perceived clinical appropriateness, where a score of 1 indicated a recommendation that would not be applied in the evaluator’s own clinical practice, and a score of 5 indicated a recommendation considered precisely appropriate and consistent with accepted standards of care (Table 2 ). Scores of 4 or 5 were prespecified as reflecting clinically appropriate management. Table 2 Definitions of the anchored 5-point Likert scale for clinical assessment Value Anchored definition 1 Clinically inappropriate ; I would not apply this recommendation in my own clinical practice. 2 Generally inappropriate ; may be considered only in rare or highly selected cases with significant reservations. 3 Acceptable but debatable (“grey-zone”) ; reasonable but requires additional information, multidisciplinary discussion, or individualized judgment. 4 Clinically appropriate and safe ; a correct and reasonable approach consistent with current practice. 5 Highly appropriate and safe ; a well-justified, guideline-aligned approach that I would confidently apply in routine practice. Statistical Analysis Given the presence of repeated measurements and a cross-classified data structure involving both surgeons and clinical scenarios, linear mixed-effects models were employed for statistical analysis. Surgeon experience group was included as a fixed effect, while random intercepts for Surgeon ID and Scenario ID were specified to account for inter-rater variability and inherent differences in scenario complexity. Likert-scale ratings were treated as approximately continuous variables, consistent with prior methodological literature. The Mid-Career surgeon group was designated as the reference category for between-group comparisons. As a sensitivity analysis accounting for the ordinal nature of the 5-point Likert-scale outcome, proportional odds ordinal logistic regression models were fitted. Surgeon experience group, scenario type (predefined grey-zone vs guideline-based), and their interaction were included to evaluate the robustness of the primary mixed-effects findings. Inter-rater reliability was evaluated using the intraclass correlation coefficient (ICC). A two-way random-effects model with absolute agreement (ICC[ 2 , 1 ]) was applied, reflecting the study design in which both raters and clinical scenarios were considered random effects. ICC values were interpreted according to established methodological conventions. Inter-rater reliability estimates were interpreted in the context of the cross-classified mixed-effects model, with single-measure ICC reflecting individual judgment variability and average-measure ICC reflecting consensus reliability. All statistical tests were two-sided, and statistical significance was defined as p < 0.05. Summary statistics are reported as mean ± standard deviation unless otherwise specified. For Fig. 2 , error bars indicate standard error of the mean. Models were fitted using restricted maximum likelihood estimation; degrees of freedom were approximated using the Satterthwaite method. All analyses were conducted in accordance with institutional ethical standards. AI-Assisted Writing Disclosure During the preparation of this manuscript, the authors used a generative artificial intelligence tool to support language editing and clarity. After using this tool, the authors reviewed and edited the content and take full responsibility for the content of the published article. Data Availability All data generated or analysed during this study are included in this published article and its Supplementary Information files. Code Availability The analysis scripts used to fit the mixed-effects and ordinal models and to compute ICC estimates are not publicly available but may be made available to qualified researchers on reasonable request from the corresponding author. Declarations Author Contributions B.B.B. and F.K. designed the study and scenarios. M.R. and S.O. performed data collection and initial formatting. B.B.B. conducted the statistical analysis. All authors interpreted the data. B.B.B. wrote the main manuscript text. All authors reviewed and approved the final manuscript. Acknowledgements The authors thank the cardiovascular surgeons who participated as independent expert evaluators and provided independent assessments of the AI-generated recommendations. Ethics Statement This study involved the evaluation of hypothetical clinical scenarios and expert assessments of AI-generated recommendations and did not involve human participants, patient data, or clinical interventions. Informed consent was obtained from all expert participants involved in the study prior to their participation. In accordance with institutional policies and local regulations, formal ethics committee approval was not required. Patient Consent Statement Patient consent was not applicable to this study because no individual patient data, images, or identifiable information were included. Funding Sources This research received no external funding. Disclosures The authors declare no competing interests. References Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit. Health . 2 , e0000198 (2023). Gilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? Implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9 , e45312 (2023). Hashimoto, D. A., Rosman, G., Rus, D. & Meireles, O. R. Artificial intelligence in surgery: Promises and perils. Ann. Surg. 268 , 70–76 (2018). Loftus, T. J. et al. Artificial intelligence and surgical decision-making. JAMA Surg. 155 , 148–158 (2020). Senders, J. T. et al. Machine learning and neurosurgical outcome prediction: A systematic review. World Neurosurg. 109 , 476–486e1 (2018). Topol, E. J. High-performance medicine: The convergence of human and artificial intelligence. Nat. Med. 25 , 44–56 (2019). Zhang, C. et al. Examining the role of large language models in orthopedics: A systematic review. J. Med. Internet Res. 26 , e59607 (2024). Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25 , 24–29 (2019). Ericsson, K. A. Deliberate practice and acquisition of expert performance: A general overview. Acad. Emerg. Med. 15 , 988–994 (2008). Kahneman, D. & Klein, G. Conditions for intuitive expertise: A failure to disagree. Am. Psychol. 64 , 515–526 (2009). Cabitza, F., Rasoini, R. & Gensini, G. F. Unintended consequences of machine learning in medicine. JAMA 318 , 517–518 (2017). Neumann, F. J. et al. ESC/EACTS guidelines on myocardial revascularization. Eur. Heart J. 40 , 87–165 (2019). Otto, C. M. et al. ACC/AHA guideline for the management of patients with valvular heart disease: Executive summary. Circulation 143 , e35–e71 (2021). Shortliffe, E. H. & Sepúlveda, M. J. Clinical decision support in the era of artificial intelligence. JAMA 320 , 2199–2200 (2018). Guyatt, G. et al. GRADE guidelines: Introduction—GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64 , 383–394 (2011). Additional Declarations No competing interests reported. Supplementary Files SUPPLEMENTARYTABLES1.docx Supplementary Table S1: Full list of the 50 cardiovascular surgical clinical scenarios included in the study, together with the corresponding AI-generated recommendations. Scenarios were designed to represent routine and complex decision-making situations across coronary, valvular, aortic, and peripheral vascular disease domains. Predefined guideline-ambiguous (“grey-zone”) scenarios are explicitly indicated. All AI-generated responses were produced using the same standardized instruction and recorded in a result-oriented format without supplementary explanations. SourceData.xlsx Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 17 Apr, 2026 Reviewers agreed at journal 09 Apr, 2026 Reviewers invited by journal 09 Apr, 2026 Editor assigned by journal 08 Apr, 2026 Editor invited by journal 12 Feb, 2026 Submission checks completed at journal 07 Feb, 2026 First submitted to journal 07 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8793975","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":622784316,"identity":"c06c5f2b-3054-4f79-8456-211b883abc6a","order_by":0,"name":"Bedirhan Bugra Bayici","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA90lEQVRIiWNgGAWjYBACAyA6wNgAZgBBBRAzMzeQouUMSAsjYS0McC2MbWASvxZz9uaNhyt32OWZS6Rf+/BzXm00fztQy4+KbTi1WPYcKzh49kxyseWMnOKZvduO5844zNjA2HPmNm6H3cgxONjYxpy44UZOMgPvtmO5DUAtzIxteLTcfwPSUg/Wwvh3zrHc+QS13OABaTkM1JJ+mJm3oSZ3AyEtlj1pBUAtxxN39rxhZpY5diB3I1DLQXx+MWc/vPljY1t14nb29MeMb2rqcuedP3zwwY8K3FqQAA8oag6DmQeIUQ8E7A+ARB2RikfBKBgFo2AkAQAO/2ZWX1UOewAAAABJRU5ErkJggg==","orcid":"","institution":"Kosuyolu Yuksek Ihtisas Egitim ve Arastirma Hastanesi","correspondingAuthor":true,"prefix":"","firstName":"Bedirhan","middleName":"Bugra","lastName":"Bayici","suffix":""},{"id":622784319,"identity":"1c273552-1f87-4750-bd87-7089b2976c43","order_by":1,"name":"Fatih Kizilyel","email":"","orcid":"","institution":"Kosuyolu Yuksek Ihtisas Egitim ve Arastirma Hastanesi","correspondingAuthor":false,"prefix":"","firstName":"Fatih","middleName":"","lastName":"Kizilyel","suffix":""},{"id":622784320,"identity":"5ae3ae35-1d23-49c5-b430-9320847b20e8","order_by":2,"name":"Mehmet Rum","email":"","orcid":"","institution":"Kosuyolu Yuksek Ihtisas Egitim ve Arastirma Hastanesi","correspondingAuthor":false,"prefix":"","firstName":"Mehmet","middleName":"","lastName":"Rum","suffix":""},{"id":622784322,"identity":"b08eb36f-3996-46c8-8ded-97c3e537fc03","order_by":3,"name":"Safa Ozcelik","email":"","orcid":"","institution":"Kosuyolu Yuksek Ihtisas Egitim ve Arastirma Hastanesi","correspondingAuthor":false,"prefix":"","firstName":"Safa","middleName":"","lastName":"Ozcelik","suffix":""}],"badges":[],"createdAt":"2026-02-05 08:09:52","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8793975/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8793975/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107482583,"identity":"00de1c45-5c6b-43b6-92db-bfceb6eb3b9b","added_by":"auto","created_at":"2026-04-22 02:24:08","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":220104,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of perceived clinical appropriateness scores for AI-generated cardiovascular surgical recommendations by surgeon experience. Boxplots show median and interquartile range; diamonds indicate mean Likert scores. Early- and Mid-Career surgeons showed similar ratings (p = 0.750), whereas Senior surgeons assigned lower scores (p = 0.001), based on a linear mixed-effects model accounting for repeated measures and cross-classified clustering.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8793975/v1/20e0a07ccf47319e2840ad8f.png"},{"id":107482591,"identity":"8a9f2b15-3ef9-4967-8c94-c87493a484f0","added_by":"auto","created_at":"2026-04-22 02:24:08","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1268063,"visible":true,"origin":"","legend":"\u003cp\u003eMean perceived clinical appropriateness scores for AI-generated recommendations in standard guideline-based and predefined guideline-ambiguous (“grey-zone”) scenarios, stratified by surgeon experience. Bars represent mean Likert scores and error bars indicate standard error. Grey-zone scenarios were associated with lower ratings across all experience groups.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8793975/v1/32a8287a4919001bf9c225bb.png"},{"id":107705081,"identity":"d2c21da0-997d-4106-8c1d-ee5777529aaa","added_by":"auto","created_at":"2026-04-24 09:07:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1707236,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8793975/v1/1f5b7b4c-aaa3-4ff8-bdb2-78762ccef4e6.pdf"},{"id":107254863,"identity":"0d5d135e-67c2-45ae-82ba-815cfc4894d1","added_by":"auto","created_at":"2026-04-19 12:06:09","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":29125,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSupplementary Table S1:\u003c/strong\u003e Full list of the 50 cardiovascular surgical clinical scenarios included in the study, together with the corresponding AI-generated recommendations. Scenarios were designed to represent routine and complex decision-making situations across coronary, valvular, aortic, and peripheral vascular disease domains. Predefined guideline-ambiguous (“grey-zone”) scenarios are explicitly indicated. All AI-generated responses were produced using the same standardized instruction and recorded in a result-oriented format without supplementary explanations.\u003c/p\u003e","description":"","filename":"SUPPLEMENTARYTABLES1.docx","url":"https://assets-eu.researchsquare.com/files/rs-8793975/v1/6d004e20c37ba7498d838655.docx"},{"id":107254865,"identity":"77eee28b-bffc-4d16-991d-c88e38b5a9cd","added_by":"auto","created_at":"2026-04-19 12:06:09","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":482195,"visible":true,"origin":"","legend":"","description":"","filename":"SourceData.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-8793975/v1/9b7309c2d9881f850edf7ba1.xlsx"}],"financialInterests":"No competing interests reported.","formattedTitle":"\u003cp\u003eExpert Appraisal of AI-Generated Recommendations in Cardiovascular Surgical Decision-Making: A Multisurgeon Study\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eArtificial intelligence (AI) has evolved from a purely theoretical concept into an increasingly integrated component of modern clinical practice. Large Language Models (LLMs) have demonstrated strong performance in standardized assessments such as the United States Medical Licensing Examination (USMLE) and specialty board examinations.\u0026sup1;˒\u0026sup2; However, whether such performance translates into reliable and clinically acceptable decision-making in real-world surgical settings\u0026mdash;where uncertainty, risk, and contextual judgment are central\u0026mdash;remains unclear.\u003c/p\u003e \u003cp\u003e Cardiovascular surgery represents a particularly demanding clinical domain, characterized by high procedural risk, complex patient comorbidities, and frequent reliance on advanced clinical judgment that extends beyond established guidelines. Surgical decision-making in this field often requires balancing population-based evidence with individualized patient factors, including frailty, anatomical variability, and cumulative operative risk. The distinction between \u0026ldquo;textbook correctness\u0026rdquo; and true clinical judgment is therefore critical when evaluating the role of AI in surgical care.\u0026sup3;\u003c/p\u003e \u003cp\u003eThe release of a next-generation large language model (GPT-5.2) in December 2025 marked a further step in the evolution of AI systems designed for complex reasoning and multistep inference. As such models continue to advance, their potential roles as health information resources for patients and as supportive tools for clinicians have attracted growing interest. However, independent clinical validation\u0026mdash;particularly in unprompted, specialty-specific surgical decision-making contexts\u0026mdash;remains limited. In complex and ambiguous clinical scenarios, AI-generated recommendations are often perceived as reflecting theoretical knowledge rather than fully capturing the nuances of individualized surgical judgment.⁴ Nevertheless, ongoing developments in AI architecture continue to raise the possibility that these limitations may be progressively addressed.⁵\u003c/p\u003e \u003cp\u003eAccordingly, this study was designed as an expert appraisal of AI-generated surgical recommendations rather than a validation of clinical correctness or patient-level outcomes. In addition, we sought to examine whether expert appraisal of AI-generated recommendations varies according to surgeon experience, with particular attention to guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) clinical scenarios. By stratifying clinicians based on professional seniority, this study provides a focused assessment of how surgical experience influences the interpretation and acceptance of AI-supported clinical guidance.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eA total of 14 cardiovascular surgeons participated in the study, stratified into Early-Career (n\u0026thinsp;=\u0026thinsp;5), Mid-Career (n\u0026thinsp;=\u0026thinsp;4), and Senior (n\u0026thinsp;=\u0026thinsp;5) groups. Across 700 individual surgeon evaluations, AI-generated recommendations received a high overall mean rating of 4.31\u0026thinsp;\u0026plusmn;\u0026thinsp;0.8. Overall, 86% of recommendations were classified as clinically appropriate, defined as a Likert score of 4 or 5.\u003c/p\u003e \u003cp\u003eSubgroup analysis demonstrated differences in Likert-scale ratings according to surgeon experience level. Using the Mid-Career group as the reference category in the linear mixed-effects model, no statistically significant difference was observed between Early-Career and Mid-Career surgeons (p\u0026thinsp;=\u0026thinsp;0.750). In contrast, Senior surgeons assigned significantly lower scores compared with the Mid-Career group (estimated mean difference\u0026thinsp;\u0026minus;\u0026thinsp;0.27, p\u0026thinsp;=\u0026thinsp;0.001). Furthermore, guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) scenarios were independently associated with lower appropriateness ratings (estimated mean difference\u0026thinsp;\u0026minus;\u0026thinsp;0.19, p\u0026thinsp;=\u0026thinsp;0.023). This experience-dependent pattern is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, and the corresponding mixed-effects model estimates are summarized in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFixed effects from the linear mixed-effects model assessing surgeon experience and scenario type\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFixed Effect (Reference: Mid-Career / Standard)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEstimate (β)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e95% CI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ep-value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSurgeon Experience\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEarly-Career vs Mid-Career\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-0.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-0.19 to +\u0026thinsp;0.14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.750\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSenior vs Mid-Career\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-0.27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-0.44 to -0.11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eScenario Type\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGrey-zone vs Standard\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e-0.19\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e-0.36 to -0.03\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.023\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eInter-rater reliability among surgeons was assessed using intraclass correlation coefficients with absolute agreement. The single-measure ICC (ICC[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]) was 0.25, reflecting expected variability in individual surgeons\u0026rsquo; risk thresholds across complex clinical scenarios. In contrast, the average-measure ICC (ICC[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]) was 0.82, indicating excellent reliability of the aggregated expert ratings.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eGrey-Zone Scenarios\u003c/h2\u003e \u003cp\u003e High levels of agreement with AI-generated recommendations were observed across all experience groups in standard, guideline-based scenarios. Predefined guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) scenarios were associated with lower appropriateness ratings overall (β = \u0026minus;0.19, 95% CI\u0026thinsp;\u0026minus;\u0026thinsp;0.36 to \u0026minus;\u0026thinsp;0.03, p\u0026thinsp;=\u0026thinsp;0.023) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e), consistent with increased clinical uncertainty. Although descriptive patterns suggested lower ratings among more experienced surgeons, formal interaction testing within the adjusted mixed-effects model did not demonstrate scenario-specific effect modification, indicating additive rather than divergent effects across experience groups. Ordinal proportional odds sensitivity analyses yielded directionally consistent results, supporting the robustness of the primary mixed-effects findings.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study provides a surgeon-based appraisal of an advanced large language model in the context of cardiovascular surgical decision-making. Overall, the findings demonstrate a high level of perceived clinical appropriateness of AI-generated recommendations, consistent with prior reports evaluating large language models in medical decision-support settings. Importantly, the results highlight an experience-dependent difference in acceptability thresholds across clinical decision-making, without evidence of scenario-specific interaction effects.⁶˒⁷\u003c/p\u003e \u003cp\u003e The high level of agreement observed among early- and mid-career surgeons in guideline-based scenarios underscores the central role of established clinical guidelines in contemporary surgical decision-making. During these stages of professional development, clinical practice is often shaped by a strong emphasis on evidence-based frameworks, standardized recommendations, and accumulated theoretical knowledge. The substantial concordance between AI-generated recommendations and evaluations by early- and mid-career surgeons suggests that the model effectively reflects prevailing guideline-oriented standards of care in routine and moderately complex cardiovascular surgical scenarios.⁸\u003c/p\u003e \u003cp\u003e As surgical experience increases, clinical decision-making increasingly incorporates individualized judgment alongside guideline-based knowledge, influencing overall acceptability thresholds rather than scenario-specific evaluations. In this context, experienced surgeons may apply broader contextual considerations when evaluating management strategies, particularly in cases where evidence-based recommendations allow for flexibility. The group-level differences observed in this study reflect this shift toward experience-informed appraisal, highlighting how accumulated clinical exposure can influence the interpretation and acceptability of standardized treatment recommendations rather than indicating differences in objective clinical correctness.⁹\u003c/p\u003e \u003cp\u003eThe substantial agreement observed between early- and mid-career surgeons carries important clinical implications. The absence of a significant difference between these groups suggests a shared evaluative framework grounded in contemporary evidence-based practice. The high level of concordance between AI-generated recommendations and assessments by these surgeons indicates that the model\u0026rsquo;s outputs align closely with prevailing guideline-oriented approaches commonly applied in routine cardiovascular surgical decision-making.\u003c/p\u003e \u003cp\u003e The lower ratings assigned by senior surgeons reflect an experience-dependent acceptability threshold rather than diminished confidence in either clinical guidelines or AI-generated recommendations. This threshold effect represents a qualitative difference in evaluative perspective, whereby more experienced surgeons apply a higher bar for endorsing standardized management strategies across a range of clinical contexts. Consistent with the study\u0026rsquo;s results, agreement between AI-generated recommendations and expert appraisal remained high among early- and mid-career surgeons, whereas relatively lower ratings were observed among surgeons with \u0026ge;\u0026thinsp;20 years of experience, consistent with this acceptability threshold. Importantly, this divergence reflects differences in acceptance rather than differences in objective clinical correctness.\u0026sup1;⁰˒\u0026sup1;\u0026sup1; However, this pattern may also reflect differences in risk tolerance, expectations for justification/traceability, or baseline skepticism toward AI decision support, rather than differences in objective correctness.\u003c/p\u003e \u003cp\u003eThe inter-rater reliability findings further contextualize the observed experience-dependent variation in expert appraisal. While individual surgeon ratings demonstrated modest agreement, as reflected by a lower single-measure intraclass correlation coefficient, the aggregated expert assessment showed excellent reliability, with a high average-measure ICC. This pattern indicates that heterogeneity in individual evaluations represents meaningful differences in clinical judgment rather than measurement inconsistency. Importantly, the strong reliability of the consensus-based ratings supports the robustness of surgeon-derived appraisal as the primary outcome measure in this study.\u003c/p\u003e \u003cp\u003e In this study, \u0026ldquo;grey-zone\u0026rdquo; scenarios were operationally defined a priori using prespecified criteria capturing guideline transitions, therapeutic equivalence, and evidence gaps (Methods). The lower appropriateness ratings observed in these scenarios likely reflect inherent clinical uncertainty and individualized decision-making trade-offs rather than deficiencies in guideline adherence or AI-generated recommendations. Consistent with this interpretation, no scenario-specific effect modification by surgeon experience was observed, underscoring the complementary roles of human expertise and artificial intelligence in managing complex clinical situations.\u0026sup1;\u0026sup2;˒\u0026sup1;\u0026sup3;\u003c/p\u003e \u003cp\u003eTaken together, these findings suggest that large language models may serve as complementary decision-support resources in cardiovascular surgery rather than autonomous decision-makers. AI-generated recommendations appear well suited to assist clinicians in routine and moderately complex scenarios by providing structured, guideline-consistent analytical input. However, in clinical situations characterized by uncertainty, competing risks, or individualized trade-offs, human expertise remains essential to contextualize recommendations and integrate factors that extend beyond algorithmic reasoning. Within this framework, artificial intelligence may enhance efficiency and consistency in surgical planning while preserving clinician responsibility for final decision-making.\u003c/p\u003e \u003cp\u003eThe experience-dependent differences observed in this study should be interpreted as differences in evaluative thresholds rather than evidence of context-specific variation in AI performance. While AI systems demonstrate substantial alignment with guideline-based practice, particularly in routine and moderately complex scenarios, grey-zone situations are likely to remain an inherent component of cardiovascular surgery. In such contexts, where clinical uncertainty and patient-specific risk are prominent, AI-generated recommendations alone may be insufficient without integration of expert clinical judgment.\u0026sup1;⁴˒\u0026sup1;⁵\u003c/p\u003e \u003cp\u003eCardiovascular surgery has long been grounded in collaborative decision-making frameworks. Contemporary practice\u0026mdash;shaped by increasing procedural complexity, heightened medicolegal considerations, and expanding patient longevity\u0026mdash;has further reinforced the importance of interdisciplinary evaluation. Within this context, artificial intelligence may serve as an additional analytical resource that complements, rather than replaces, collective clinical expertise, supporting structured discussion while preserving the central role of human judgment in complex surgical decisions.⁵\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003e GPT-5.2\u0026ndash;generated recommendations were perceived by cardiovascular surgeons as clinically appropriate and generally consistent with contemporary guideline-based practice in routine and moderately complex surgical scenarios. Surgeons with up to 19 years of experience demonstrated a high level of agreement regarding the acceptability of AI-generated decision support. In contrast, more experienced surgeons tended to apply a higher evaluative threshold when appraising AI-generated decision support, reflecting differences in clinical acceptability rather than objective correctness.\u003c/p\u003e \u003cp\u003e Overall, these findings suggest that artificial intelligence may serve as a complementary decision-support resource in cardiovascular surgery by providing structured, guideline-consistent input while preserving the essential role of clinician-led, experience-based judgment in complex and individualized surgical decision-making.\u003c/p\u003e"},{"header":"Limitations","content":"\u003cp\u003eSeveral limitations should be acknowledged. Although the total number of scenarios and evaluations was substantial, the number of participating surgeons was relatively limited, and all were drawn from a single national healthcare setting, which may limit generalizability. The study relied on standardized, case-based scenarios rather than real-time clinical encounters; therefore, the findings reflect expert appraisal of structured decision-making rather than bedside performance.\u003c/p\u003e \u003cp\u003eSurgeons were aware that the recommendations were generated by an artificial intelligence system, which may have influenced scoring behavior through expectation or skepticism effects, particularly among more experienced clinicians. However, this design choice reflects real-world clinical conditions, in which clinicians typically engage with AI decision-support tools with explicit awareness of their artificial origin. Clinical appropriateness was assessed using an anchored Likert scale, which inherently reflects subjective judgment; however, this subjectivity represents an integral component of real-world clinical decision-making. Consistent with this, inter-rater reliability analysis demonstrated excellent agreement at the aggregate level despite expected variability in individual ratings.\u003c/p\u003e \u003cp\u003e In addition, this study did not include a direct baseline comparison with guideline excerpts or human-authored recommendations. This design choice was intentional, as the primary objective was to evaluate surgeon appraisal of AI-generated recommendations in isolation, reflecting how such systems are most likely to be encountered in real-world decision-support settings. Comparative evaluations against guideline text or expert consensus statements represent an important direction for future research, but were beyond the scope of the present study. Because web browsing was enabled, the external sources retrieved by the model may change over time; although we used deterministic sampling and separate sessions, exact reproducibility would require archiving the retrieved sources and access dates.\u003c/p\u003e \u003cp\u003eFinally, the study focused on perceived clinical appropriateness rather than patient-level outcomes. The inclusion of predefined guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) scenarios was intentional, reflecting clinically relevant situations in which guidelines are advisory rather than prescriptive.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eClinical Scenarios and Surgeon Appraisal\u003c/h2\u003e \u003cp\u003eFifty original cardiovascular surgical scenarios were developed to represent a broad spectrum of contemporary clinical practice, including coronary artery disease, valvular heart disease, aortic pathology, and peripheral vascular disease. Scenarios were designed to reflect realistic decision-making contexts encountered in routine clinical practice. Approximately 20% of cases were predefined \u003cem\u003ea priori\u003c/em\u003e as guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) scenarios, characterized by limited or conflicting guideline recommendations, emerging evidence, or the need for individualized risk\u0026ndash;benefit assessment.\u003c/p\u003e \u003cp\u003eAll clinical scenarios and corresponding AI-generated recommendations are provided in Supplementary Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e, with predefined guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) scenarios explicitly indicated.\u003c/p\u003e \u003cp\u003e To classify scenarios as guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;), predefined operational criteria were applied beyond low-level evidence classifications. Grey-zone designation was finalized a priori by consensus among three board-certified cardiovascular surgeons, based on the operational criteria below. A scenario was designated as grey-zone if it met at least one of the following conditions: (1) guideline transition, defined as recent updates with revised threshold values not yet uniformly adopted in clinical practice (e.g., updated left ventricular dimension cut-offs in aortic regurgitation); (2) therapeutic equivalence, involving multiple guideline-endorsed management strategies requiring individualized Heart Team\u0026ndash;based decision-making (e.g., PCI versus CABG in low anatomic complexity); or (3) evidence gap, characterized by the absence of high-quality comparative randomized data necessitating reliance on expert consensus. The designation of scenarios as guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) was intended as an operational classification rather than an assertion of universal clinical ambiguity. Scenarios were selected to reflect areas of contemporary practice in which guideline recommendations permit flexibility, competing strategies are acceptable, or thresholds are undergoing transition.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003ePrompt Engineering and Model Configuration\u003c/h3\u003e\n\u003cp\u003eAll scenarios were processed using the GPT-5.2 model (version gpt-5.2-turbo-2025-12-11). To ensure reproducibility and minimize stochastic variability, the model temperature was set to 0.0 with top_p set to 1.0. A standardized system instruction was used for all scenarios: \u0026ldquo;You are an expert cardiovascular surgeon acting as a senior consultant. Based on the provided clinical scenario, recommend the most appropriate management strategy in accordance with current clinical guidelines.\u0026rdquo; Web browsing functionality was enabled to allow access to up-to-date clinical evidence. To minimize contextual carryover effects, each scenario was submitted and evaluated in a separate session. AI-generated recommendations were recorded in a result-oriented format without supplementary explanations or justifications.\u003c/p\u003e \u003cp\u003eFourteen actively practicing cardiovascular surgeons independently reviewed the AI-generated recommendations. All participating surgeons were informed that the responses had been generated by an artificial intelligence system and were asked to evaluate each recommendation according to their own clinical judgment. Surgeons were stratified by years of independent practice into Early-Career (1\u0026ndash;9 years), Mid-Career (10\u0026ndash;19 years), and Senior (\u0026ge;\u0026thinsp;20 years) groups.\u003c/p\u003e \u003cp\u003eEach AI-generated recommendation was assessed using an anchored 5-point Likert scale designed to capture perceived clinical appropriateness, where a score of 1 indicated a recommendation that would not be applied in the evaluator\u0026rsquo;s own clinical practice, and a score of 5 indicated a recommendation considered precisely appropriate and consistent with accepted standards of care (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Scores of 4 or 5 were prespecified as reflecting clinically appropriate management.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDefinitions of the anchored 5-point Likert scale for clinical assessment\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eValue\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAnchored definition\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eClinically inappropriate\u003c/b\u003e; I would not apply this recommendation in my own clinical practice.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eGenerally inappropriate\u003c/b\u003e; may be considered only in rare or highly selected cases with significant reservations.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eAcceptable but debatable (\u0026ldquo;grey-zone\u0026rdquo;)\u003c/b\u003e; reasonable but requires additional information, multidisciplinary discussion, or individualized judgment.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eClinically appropriate and safe\u003c/b\u003e; a correct and reasonable approach consistent with current practice.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eHighly appropriate and safe\u003c/b\u003e; a well-justified, guideline-aligned approach that I would confidently apply in routine practice.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eGiven the presence of repeated measurements and a cross-classified data structure involving both surgeons and clinical scenarios, linear mixed-effects models were employed for statistical analysis. Surgeon experience group was included as a fixed effect, while random intercepts for Surgeon ID and Scenario ID were specified to account for inter-rater variability and inherent differences in scenario complexity. Likert-scale ratings were treated as approximately continuous variables, consistent with prior methodological literature. The Mid-Career surgeon group was designated as the reference category for between-group comparisons.\u003c/p\u003e \u003cp\u003eAs a sensitivity analysis accounting for the ordinal nature of the 5-point Likert-scale outcome, proportional odds ordinal logistic regression models were fitted. Surgeon experience group, scenario type (predefined grey-zone vs guideline-based), and their interaction were included to evaluate the robustness of the primary mixed-effects findings.\u003c/p\u003e \u003cp\u003eInter-rater reliability was evaluated using the intraclass correlation coefficient (ICC). A two-way random-effects model with absolute agreement (ICC[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]) was applied, reflecting the study design in which both raters and clinical scenarios were considered random effects. ICC values were interpreted according to established methodological conventions. Inter-rater reliability estimates were interpreted in the context of the cross-classified mixed-effects model, with single-measure ICC reflecting individual judgment variability and average-measure ICC reflecting consensus reliability.\u003c/p\u003e \u003cp\u003eAll statistical tests were two-sided, and statistical significance was defined as p\u0026thinsp;\u0026lt;\u0026thinsp;0.05. Summary statistics are reported as mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation unless otherwise specified. For Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, error bars indicate standard error of the mean. Models were fitted using restricted maximum likelihood estimation; degrees of freedom were approximated using the Satterthwaite method. All analyses were conducted in accordance with institutional ethical standards.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eAI-Assisted Writing Disclosure\u003c/h2\u003e \u003cp\u003eDuring the preparation of this manuscript, the authors used a generative artificial intelligence tool to support language editing and clarity. After using this tool, the authors reviewed and edited the content and take full responsibility for the content of the published article.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eData Availability\u003c/h2\u003e \u003cp\u003eAll data generated or analysed during this study are included in this published article and its Supplementary Information files.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eCode Availability\u003c/h2\u003e \u003cp\u003eThe analysis scripts used to fit the mixed-effects and ordinal models and to compute ICC estimates are not publicly available but may be made available to qualified researchers on reasonable request from the corresponding author.\u003c/p\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003cbr\u003eB.B.B. and F.K. designed the study and scenarios. M.R. and S.O. performed data collection and initial formatting. B.B.B. conducted the statistical analysis. All authors interpreted the data. B.B.B. wrote the main manuscript text. All authors reviewed and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors thank the cardiovascular surgeons who participated as independent expert evaluators and provided independent assessments of the AI-generated recommendations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eEthics Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study involved the evaluation of hypothetical clinical scenarios and expert assessments of AI-generated recommendations and did not involve human participants, patient data, or clinical interventions. \u003cstrong\u003eInformed consent was obtained from all expert participants involved in the study prior to their participation.\u0026nbsp;\u003c/strong\u003eIn accordance with institutional policies and local regulations, formal ethics committee approval was not required.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePatient Consent Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePatient consent was not applicable to this study because no individual patient data, images, or identifiable information were included.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding Sources\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research received no external funding.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDisclosures\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eKung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. \u003cem\u003ePLoS Digit. Health\u003c/em\u003e. \u003cb\u003e2\u003c/b\u003e, e0000198 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? Implications of large language models for medical education and knowledge assessment. \u003cem\u003eJMIR Med. Educ.\u003c/em\u003e \u003cb\u003e9\u003c/b\u003e, e45312 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHashimoto, D. A., Rosman, G., Rus, D. \u0026amp; Meireles, O. R. Artificial intelligence in surgery: Promises and perils. \u003cem\u003eAnn. Surg.\u003c/em\u003e \u003cb\u003e268\u003c/b\u003e, 70\u0026ndash;76 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLoftus, T. J. et al. Artificial intelligence and surgical decision-making. \u003cem\u003eJAMA Surg.\u003c/em\u003e \u003cb\u003e155\u003c/b\u003e, 148\u0026ndash;158 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSenders, J. T. et al. Machine learning and neurosurgical outcome prediction: A systematic review. \u003cem\u003eWorld Neurosurg.\u003c/em\u003e \u003cb\u003e109\u003c/b\u003e, 476\u0026ndash;486e1 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTopol, E. J. High-performance medicine: The convergence of human and artificial intelligence. \u003cem\u003eNat. Med.\u003c/em\u003e \u003cb\u003e25\u003c/b\u003e, 44\u0026ndash;56 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, C. et al. Examining the role of large language models in orthopedics: A systematic review. \u003cem\u003eJ. Med. Internet Res.\u003c/em\u003e \u003cb\u003e26\u003c/b\u003e, e59607 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEsteva, A. et al. A guide to deep learning in healthcare. \u003cem\u003eNat. Med.\u003c/em\u003e \u003cb\u003e25\u003c/b\u003e, 24\u0026ndash;29 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEricsson, K. A. Deliberate practice and acquisition of expert performance: A general overview. \u003cem\u003eAcad. Emerg. Med.\u003c/em\u003e \u003cb\u003e15\u003c/b\u003e, 988\u0026ndash;994 (2008).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKahneman, D. \u0026amp; Klein, G. Conditions for intuitive expertise: A failure to disagree. \u003cem\u003eAm. Psychol.\u003c/em\u003e \u003cb\u003e64\u003c/b\u003e, 515\u0026ndash;526 (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCabitza, F., Rasoini, R. \u0026amp; Gensini, G. F. Unintended consequences of machine learning in medicine. \u003cem\u003eJAMA\u003c/em\u003e \u003cb\u003e318\u003c/b\u003e, 517\u0026ndash;518 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNeumann, F. J. et al. ESC/EACTS guidelines on myocardial revascularization. \u003cem\u003eEur. Heart J.\u003c/em\u003e \u003cb\u003e40\u003c/b\u003e, 87\u0026ndash;165 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOtto, C. M. et al. ACC/AHA guideline for the management of patients with valvular heart disease: Executive summary. \u003cem\u003eCirculation\u003c/em\u003e \u003cb\u003e143\u003c/b\u003e, e35\u0026ndash;e71 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShortliffe, E. H. \u0026amp; Sep\u0026uacute;lveda, M. J. Clinical decision support in the era of artificial intelligence. \u003cem\u003eJAMA\u003c/em\u003e \u003cb\u003e320\u003c/b\u003e, 2199\u0026ndash;2200 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuyatt, G. et al. GRADE guidelines: Introduction\u0026mdash;GRADE evidence profiles and summary of findings tables. \u003cem\u003eJ. Clin. Epidemiol.\u003c/em\u003e \u003cb\u003e64\u003c/b\u003e, 383\u0026ndash;394 (2011).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Artificial intelligence, Large language models, Cardiovascular surgery, Clinical decision-making, Clinical appropriateness","lastPublishedDoi":"10.21203/rs.3.rs-8793975/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8793975/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003e Large Language Models (LLMs) have demonstrated strong performance in standardized medical examinations; however, their reliability in real-world surgical decision-making, particularly in complex and guideline-ambiguous clinical scenarios, remains uncertain. Cardiovascular surgery, characterized by high risk, complex comorbidities, and frequent reliance on individualized clinical judgment, represents a critical domain for examining how artificial intelligence (AI)\u0026ndash;generated recommendations are appraised by clinicians operating in high-risk, judgment-intensive settings. Accordingly, this study aimed to present an expert appraisal of the perceived clinical appropriateness of GPT-5.2\u0026ndash;generated recommendations as evaluated by cardiovascular surgeons, and to examine whether appraisal varies according to surgeon experience and scenario type, including guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) decision-making contexts.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003e Fifty original cardiovascular surgical scenarios\u0026mdash;covering coronary, valvular, aortic, and peripheral vascular diseases\u0026mdash;were developed, including approximately 20% predefined guideline-ambiguous (\u0026ldquo;grey-zone\u0026rdquo;) cases. All scenarios were processed using GPT-5.2 with a standardized system instruction and deterministic settings (temperature 0.0; top_p 1.0), with web browsing enabled; each scenario was run in a separate session to minimize contextual carryover. Fourteen actively practicing cardiovascular surgeons independently evaluated the AI-generated recommendations using an anchored 5-point Likert scale capturing perceived clinical appropriateness rather than objective correctness or patient-level outcomes, with scores of 4\u0026ndash;5 prespecified as indicating clinically appropriate management. Surgeons were stratified into Early-Career (1\u0026ndash;9 years), Mid-Career (10\u0026ndash;19 years), and Senior (\u0026ge;\u0026thinsp;20 years) groups. Differences in ratings were analyzed using linear mixed-effects models accounting for repeated measures and cross-classified clustering at both surgeon and scenario levels.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eAcross 700 individual evaluations, AI-generated recommendations received a high overall mean rating of 4.31\u0026thinsp;\u0026plusmn;\u0026thinsp;0.8, with 86% of responses classified as clinically appropriate (Likert score 4\u0026ndash;5). Ratings did not differ significantly between Early- and Mid-Career surgeons (p\u0026thinsp;=\u0026thinsp;0.750), whereas Senior surgeons assigned lower scores than the Mid-Career group (mean difference\u0026thinsp;\u0026minus;\u0026thinsp;0.27, p\u0026thinsp;=\u0026thinsp;0.001). Grey-zone scenarios received lower ratings overall (mean difference\u0026thinsp;\u0026minus;\u0026thinsp;0.19, p\u0026thinsp;=\u0026thinsp;0.023). Sensitivity analyses using ordinal proportional odds regression yielded directionally consistent results.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eIn this surgeon-based appraisal study, AI-generated recommendations demonstrated high perceived clinical appropriateness in cardiovascular surgical decision-making. Differences in ratings reflected an experience-dependent acceptability threshold, alongside an additive scenario-type effect, without evidence of Experience \u0026times; ScenarioType interaction. These findings suggest that large language models may function as complementary decision-support resources in cardiovascular surgery, while underscoring the continued importance of expert judgment in complex and individualized clinical scenarios.\u003c/p\u003e","manuscriptTitle":"Expert Appraisal of AI-Generated Recommendations in Cardiovascular Surgical Decision-Making: A Multisurgeon Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-19 12:06:05","doi":"10.21203/rs.3.rs-8793975/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-04-17T21:28:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"6353996771738383354782653605055642434","date":"2026-04-09T13:08:26+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-09T11:32:46+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-08T05:55:54+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-02-12T19:20:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-07T08:24:41+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2026-02-07T08:14:48+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1b490a5c-e192-4039-b6c5-ae84c6ae1ac0","owner":[],"postedDate":"April 19th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":66494393,"name":"Health sciences/Cardiology"},{"id":66494394,"name":"Health sciences/Diseases"},{"id":66494395,"name":"Health sciences/Health care"},{"id":66494396,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-04-19T12:06:05+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-19 12:06:05","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8793975","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8793975","identity":"rs-8793975","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}