Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians

Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians Shelly Soffer, Dafna Nesselroth, Keren Pragier, Roi Anteby, Donald Apakama, and 15 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5382879/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Importance. Medical ethics is inherently complex, shaped by a broad spectrum of opinions, experiences, and cultural perspectives. The integration of large language models (LLMs) in healthcare is new and requires an understanding of their consistent adherence to ethical standards. Objective. To compare the agreement rates in answering questions based on ethically ambiguous situations between three frontier LLMs (GPT-4, Gemini-pro-1.5, and Llama-3-70b) and a multi-disciplinary physician group. Methods. In this cross-sectional study, three LLMs generated 1,248 medical ethics questions. These questions were derived based on the principles outlined in the American College of Physicians Ethics Manual. The topics spanned traditional, inclusive, interdisciplinary, and contemporary themes. Each model was then tasked in answering all generated questions. Twelve practicing physicians evaluated and responded to a randomly selected 10% subset of these questions. We compared agreement rates in question answering among the physicians, between the physicians and LLMs, and among LLMs. Results. The models generated a total of 3,744 answers. Despite physicians perceiving the questions' complexity as moderate, with scores between 2 and 3 on a 5-point scale, their agreement rate was only 55.9%. The agreement between physicians and LLMs was also low at 57.9%. In contrast, the agreement rate among LLMs was notably higher at 76.8% (p < 0.001), emphasizing the consistency in LLM responses compared to both physician-physician and physician-LLM agreement. Conclusions. LLMs demonstrate higher agreement rates in ethically complex scenarios compared to physicians, suggesting their potential utility as consultants in ambiguous ethical situations. Future research should explore how LLMs can enhance consistency while adapting to the complexities of real-world ethical dilemmas. Humanities/Complex networks Physical sciences/Mathematics and computing Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction Ethics in medicine, focusing on moral goodness, badness, right, and wrong 1 , varies significantly depending on societal and contextual factors 2 – 4 . While ethicists can navigate these complexities, physicians, with variable experience, face challenges in making decisions in ambiguous and intricate ethical dilemmas 5 – 7 . The emergence of large language models (LLMs) like GPT-4 8,9 , Gemini 10 , and Llama-3 11 has revolutionized language processing and generation in medicine 12 , 13 , prompting an evaluation of their performance in ethically ambiguous situations. Empathy and ethical reasoning have traditionally been viewed as uniquely human traits 14 – 16 . However, recent studies indicate that AI can mimic these attributes, delivering empathetic and ethically reasoned responses to patient inquiries 9 , 17 . Systematic approaches have demonstrated that ethical reasoning can be integrated into clinical practice, raising the question of whether LLMs can be similarly trained to handle ethical issues 18 . This study aims to explore the consistency of LLMs in navigating ethical ambiguities compared to medical professionals. By tasking LLMs with formulating complex ethical questions and comparing the agreement rates among the models and between the models and physicians, this research seeks to evaluate the potential of LLMs as consultants in ethical decision-making within healthcare. METHODS Study Design This cross-sectional study evaluated the performance of three LLMs—GPT-4, Gemini-pro-1.5, and Llama-3-70b—in generating and responding to complex medical ethics questions. The study compared the agreement rates among these models and between the models and a group of twelve practicing physicians. The study design is illustrated in Fig. 1 . The research project was approved by the Mount Sinai Institutional Review Board (IRB). Prompt Design for Question Creation To generate complex ethical questions, we crafted detailed prompts for each LLM, targeting the production of ethically ambiguous, and challenging scenarios. Each prompt directed the models to create multiple-choice questions based on specific instructions. We derived topics and sub-topics from the American College of Physicians Ethics Manual 19 , which outlines a comprehensive framework of six subjects and 52 sub-subjects ( Supplementary eTable 1) . We targeted four categories—traditional, inclusive, interdisciplinary, and contemporary—to address a broad spectrum of issues in medical ethics and professionalism 2 , 19 . We refined each prompt through iterative engagements with the three LLMs, ensuring the generation of questions that were both ethically ambiguous and complex. Following the prompt design, each LLM was tasked with generating two questions for each of the four categories across all 52 sub-subjects outlined in the American College of Physicians Ethics Manual. This approach resulted in the production of 416 questions per model. The complete prompt with the JSON directive and formatting is in Supplementary eFigure 1 . Prompt Design for Answering and Reasoning To assess LLMs' capacity to analyze and respond to the generated ethical questions, we presented them with a second prompt. This prompt required the models to answer the questions, articulate their reasoning, and identify the ethical principles guiding their responses. Additionally, the LLMs evaluated each question on dimensions of complexity, novelty, inclusiveness, significance, and relevancy, and categorized each by medical topic. The complete font with the JSON directive and formatting is presented in Supplementary eFigure 2. The final version of the prompt used for question generation and the prompt used for answering and reasoning are displayed in Fig. 2 A-B. Analysis on Agreement Rates We analyzed responses from each LLM to questions they generated as well as to those generated by other models (LLM-LLM). We specifically examined the extent of agreement, comparing responses across different subjects, sub-subjects, and question types. LLM-LLM agreement calculation was limited to each LLM answering the other two LLMs questions. Human Comparison To compare the ethical reasoning of the models with that of human experts, 12 physicians independently answered a random 10% sample of the questions generated by each model (amounting to 125 questions per physician). To prevent bias, the physicians were blinded to the identity of the LLMs. We examined the level of agreement among the physicians themselves (physician-physician) as well as the agreement between the physicians' answers and those provided by the models (physician-LLM). Additionally, the physicians classified the questions for complexity, novelty, relevance, significance and inclusiveness. The physicians specialize in various medical fields and come from different backgrounds and countries ( Supplementary eTable 2) . The prompt for the physicians is presented in the Supplementary eFigure 3 . Experimental Setup We conducted GPT-4 experiments through the Mount Sinai Hospital (MSH) Azure tenant API using gpt-4-0125-preview. We conducted Gemini experiments through Google's API setup using Gemini-pro-1.5 version. In contrast, inferences for Llama-3 were performed on a local cluster equipped with 4xH100 GPUs, utilizing Llama-3-70b-instruct 16-bit version. We used default hyper-parameters (temperature, top-k, max-length, etc.) for all models. We utilized Python (3.10) for data analyses. We used several Python libraries to facilitate data processing, model interaction, and analysis: NumPy (1.26.4) for numerical computations, Pandas (2.1.4) for data manipulation, Scikit-Learn (1.3.0) for statistical analysis, Hugging Face's Transformers (4.37.2) and torch (2.2.2 + cu121) for accessing pre-trained NLP models, and the json module (2.0.9) for handling JSON data formats. Statistical and Error Analysis For the analysis of binary data, Chi-square tests were used to compare response frequencies. Ordinal data were examined using Kruskal-Wallis tests. P-values less than 0.05 were considered statistically significant. We conducted a qualitative error analysis to pinpoint the sources of disagreement among the models and among physicians. This involved a review of conflicting responses, categorizing the underlying reasons for these discrepancies. Our examination focused on ambiguity, oversimplification of complex issues, and lack of contextual detail in generalizations. We further explored how each model prioritized different ethical principles and identified patterns of disagreement across specific sub-subjects and question types. This analysis elucidated key factors contributing to the inconsistencies observed in ethical reasoning. RESULTS Each model produced a distinct set of ethical questions, resulting in a total of 1248 questions across all models (416 for each model). These questions were answered by the model and then cross-evaluated, with each model answering questions generated by itself and the other models (a total of 3744 answers). Figures 3 A-C present examples of generated questions and the model's answers. A table with all questions and answers by the different models is provided in eTable 2 . Analysis of Consensus in Models' Responses Model agreements in answering ethical questions are presented in Fig. 4 and Supplementary eTable 3 . Agreement scores ranged from 70.7% for GPT-4 answering Gemini-generated questions to 84.4% for Llama-3 answering its own questions. Each model generally demonstrated higher agreement when answering its own questions, with GPT-4 at 81.2%, Gemini at 81.5%, and Llama-3 at 84.4%. However, cross-model agreements were lower, such as 70.7% for GPT-4 on Gemini questions and 75.0% for Gemini on GPT-4 questions. The overall LLM-LLM agreement rate (for each LLM answering other LLMs questions) was 76.8%. Models' Agreement by Subject, Sub-subject, and Type Supplementary eTable 4 presents the percentage of agreement across various subjects. Agreement ranged from 83.9% in "The Physician's Relationship to Other Clinicians" to 71.1% in "The Ethics of Practice" (p = 0.003). Supplementary eTable 5 illustrates the percentage of agreement among LLMs across various medical ethical sub-subjects. The lowest agreement was observed in "Placebo Controls" at 55.6%, followed by "Conflicts of Interest" at 61.1%, and "Informed Decision Making and Consent" as well as "Physician-Assisted Suicide and Euthanasia," both at 63.9%. Conversely, the sub-subjects with the highest agreement included "Decisions About Reproduction" at 91.7% and "Expert Witnesses" at 93.1%. The agreements across different question types are presented in Supplementary eTable 6 . Analysis of Question Characteristics All models classified the questions as inclusive and significant ( Supplementary eTable 7 ). The complexity and relevance grades given by the models are presented in Supplementary eTable 8. Analysis of Ethical Principles and Reasoning The analysis conducted by the models revealed a high level of agreement on key ethical principles. Beneficence, autonomy, justice, and non-maleficence were the most prominent principles identified by the models. Other frequently cited principles included confidentiality, respect for persons, professionalism, privacy, informed consent, and transparency. Supplement eFigure 4A-C show co-occurrence graphs that present the accordance of the top ten frequent principles for each model. Classification of Questions by Medical Topics Each model tagged the questions to a specific group, which were then grouped into 10 broad categories. The categories and their frequency are presented in the Supplementary eTable 9 . Agreement proportions did not differ between the groups. Of note, some specific topics were recurrent, with transgender issues being particularly prominent accounting for approximately 35% of the inclusiveness questions. The GPT-4 model generated the most questions related to transgender topics, with a total of 41 unique questions, followed by the Gemini model with 35 questions and the Llama-3 model with 31 questions. Physician Assessment of Model-Generated Questions The novelty, inclusiveness, significance, complexity, and relevance grades rated by physicians across the different models are detailed in Table 1 . All models produced questions with high significance (over 90%) and moderate complexity (scores between 2 and 3 out of 5). About half of the questions were novel, and roughly 60% were rated as inclusive. Table 1 Novelty, inclusiveness, significance, complexity and relevance grades rated by physicians across the different models Metrics GPT-4 Gemini Llama-3 P value Significance 452/492 (91.9%) 461/492 (93.7%) 461/492 (93.7%) 0.454 Complexity 3.0 (2.0–4.0) 2.0 (1.0–3.0) 3.0 (2.0–4.0) < 0.001 Novelty 240/492 (48.8%) 202/492 (41.1%) 248/492 (50.4%) < 0.006 Inclusiveness 313/492 (63.6%) 297/492 (60.4%) 283/492 (57.5%) 0.132 Relevance 3.0 (2.0–4.0) 3.0 (2.0–4.0) 3.0 (2.0–4.0) 0.012 Analysis of Physician Consensus and Physician-Model Agreement Physician-physician agreement was significantly lower than LLM-LLM agreement (55.9% vs. 76.8%, p < 0.001), highlighting considerable variability among human physicians. Figure 5 A presents a heatmap of the percentage of agreement among physicians. The overall physician-LLM agreement (57.9%) was also significantly lower than LLM-LLM agreement (76.8%) (p < 0.001). Figure 5 B illustrates the agreement levels between each physician and the three models. The overall physician-LLMs agreement rates across different models ranged from 54.9% (physician-Llama-3) to 61.8% (physician-Gemini), with no statistically significant differences among the models ( Supplementary eTable 10 ). A table with the questions and answers by all physicians is provided in Supplementary Excel File 2 . Error Analysis and Discrepancy Evaluation Several factors contributed to the observed disagreements among the models and physicians: Ambiguity and lack of prioritization : Many disagreements arose when reasoning included multiple ethical principles without clearly indicating how to balance them. For example, GPT-4 often incorporated a broad range of principles, including equity and patient-centered care, but struggled to prioritize these when they conflicted with each other. Oversimplification of complex issues : Simplifying complex ethical dilemmas without considering all relevant factors and principles led to differing interpretations. Llama-3, for instance, tended to emphasize cultural sensitivity and respect for autonomy, which sometimes oversimplified the nuanced requirements of specific medical contexts. Generalization : Providing general reasoning without addressing specific contextual details and nuances resulted in varied conclusions. Gemini frequently emphasized traditional principles like autonomy and non-maleficence without sufficiently considering the unique aspects of each scenario. Specific sub-subjects with high disagreement rates included "Conflicts of Interest" ( Examples: Supplementary Excel File 1; rows 1068, 2318) , "Informed Decision Making and Consent" ( Examples: Supplementary Excel File 1; rows 2954, 2955 ), “Sexual Contact Between Physician and Patient" ( Examples: Supplementary Excel File 1; row 3024 ), and "Physician-Assisted Suicide and Euthanasia" ( Examples: Supplementary Excel File 1; rows 613, 3529 ). Interdisciplinary (n = 90) and contemporary (n = 78) questions exhibited the highest rates of disagreement among models. For example, GPT-4's diverse topic range, including transgender health and cognitive impairments, often led to differing ethical interpretations compared to Gemini's more traditional focus and Llama-3's culturally sensitive approach ( Examples: Supplementary Excel File 1; rows 1541, 458 ). Similarly, human physicians showed varied interpretations on interdisciplinary and contemporary issues, such as collaboration with social workers on patient care plans, involving legal advisors in decisions for patients with diminished capacity, and areas like genetic testing and its impact on patient privacy (Examples: Supplementary Excel File 1; row 2919) . Another recurring pattern of disagreement involved scenarios where physicians consulted ethics committees, legal advisors, or peers versus making decisions independently ( Examples: Supplementary Excel File 1; rows 3, 19, 241, 458, 896 ). In cases like a Jehovah's Witness refusing a blood transfusion or disclosing a pregnancy, models varied between respecting patient autonomy and seeking external intervention. This included actions such as contacting the courts to request a legal order or informing child protective services. The analysis of the dataset reveals instances where there was a consensus among LLMs on the correct answers, yet significant disagreement persisted among humans. For example, in questions addressing patient autonomy and end-of-life care, the LLMs uniformly selected answers that aligned with established ethical principles such as autonomy and beneficence. However, human experts exhibited divergent views, reflecting their nuanced interpretations and the influence of personal, cultural, and professional experiences on ethical decision-making ( Examples: Supplementary Excel File 1; rows 550, 840 ). Another notable example of the divergence between LLMs and human arises in the context of medical confidentiality. In scenarios where LLMs uniformly agreed on the ethical course of action—such as maintaining strict confidentiality in cases involving adolescent patients seeking advice on sexual health—human experts displayed a range of opinions ( Examples: Supplementary Excel File 1; row 435 ). Some experts argued for exceptions to confidentiality based on the potential risks to the patient or others, while others adhered strictly to the principle of patient autonomy. DISCUSSION LLMs, such as GPT-4, Gemini-pro-1.5, and Llama-3-70b, demonstrated higher agreement rates in ethically ambiguous situations compared to physicians. Specifically, the agreement rates among LLMs were significantly higher (76.8%) than those among physicians (55.9%), and the between LLMs and physicians (57.9%). This suggests that LLMs adhere more consistently to established ethical principles like beneficence and autonomy, whereas physicians exhibit a broader range of interpretations influenced by personal, cultural, and professional backgrounds. Previous research supports the notion that LLMs can provide ethical guidance that is perceived as highly virtuous, intelligent, and trustworthy, and in some cases, superior to humans 9 , 20 – 22 . Our study extends this understanding by evaluating the ability of LLMs to generate and respond to complex ethical dilemmas consistent with the American College of Physicians Ethics Manual 19 . The high significance ratings above 90%, novelty ratings of about 50%, and inclusiveness ratings of approximately 60%, underscore LLMs ability to produce pertinent and thought-provoking content. Although complexity scores were judged as moderate by humans, this contrasts with the low agreement rate among physicians on answers, which suggests non-trivial questions. Rao et al. argue against aligning LLMs with specific ethical principles, advocating instead for the infusion of generic ethical reasoning capabilities 21 . They found that while LLMs like GPT-4 can demonstrate nearly perfect ethical reasoning, they still exhibit biases towards Western values. Beneficence, autonomy, justice, and non-maleficence, the principles predominantly identified by the models’ reasoning, are emphasized in Western bioethics, particularly through principlism, which is prevalent in Western medical ethics education and practice 5 . Due to the lack of transparency in LLMs, understanding their ethical decision-making process and their ability to account for diverse cultural nuances remains a significant challenge. The comparison with human experts underscored the intricacies of ethical decision-making. Practicing physicians, diverse in cultural, demographics, disciplines, and training backgrounds, regularly confront complex medical ethics issues. In our study, disagreement among 12 physicians were significant, above 40%, paralleling the high discord observed between the models and the physicians. This indicates that ethical reasoning in medical contexts is profoundly subjective, influenced by cultural and personal factors and biases 23 , 24 . Such variability among physicians highlights the complex nature of medical ethics and the inherent difficulties in reaching a consensus. The better uniformity in algorithmic responses, compared to the diverse human perspectives, suggests that algorithms may excel in capturing the deep context embedded by ethical thought leaders. This consideration invites a nuanced reflection: if ethical principles, articulated by experts, are open to interpretation, could algorithms not serve as accurate consultants? By being meticulously context-aware and with careful training to reduce bias, algorithms could potentially offer consistent consultations on established ethical frameworks. This is not to claim brilliance on the part of the algorithms, but rather to highlight their capacity for accurate and considered interpretation, qualities that are especially valuable in the high-stakes, variably interpreted realm of medical ethics. Conversely, the disagreement between humans might emphasize the concept of value pluralism , which acknowledges that multiple, sometimes conflicting, values can be equally correct and fundamental 4 , 25 . This perspective is crucial in medical ethics, where decisions often involve balancing diverse and competing values. Even if LLMs can create ethical consistency using the knowledge of ethics experts, it does not prevent the possibility of moral dilemmas, as sometimes multiple actions that ought to be done cannot be accomplished simultaneously 26 . Our study highlights the importance of equipping LLMs with the ability to navigate this complexity, ensuring they can handle a variety of ethical frameworks and cultural contexts without defaulting to a single moral stance. This study has several limitations. Firstly, we used only three LLMs, whereas many more are available, potentially offering diverse insights. Secondly, the multiple-choice question format may not fully capture the intricacies and nuances of real-world ethical practice. Furthermore, the ethical scenarios we presented were hypothetical, potentially lacking the dynamic nature of medical situations. This limitation is critical as the high disagreement rate among humans in these simplified scenarios suggests that real-world complexities could pose even greater challenges for physicians. Our focus was on generating and responding to questions rather than engaging in continuous ethical discourse, which might limit the depth of ethical analysis. Additionally, our sample size was limited to 12 physicians, which does not adequately represent the global medical community's diversity. Moreover, we limited the exploration to default hyper-parameters set in the models' API calls. We also did not experiment with techniques such as fine-tuning or retrieval augmented generation (RAG), opting instead to consider "out-of-the-box" performance 13 , 27 . Lastly, the evaluation of responses relied on a diverse group of physicians, and not ethical experts – however, this represents real-world setting, where most physicians are not ethical experts. In conclusion, LLMs exhibit significantly higher in-silico agreement and consistency rates as those of physicians in difficult ethical dilemmas, suggesting their potential as consultants in ambiguous ethical scenarios. However, the value pluralism inherent in human decisions, which often balance diverse and conflicting stances, presents a challenge. While LLMs can streamline ethical consistency, they must also adapt to the complexities of moral dilemmas involving multiple valid actions. This juxtaposition invites careful consideration and ongoing dialogue about how technology might enhance, yet respect, the multifaceted nature of medical ethics. Declarations Acknowledgment Conflict of Interest Disclosures. Joshua Lampert declares a consulting role with Viz.ai. All other authors report no conflicts of interest related to this study. Data Access, Responsibility, and Analysis . Dr. Eyal Klang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Data Sharing Statement. Data supporting the findings of this study are included within the article and its supplementary materials. Funding. This work was supported in part by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Author Contribution Shelly Soffer, Benjamin S Glicksberg, Eyal Klang, and Girish N Nadkarni planned the research. Shelly Soffer, Eyal Klang, and Girish N Nadkarni wrote the manuscript. Eyal Klang conducted the analysis and statistical work. Moran Gendler planned and designed Figure 1 and also designed Figures 4 and 5A-B. Data collection was carried out by Dafna Nesselroth, Keren Pragier, Roi Anteby, Donald Apakama, Emma Holmes, Ashwin Shreekant Sawant, Ethan Abbott, Lauren Alyse Lepow, Ishita Vasudev, Joshua Lampert, and Nir Horesh. All authors edited the manuscript.All authors reviewed and approved the final manuscript. Data Availability Data is provided within the manuscript or supplementary information files References Kant I. The Metaphysics of Morals. 1785. Practical Philosophy. 1996:353–604. Parsa-Parsi RW. The International Code of Medical Ethics of the World Medical Association. Jama. Oct 13 2022. Inglehart R, Welzel C. The WVS cultural map of the world. World Values Survey. 2010. Lo B, Malina D, Pittman G, Morrissey S. Fundamentals of Medical Ethics—A New Perspective Series. Vol 389: Mass Medical Soc; 2023:2392–2394. Gillon R. Medical ethics: four principles plus attention to scope. Bmj. Jul 16 1994;309(6948):184–188. Varkey B. Principles of Clinical Ethics and Their Application to Practice. Medical principles and practice: international journal of the Kuwait University, Health Science Centre. 2021;30(1):17–28. DuVal G, Clarridge B, Gensler G, Danis M. A national survey of US internists’ experiences with ethical dilemmas and ethics consultation. Journal of general internal medicine. 2004;19:251–258. Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023. Lahat A, Sharif K, Zoabi N, et al. Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4. Journal of medical Internet research. 2024;26:e54571. Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023. Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. 2023. Soroush A, Glicksberg BS, Zimlichman E, et al. Large Language Models Are Poor Medical Coders—Benchmarking of Medical Code Querying. NEJM AI. 2024;1(5):AIdbp2300040. Glicksberg BS, Timsina P, Patel D, Sawant A. Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room. May 21 2024. Hendelman W, Byszewski A. Formation of medical student professional identity: categorizing lapses of professionalism, and the learning environment. BMC medical education. 2014;14:1–10. Decety J, Cowell JM. Friends or foes: Is empathy necessary for moral behavior? Perspectives on psychological science. 2014;9(5):525–537. Parviainen J, Rantala J. Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care. Medicine, Health Care and Philosophy. 2022;25(1):61–71. Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. Jun 1 2023;183(6):589–596. Kaldjian LC, Weir RF, Duffy TP. A clinician's approach to clinical ethical reasoning. J Gen Intern Med. Mar 2005;20(3):306–311. Sulmasy LS, Bledsoe TA. American College of Physicians Ethics Manual: Seventh Edition. Annals of internal medicine. Jan 15 2019;170(2_Suppl):S1-s32. Dillion D, Mondal D, Tandon N, Gray K. Large Language Models as Moral Experts? GPT-4o Outperforms Expert Ethicist in Providing Moral Guidance. Rao A, Khandelwal A, Tanmay K, Agarwal U, Choudhury M. Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs. arXiv preprint arXiv:2310.07251. 2023. Aharoni E, Fernandes S, Brady DJ, et al. Attributions toward artificial agents in a modified Moral Turing Test. Scientific reports. 2024;14(1):8458. Genuis SJ, Lipp C. Ethical diversity and the role of conscience in clinical medicine. International Journal of Family Medicine. 2013;2013(1):587541. Mashayekhi J, Mafinejad MK, Changiz T, et al. Exploring medical ethics’ implementation challenges: A qualitative study. Journal of education and health promotion. 2021;10(1). James W. The moral philosopher and the moral life. The International Journal of Ethics. 1891;1(3):330–354. Williams B. Ethical consistency. Essays on moral realism. 1988:41–58. Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems. 2020;33:9459–9474. Additional Declarations No competing interests reported. Supplementary Files SupplementaryFile1.xlsx SupplementaryFile2.xlsx SupplementaryOnlinecontant.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5382879","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":376680263,"identity":"d0e257bf-ea04-4b70-9c22-20f47042ca7e","order_by":0,"name":"Shelly Soffer","email":"","orcid":"","institution":"Rabin Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Shelly","middleName":"","lastName":"Soffer","suffix":""},{"id":376680264,"identity":"7488bf38-8fb8-4320-b13a-67af02d85cdc","order_by":1,"name":"Dafna Nesselroth","email":"","orcid":"","institution":"Meuhedet Health Services","correspondingAuthor":false,"prefix":"","firstName":"Dafna","middleName":"","lastName":"Nesselroth","suffix":""},{"id":376680265,"identity":"2cea592f-0802-442b-b9f0-01614798f68c","order_by":2,"name":"Keren Pragier","email":"","orcid":"","institution":"Maccabi Institute for Health Services Research","correspondingAuthor":false,"prefix":"","firstName":"Keren","middleName":"","lastName":"Pragier","suffix":""},{"id":376680266,"identity":"d25c6a58-370f-4030-9672-bbcd804e2f25","order_by":3,"name":"Roi Anteby","email":"","orcid":"","institution":"Massachusetts General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Roi","middleName":"","lastName":"Anteby","suffix":""},{"id":376680267,"identity":"1422f82f-a2d5-4487-a547-5630a96b581e","order_by":4,"name":"Donald Apakama","email":"","orcid":"","institution":"Mount Sinai Hospital","correspondingAuthor":false,"prefix":"","firstName":"Donald","middleName":"","lastName":"Apakama","suffix":""},{"id":376680268,"identity":"5bfff2cd-9605-4325-a78a-b4dadcc2acef","order_by":5,"name":"Emma Holmes","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Emma","middleName":"","lastName":"Holmes","suffix":""},{"id":376680269,"identity":"63904a6a-6fa0-4437-bbb0-4f1858615768","order_by":6,"name":"Ashwin Shreekant Sawant","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Ashwin","middleName":"Shreekant","lastName":"Sawant","suffix":""},{"id":376680270,"identity":"6d052e86-b048-4937-a2d5-d16b817856bc","order_by":7,"name":"Ethan Abbott","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Ethan","middleName":"","lastName":"Abbott","suffix":""},{"id":376680271,"identity":"f5bc1609-a0dc-44e0-87ca-157bc2766fb8","order_by":8,"name":"Lauren Alyse Lepow","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Lauren","middleName":"Alyse","lastName":"Lepow","suffix":""},{"id":376680272,"identity":"7adce9dc-0f72-40c8-9040-b9082618fafd","order_by":9,"name":"Ishita Vasudev","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Ishita","middleName":"","lastName":"Vasudev","suffix":""},{"id":376680273,"identity":"d25daffb-7ad6-4d7a-ac5f-72e008445be9","order_by":10,"name":"Joshua Lampert","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Joshua","middleName":"","lastName":"Lampert","suffix":""},{"id":376680274,"identity":"96e83924-2018-463e-89c5-0d994b1e2563","order_by":11,"name":"Moran Gendler","email":"","orcid":"","institution":"Azrieli Faculty of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Moran","middleName":"","lastName":"Gendler","suffix":""},{"id":376680275,"identity":"dabcaf5b-edd5-4e3f-a2f6-9f5a807f29ef","order_by":12,"name":"Nir Horesh","email":"","orcid":"","institution":"Cleveland Clinic Florida","correspondingAuthor":false,"prefix":"","firstName":"Nir","middleName":"","lastName":"Horesh","suffix":""},{"id":376680276,"identity":"62669efd-62ae-44a3-b072-6afcb4b6b4cb","order_by":13,"name":"Orly Efros","email":"","orcid":"","institution":"Sheba Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Orly","middleName":"","lastName":"Efros","suffix":""},{"id":376680277,"identity":"d515dd87-7225-4ca5-ae72-9f48ba3df019","order_by":14,"name":"Benjamin S Glicksberg","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Benjamin","middleName":"S","lastName":"Glicksberg","suffix":""},{"id":376680278,"identity":"25dc79a2-df29-4f69-a0de-cb93f5134b11","order_by":15,"name":"Robert Freeman","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Robert","middleName":"","lastName":"Freeman","suffix":""},{"id":376680279,"identity":"e7b1f78b-f689-421f-9d42-81b2bb191c83","order_by":16,"name":"David L Reich","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"L","lastName":"Reich","suffix":""},{"id":376680280,"identity":"0a8a887e-95be-4e11-95b3-7676ab7690ba","order_by":17,"name":"Alexander W Charney","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Alexander","middleName":"W","lastName":"Charney","suffix":""},{"id":376680281,"identity":"156355cd-ecfc-4781-a870-91c1e73bbdb9","order_by":18,"name":"Girish N Nadkarni","email":"","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":false,"prefix":"","firstName":"Girish","middleName":"N","lastName":"Nadkarni","suffix":""},{"id":376680283,"identity":"335b4e37-8353-4a57-9774-92b2443de49a","order_by":19,"name":"Eyal Klang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGklEQVRIiWNgGAWjYBAC+wYGxgOMbRZAJvMBiBAzECfg0WIAxAcYeySAFBtYnQSRWmaAtPAYQLQQAgYSyQ8OMG6QkDfnX/N1w8e2O3Xy7bwHPzxgsMmXd8DhF4k0gwOMDyQMd854u+3mzLZnEgaH+ZIlEhjSLDcewGVLAlgL44YbZ7fd5m07LGHAzAMUZDhsYNiAS0v6B5DD7DfcOPMMrEW+mcf4B34tOQYg7yduON/DBtbCcJjHDGyLPC7v87wpOJDYI5G84Qab2c0Z5w5LbgBqsUgwSDMwwKHFvj1944OPbTa2G84ffnbjQ9lhfvn+M8Y3f1TYGMjjcBiDQAI04iQSUGwHogM4tPAfQGfAAE5bRsEoGAWjYKQBAGEWYqzah+uHAAAAAElFTkSuQmCC","orcid":"","institution":"Icahn School of Medicine at Mount Sinai","correspondingAuthor":true,"prefix":"","firstName":"Eyal","middleName":"","lastName":"Klang","suffix":""}],"badges":[],"createdAt":"2024-11-03 16:08:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5382879/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5382879/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":69073188,"identity":"0574c74e-ecff-4531-9bda-e0f2c96de13c","added_by":"auto","created_at":"2024-11-15 10:25:21","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":138070,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of the Study Design Comparing Agreement Rates on Medical Ethics Questions Between Large Language Models and Physicians.\u003c/p\u003e\n\u003cp\u003eThree large language models (LLMs) generated 1,248 ethically complex questions across four categories: traditional, interdisciplinary, inclusive, and contemporary. Each model answered all questions, resulting in 3,744 responses. Twelve physicians each answered a random 10% subset of these questions (125 responses per physician). Agreement rates were calculated for three comparisons: LLM versus LLM, physician versus physician, and physician versus LLM. The LLMs demonstrated a higher agreement rate (76.8%) compared to physician-physician (55.9%) and physician-LLM (57.9%) agreement rates.\u003c/p\u003e","description":"","filename":"Figure1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/906958c050038dd2cd1872bf.jpg"},{"id":69073185,"identity":"9897044c-fbdf-4af0-8f70-e7fcb81b9a52","added_by":"auto","created_at":"2024-11-15 10:25:21","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1039752,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003e(A) \u003c/strong\u003ePrompt for question creation and \u003cstrong\u003e(B) \u003c/strong\u003eprompt for answering and reasoning\u003c/p\u003e","description":"","filename":"Figure2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/5d8509bd4e1f6a8e8450cedd.jpg"},{"id":69072041,"identity":"b622cc87-3a6f-46fc-9ba3-c1ff25dcc1db","added_by":"auto","created_at":"2024-11-15 10:17:21","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1284471,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eA-C. \u003c/strong\u003eExamples of generated questions and the model's answers\u003c/p\u003e","description":"","filename":"Figure3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/0e78b509f6c556dab3274222.jpg"},{"id":69072035,"identity":"9917048d-b419-4de9-b4cc-736ac2941ae8","added_by":"auto","created_at":"2024-11-15 10:17:21","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":43214,"visible":true,"origin":"","legend":"\u003cp\u003eHeatmap describing the agreement across models.\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/28c7c986297e531ce1330480.png"},{"id":69072036,"identity":"e987b871-abb7-4b75-9812-63d9a5fad581","added_by":"auto","created_at":"2024-11-15 10:17:21","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":272991,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003e(A) \u003c/strong\u003eHeatmap illustrating the agreement among physicians regarding their responses to the LLMs medical ethical questions. \u003cstrong\u003e(B)\u003c/strong\u003e Heatmap demonstrating the agreement between individual physicians and the three different LLMs: GPT-4, Gemini, and Llama-3.\u003c/p\u003e","description":"","filename":"Figure5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/8ae2cfae558b7491b1f7d48b.jpg"},{"id":85525825,"identity":"2647e3aa-4a5f-4f82-8d7d-4303b3e6acfe","added_by":"auto","created_at":"2025-06-27 00:16:28","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3982955,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/1d7e1d2d-ec71-4449-9955-d21b4af6888e.pdf"},{"id":69073186,"identity":"b8625952-7516-4d68-951f-d938963f710a","added_by":"auto","created_at":"2024-11-15 10:25:21","extension":"xlsx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1154271,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryFile1.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/4292bb3736dab5cf5d8bdcb6.xlsx"},{"id":69073443,"identity":"8b389419-b714-4f3c-bdb5-d46037d07b80","added_by":"auto","created_at":"2024-11-15 10:33:21","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":437281,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryFile2.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/7384ece6fba752224286f2cf.xlsx"},{"id":69072040,"identity":"4d7df9e1-7635-4ef8-bd09-dcf0b4176bde","added_by":"auto","created_at":"2024-11-15 10:17:21","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":875767,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryOnlinecontant.docx","url":"https://assets-eu.researchsquare.com/files/rs-5382879/v1/c51d1b421b29667414dbe7f1.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians","fulltext":[{"header":"Introduction","content":"\u003cp\u003eEthics in medicine, focusing on moral goodness, badness, right, and wrong \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e, varies significantly depending on societal and contextual factors \u003csup\u003e\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. While ethicists can navigate these complexities, physicians, with variable experience, face challenges in making decisions in ambiguous and intricate ethical dilemmas \u003csup\u003e\u003cspan additionalcitationids=\"CR6\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. The emergence of large language models (LLMs) like GPT-4 \u003csup\u003e8,9\u003c/sup\u003e, Gemini \u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e, and Llama-3 \u003csup\u003e11\u003c/sup\u003e has revolutionized language processing and generation in medicine \u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e, prompting an evaluation of their performance in ethically ambiguous situations.\u003c/p\u003e \u003cp\u003eEmpathy and ethical reasoning have traditionally been viewed as uniquely human traits \u003csup\u003e\u003cspan additionalcitationids=\"CR15\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. However, recent studies indicate that AI can mimic these attributes, delivering empathetic and ethically reasoned responses to patient inquiries \u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. Systematic approaches have demonstrated that ethical reasoning can be integrated into clinical practice, raising the question of whether LLMs can be similarly trained to handle ethical issues \u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThis study aims to explore the consistency of LLMs in navigating ethical ambiguities compared to medical professionals. By tasking LLMs with formulating complex ethical questions and comparing the agreement rates among the models and between the models and physicians, this research seeks to evaluate the potential of LLMs as consultants in ethical decision-making within healthcare.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy Design\u003c/h2\u003e \u003cp\u003eThis cross-sectional study evaluated the performance of three LLMs\u0026mdash;GPT-4, Gemini-pro-1.5, and Llama-3-70b\u0026mdash;in generating and responding to complex medical ethics questions. The study compared the agreement rates among these models and between the models and a group of twelve practicing physicians. The study design is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The research project was approved by the Mount Sinai Institutional Review Board (IRB).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003ePrompt Design for Question Creation\u003c/h3\u003e\n\u003cp\u003eTo generate complex ethical questions, we crafted detailed prompts for each LLM, targeting the production of ethically ambiguous, and challenging scenarios. Each prompt directed the models to create multiple-choice questions based on specific instructions. We derived topics and sub-topics from the American College of Physicians Ethics Manual \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e, which outlines a comprehensive framework of six subjects and 52 sub-subjects (\u003cb\u003eSupplementary eTable 1)\u003c/b\u003e.\u003c/p\u003e \u003cp\u003eWe targeted four categories\u0026mdash;traditional, inclusive, interdisciplinary, and contemporary\u0026mdash;to address a broad spectrum of issues in medical ethics and professionalism \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. We refined each prompt through iterative engagements with the three LLMs, ensuring the generation of questions that were both ethically ambiguous and complex.\u003c/p\u003e \u003cp\u003eFollowing the prompt design, each LLM was tasked with generating two questions for each of the four categories across all 52 sub-subjects outlined in the American College of Physicians Ethics Manual. This approach resulted in the production of 416 questions per model.\u003c/p\u003e \u003cp\u003eThe complete prompt with the JSON directive and formatting is in \u003cb\u003eSupplementary eFigure 1\u003c/b\u003e.\u003c/p\u003e\n\u003ch3\u003ePrompt Design for Answering and Reasoning\u003c/h3\u003e\n\u003cp\u003eTo assess LLMs' capacity to analyze and respond to the generated ethical questions, we presented them with a second prompt. This prompt required the models to answer the questions, articulate their reasoning, and identify the ethical principles guiding their responses. Additionally, the LLMs evaluated each question on dimensions of complexity, novelty, inclusiveness, significance, and relevancy, and categorized each by medical topic. The complete font with the JSON directive and formatting is presented in \u003cb\u003eSupplementary eFigure 2.\u003c/b\u003e\u003c/p\u003e \u003cp\u003eThe final version of the prompt used for question generation and the prompt used for answering and reasoning are displayed in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eA-B.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003eAnalysis on Agreement Rates\u003c/h3\u003e\n\u003cp\u003eWe analyzed responses from each LLM to questions they generated as well as to those generated by other models (LLM-LLM). We specifically examined the extent of agreement, comparing responses across different subjects, sub-subjects, and question types. LLM-LLM agreement calculation was limited to each LLM answering the other two LLMs questions.\u003c/p\u003e\n\u003ch3\u003eHuman Comparison\u003c/h3\u003e\n\u003cp\u003eTo compare the ethical reasoning of the models with that of human experts, 12 physicians independently answered a random 10% sample of the questions generated by each model (amounting to 125 questions per physician). To prevent bias, the physicians were blinded to the identity of the LLMs. We examined the level of agreement among the physicians themselves (physician-physician) as well as the agreement between the physicians' answers and those provided by the models (physician-LLM). Additionally, the physicians classified the questions for complexity, novelty, relevance, significance and inclusiveness.\u003c/p\u003e \u003cp\u003eThe physicians specialize in various medical fields and come from different backgrounds and countries (\u003cb\u003eSupplementary eTable 2)\u003c/b\u003e. The prompt for the physicians is presented in the \u003cb\u003eSupplementary eFigure 3\u003c/b\u003e.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eExperimental Setup\u003c/h2\u003e \u003cp\u003eWe conducted GPT-4 experiments through the Mount Sinai Hospital (MSH) Azure tenant API using gpt-4-0125-preview. We conducted Gemini experiments through Google's API setup using Gemini-pro-1.5 version. In contrast, inferences for Llama-3 were performed on a local cluster equipped with 4xH100 GPUs, utilizing Llama-3-70b-instruct 16-bit version. We used default hyper-parameters (temperature, top-k, max-length, etc.) for all models. We utilized Python (3.10) for data analyses. We used several Python libraries to facilitate data processing, model interaction, and analysis: NumPy (1.26.4) for numerical computations, Pandas (2.1.4) for data manipulation, Scikit-Learn (1.3.0) for statistical analysis, Hugging Face's Transformers (4.37.2) and torch (2.2.2\u0026thinsp;+\u0026thinsp;cu121) for accessing pre-trained NLP models, and the json module (2.0.9) for handling JSON data formats.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eStatistical and Error Analysis\u003c/h3\u003e\n\u003cp\u003eFor the analysis of binary data, Chi-square tests were used to compare response frequencies. Ordinal data were examined using Kruskal-Wallis tests. P-values less than 0.05 were considered statistically significant. We conducted a qualitative error analysis to pinpoint the sources of disagreement among the models and among physicians. This involved a review of conflicting responses, categorizing the underlying reasons for these discrepancies. Our examination focused on ambiguity, oversimplification of complex issues, and lack of contextual detail in generalizations. We further explored how each model prioritized different ethical principles and identified patterns of disagreement across specific sub-subjects and question types. This analysis elucidated key factors contributing to the inconsistencies observed in ethical reasoning.\u003c/p\u003e"},{"header":"RESULTS","content":"\u003cp\u003eEach model produced a distinct set of ethical questions, resulting in a total of 1248 questions across all models (416 for each model). These questions were answered by the model and then cross-evaluated, with each model answering questions generated by itself and the other models (a total of 3744 answers). Figures\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eA-C present examples of generated questions and the model's answers. A table with all questions and answers by the different models is provided in \u003cb\u003eeTable 2\u003c/b\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eAnalysis of Consensus in Models' Responses\u003c/h2\u003e \u003cp\u003eModel agreements in answering ethical questions are presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e \u003cb\u003eand Supplementary eTable 3\u003c/b\u003e. Agreement scores ranged from 70.7% for GPT-4 answering Gemini-generated questions to 84.4% for Llama-3 answering its own questions. Each model generally demonstrated higher agreement when answering its own questions, with GPT-4 at 81.2%, Gemini at 81.5%, and Llama-3 at 84.4%. However, cross-model agreements were lower, such as 70.7% for GPT-4 on Gemini questions and 75.0% for Gemini on GPT-4 questions. The overall LLM-LLM agreement rate (for each LLM answering other LLMs questions) was 76.8%.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eModels' Agreement by Subject, Sub-subject, and Type\u003c/h2\u003e \u003cp\u003e \u003cb\u003eSupplementary eTable 4\u003c/b\u003e presents the percentage of agreement across various subjects. Agreement ranged from 83.9% in \"The Physician's Relationship to Other Clinicians\" to 71.1% in \"The Ethics of Practice\" (p\u0026thinsp;=\u0026thinsp;0.003).\u003c/p\u003e \u003cp\u003e \u003cb\u003eSupplementary eTable 5\u003c/b\u003e illustrates the percentage of agreement among LLMs across various medical ethical sub-subjects. The lowest agreement was observed in \"Placebo Controls\" at 55.6%, followed by \"Conflicts of Interest\" at 61.1%, and \"Informed Decision Making and Consent\" as well as \"Physician-Assisted Suicide and Euthanasia,\" both at 63.9%. Conversely, the sub-subjects with the highest agreement included \"Decisions About Reproduction\" at 91.7% and \"Expert Witnesses\" at 93.1%. The agreements across different question types are presented in \u003cb\u003eSupplementary eTable 6\u003c/b\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eAnalysis of Question Characteristics\u003c/h2\u003e \u003cp\u003eAll models classified the questions as inclusive and significant (\u003cb\u003eSupplementary eTable 7\u003c/b\u003e). The complexity and relevance grades given by the models are presented in \u003cb\u003eSupplementary eTable 8.\u003c/b\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eAnalysis of Ethical Principles and Reasoning\u003c/h2\u003e \u003cp\u003eThe analysis conducted by the models revealed a high level of agreement on key ethical principles. Beneficence, autonomy, justice, and non-maleficence were the most prominent principles identified by the models. Other frequently cited principles included confidentiality, respect for persons, professionalism, privacy, informed consent, and transparency. \u003cb\u003eSupplement eFigure 4A-C\u003c/b\u003e show co-occurrence graphs that present the accordance of the top ten frequent principles for each model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eClassification of Questions by Medical Topics\u003c/h2\u003e \u003cp\u003eEach model tagged the questions to a specific group, which were then grouped into 10 broad categories. The categories and their frequency are presented in the \u003cb\u003eSupplementary eTable 9\u003c/b\u003e. Agreement proportions did not differ between the groups.\u003c/p\u003e \u003cp\u003eOf note, some specific topics were recurrent, with transgender issues being particularly prominent accounting for approximately 35% of the inclusiveness questions. The GPT-4 model generated the most questions related to transgender topics, with a total of 41 unique questions, followed by the Gemini model with 35 questions and the Llama-3 model with 31 questions.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003ePhysician Assessment of Model-Generated Questions\u003c/h2\u003e \u003cp\u003eThe novelty, inclusiveness, significance, complexity, and relevance grades rated by physicians across the different models are detailed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. All models produced questions with high significance (over 90%) and moderate complexity (scores between 2 and 3 out of 5). About half of the questions were novel, and roughly 60% were rated as inclusive.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eNovelty, inclusiveness, significance, complexity and relevance grades rated by physicians across the different models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetrics\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGPT-4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGemini\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLlama-3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eP value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSignificance\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e452/492 (91.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e461/492 (93.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e461/492 (93.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.454\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eComplexity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3.0 (2.0\u0026ndash;4.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2.0 (1.0\u0026ndash;3.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e3.0 (2.0\u0026ndash;4.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNovelty\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e240/492 (48.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e202/492 (41.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e248/492 (50.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.006\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusiveness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e313/492 (63.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e297/492 (60.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e283/492 (57.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.132\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRelevance\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3.0 (2.0\u0026ndash;4.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.0 (2.0\u0026ndash;4.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e3.0 (2.0\u0026ndash;4.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.012\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eAnalysis of Physician Consensus and Physician-Model Agreement\u003c/h2\u003e \u003cp\u003ePhysician-physician agreement was significantly lower than LLM-LLM agreement (55.9% vs. 76.8%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), highlighting considerable variability among human physicians. Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eA presents a heatmap of the percentage of agreement among physicians.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe overall physician-LLM agreement (57.9%) was also significantly lower than LLM-LLM agreement (76.8%) (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eB illustrates the agreement levels between each physician and the three models.\u003c/p\u003e \u003cp\u003eThe overall physician-LLMs agreement rates across different models ranged from 54.9% (physician-Llama-3) to 61.8% (physician-Gemini), with no statistically significant differences among the models (\u003cb\u003eSupplementary eTable 10\u003c/b\u003e).\u003c/p\u003e \u003cp\u003eA table with the questions and answers by all physicians is provided in \u003cb\u003eSupplementary Excel File 2\u003c/b\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eError Analysis and Discrepancy Evaluation\u003c/h2\u003e \u003cp\u003eSeveral factors contributed to the observed disagreements among the models and physicians:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eAmbiguity and lack of prioritization\u003c/em\u003e: Many disagreements arose when reasoning included multiple ethical principles without clearly indicating how to balance them. For example, GPT-4 often incorporated a broad range of principles, including equity and patient-centered care, but struggled to prioritize these when they conflicted with each other.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eOversimplification of complex issues\u003c/em\u003e: Simplifying complex ethical dilemmas without considering all relevant factors and principles led to differing interpretations. Llama-3, for instance, tended to emphasize cultural sensitivity and respect for autonomy, which sometimes oversimplified the nuanced requirements of specific medical contexts.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eGeneralization\u003c/em\u003e: Providing general reasoning without addressing specific contextual details and nuances resulted in varied conclusions. Gemini frequently emphasized traditional principles like autonomy and non-maleficence without sufficiently considering the unique aspects of each scenario.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eSpecific sub-subjects with high disagreement rates included \"Conflicts of Interest\" (\u003cb\u003eExamples: Supplementary Excel File 1; rows 1068, 2318)\u003c/b\u003e, \"Informed Decision Making and Consent\" (\u003cb\u003eExamples: Supplementary Excel File 1; rows 2954, 2955\u003c/b\u003e), \u0026ldquo;Sexual Contact Between Physician and Patient\" (\u003cb\u003eExamples: Supplementary Excel File 1; row 3024\u003c/b\u003e), and \"Physician-Assisted Suicide and Euthanasia\" (\u003cb\u003eExamples: Supplementary Excel File 1; rows 613, 3529\u003c/b\u003e).\u003c/p\u003e \u003cp\u003eInterdisciplinary (n\u0026thinsp;=\u0026thinsp;90) and contemporary (n\u0026thinsp;=\u0026thinsp;78) questions exhibited the highest rates of disagreement among models. For example, GPT-4's diverse topic range, including transgender health and cognitive impairments, often led to differing ethical interpretations compared to Gemini's more traditional focus and Llama-3's culturally sensitive approach (\u003cb\u003eExamples: Supplementary Excel File 1; rows 1541, 458\u003c/b\u003e). Similarly, human physicians showed varied interpretations on interdisciplinary and contemporary issues, such as collaboration with social workers on patient care plans, involving legal advisors in decisions for patients with diminished capacity, and areas like genetic testing and its impact on patient privacy \u003cb\u003e(Examples: Supplementary Excel File 1; row 2919)\u003c/b\u003e.\u003c/p\u003e \u003cp\u003eAnother recurring pattern of disagreement involved scenarios where physicians consulted ethics committees, legal advisors, or peers versus making decisions independently (\u003cb\u003eExamples: Supplementary Excel File 1; rows 3, 19, 241, 458, 896\u003c/b\u003e). In cases like a Jehovah's Witness refusing a blood transfusion or disclosing a pregnancy, models varied between respecting patient autonomy and seeking external intervention. This included actions such as contacting the courts to request a legal order or informing child protective services.\u003c/p\u003e \u003cp\u003eThe analysis of the dataset reveals instances where there was a consensus among LLMs on the correct answers, yet significant disagreement persisted among humans. For example, in questions addressing patient autonomy and end-of-life care, the LLMs uniformly selected answers that aligned with established ethical principles such as autonomy and beneficence. However, human experts exhibited divergent views, reflecting their nuanced interpretations and the influence of personal, cultural, and professional experiences on ethical decision-making (\u003cb\u003eExamples: Supplementary Excel File 1; rows 550, 840\u003c/b\u003e). Another notable example of the divergence between LLMs and human arises in the context of medical confidentiality. In scenarios where LLMs uniformly agreed on the ethical course of action\u0026mdash;such as maintaining strict confidentiality in cases involving adolescent patients seeking advice on sexual health\u0026mdash;human experts displayed a range of opinions (\u003cb\u003eExamples: Supplementary Excel File 1; row 435\u003c/b\u003e). Some experts argued for exceptions to confidentiality based on the potential risks to the patient or others, while others adhered strictly to the principle of patient autonomy.\u003c/p\u003e \u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eLLMs, such as GPT-4, Gemini-pro-1.5, and Llama-3-70b, demonstrated higher agreement rates in ethically ambiguous situations compared to physicians. Specifically, the agreement rates among LLMs were significantly higher (76.8%) than those among physicians (55.9%), and the between LLMs and physicians (57.9%). This suggests that LLMs adhere more consistently to established ethical principles like beneficence and autonomy, whereas physicians exhibit a broader range of interpretations influenced by personal, cultural, and professional backgrounds.\u003c/p\u003e \u003cp\u003ePrevious research supports the notion that LLMs can provide ethical guidance that is perceived as highly virtuous, intelligent, and trustworthy, and in some cases, superior to humans \u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e,\u003cspan additionalcitationids=\"CR21\" citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. Our study extends this understanding by evaluating the ability of LLMs to generate and respond to complex ethical dilemmas consistent with the American College of Physicians Ethics Manual \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. The high significance ratings above 90%, novelty ratings of about 50%, and inclusiveness ratings of approximately 60%, underscore LLMs ability to produce pertinent and thought-provoking content. Although complexity scores were judged as moderate by humans, this contrasts with the low agreement rate among physicians on answers, which suggests non-trivial questions.\u003c/p\u003e \u003cp\u003eRao et al. argue against aligning LLMs with specific ethical principles, advocating instead for the infusion of generic ethical reasoning capabilities \u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e. They found that while LLMs like GPT-4 can demonstrate nearly perfect ethical reasoning, they still exhibit biases towards Western values. Beneficence, autonomy, justice, and non-maleficence, the principles predominantly identified by the models\u0026rsquo; reasoning, are emphasized in Western bioethics, particularly through principlism, which is prevalent in Western medical ethics education and practice \u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Due to the lack of transparency in LLMs, understanding their ethical decision-making process and their ability to account for diverse cultural nuances remains a significant challenge.\u003c/p\u003e \u003cp\u003eThe comparison with human experts underscored the intricacies of ethical decision-making. Practicing physicians, diverse in cultural, demographics, disciplines, and training backgrounds, regularly confront complex medical ethics issues. In our study, disagreement among 12 physicians were significant, above 40%, paralleling the high discord observed between the models and the physicians. This indicates that ethical reasoning in medical contexts is profoundly subjective, influenced by cultural and personal factors and biases \u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e,\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. Such variability among physicians highlights the complex nature of medical ethics and the inherent difficulties in reaching a consensus.\u003c/p\u003e \u003cp\u003eThe better uniformity in algorithmic responses, compared to the diverse human perspectives, suggests that algorithms may excel in capturing the deep context embedded by ethical thought leaders. This consideration invites a nuanced reflection: if ethical principles, articulated by experts, are open to interpretation, could algorithms not serve as accurate consultants? By being meticulously context-aware and with careful training to reduce bias, algorithms could potentially offer consistent consultations on established ethical frameworks. This is not to claim brilliance on the part of the algorithms, but rather to highlight their capacity for accurate and considered interpretation, qualities that are especially valuable in the high-stakes, variably interpreted realm of medical ethics.\u003c/p\u003e \u003cp\u003eConversely, the disagreement between humans might emphasize the concept of \u003cem\u003evalue pluralism\u003c/em\u003e, which acknowledges that multiple, sometimes conflicting, values can be equally correct and fundamental \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e. This perspective is crucial in medical ethics, where decisions often involve balancing diverse and competing values. Even if LLMs can create \u003cem\u003eethical consistency\u003c/em\u003e using the knowledge of ethics experts, it does not prevent the possibility of moral dilemmas, as sometimes multiple actions that ought to be done cannot be accomplished simultaneously \u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e. Our study highlights the importance of equipping LLMs with the ability to navigate this complexity, ensuring they can handle a variety of ethical frameworks and cultural contexts without defaulting to a single moral stance.\u003c/p\u003e \u003cp\u003eThis study has several limitations. Firstly, we used only three LLMs, whereas many more are available, potentially offering diverse insights. Secondly, the multiple-choice question format may not fully capture the intricacies and nuances of real-world ethical practice. Furthermore, the ethical scenarios we presented were hypothetical, potentially lacking the dynamic nature of medical situations. This limitation is critical as the high disagreement rate among humans in these simplified scenarios suggests that real-world complexities could pose even greater challenges for physicians. Our focus was on generating and responding to questions rather than engaging in continuous ethical discourse, which might limit the depth of ethical analysis. Additionally, our sample size was limited to 12 physicians, which does not adequately represent the global medical community's diversity. Moreover, we limited the exploration to default hyper-parameters set in the models' API calls. We also did not experiment with techniques such as fine-tuning or retrieval augmented generation (RAG), opting instead to consider \"out-of-the-box\" performance \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e,\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e. Lastly, the evaluation of responses relied on a diverse group of physicians, and not ethical experts \u0026ndash; however, this represents real-world setting, where most physicians are not ethical experts.\u003c/p\u003e \u003cp\u003eIn conclusion, LLMs exhibit significantly higher in-silico agreement and consistency rates as those of physicians in difficult ethical dilemmas, suggesting their potential as consultants in ambiguous ethical scenarios. However, the value pluralism inherent in human decisions, which often balance diverse and conflicting stances, presents a challenge. While LLMs can streamline ethical consistency, they must also adapt to the complexities of moral dilemmas involving multiple valid actions. This juxtaposition invites careful consideration and ongoing dialogue about how technology might enhance, yet respect, the multifaceted nature of medical ethics.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003e \u003cb\u003eAcknowledgment\u003c/b\u003e \u003c/h2\u003e \u003cp\u003e \u003cstrong\u003eConflict of Interest Disclosures.\u003c/strong\u003e \u003cp\u003eJoshua Lampert declares a consulting role with Viz.ai. All other authors report no conflicts of interest related to this study.\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003eData Access, Responsibility, and Analysis\u003c/h2\u003e \u003cp\u003e. Dr. Eyal Klang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eData Sharing Statement.\u003c/strong\u003e \u003cp\u003e Data supporting the findings of this study are included within the article and its supplementary materials.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding.\u003c/h2\u003e \u003cp\u003eThis work was supported in part by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eShelly Soffer, Benjamin S Glicksberg, Eyal Klang, and Girish N Nadkarni planned the research. Shelly Soffer, Eyal Klang, and Girish N Nadkarni wrote the manuscript. Eyal Klang conducted the analysis and statistical work. Moran Gendler planned and designed Figure 1 and also designed Figures 4 and 5A-B. Data collection was carried out by Dafna Nesselroth, Keren Pragier, Roi Anteby, Donald Apakama, Emma Holmes, Ashwin Shreekant Sawant, Ethan Abbott, Lauren Alyse Lepow, Ishita Vasudev, Joshua Lampert, and Nir Horesh. All authors edited the manuscript.All authors reviewed and approved the final manuscript.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData is provided within the manuscript or supplementary information files\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eKant I. The Metaphysics of Morals. 1785. Practical Philosophy. 1996:353\u0026ndash;604.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eParsa-Parsi RW. The International Code of Medical Ethics of the World Medical Association. Jama. Oct 13 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInglehart R, Welzel C. The WVS cultural map of the world. World Values Survey. 2010.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLo B, Malina D, Pittman G, Morrissey S. Fundamentals of Medical Ethics\u0026mdash;A New Perspective Series. Vol 389: Mass Medical Soc; 2023:2392\u0026ndash;2394.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGillon R. Medical ethics: four principles plus attention to scope. Bmj. Jul 16 1994;309(6948):184\u0026ndash;188.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVarkey B. Principles of Clinical Ethics and Their Application to Practice. Medical principles and practice: international journal of the Kuwait University, Health Science Centre. 2021;30(1):17\u0026ndash;28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDuVal G, Clarridge B, Gensler G, Danis M. A national survey of US internists\u0026rsquo; experiences with ethical dilemmas and ethics consultation. Journal of general internal medicine. 2004;19:251\u0026ndash;258.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAchiam J, Adler S, Agarwal S, et al. Gpt-4 technical report. \u003cem\u003earXiv preprint arXiv:2303.08774.\u003c/em\u003e 2023.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLahat A, Sharif K, Zoabi N, et al. Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4. Journal of medical Internet research. 2024;26:e54571.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTeam G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models. \u003cem\u003earXiv preprint arXiv:2312.11805.\u003c/em\u003e 2023.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTouvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models. \u003cem\u003earXiv preprint arXiv:2302.13971.\u003c/em\u003e 2023.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSoroush A, Glicksberg BS, Zimlichman E, et al. Large Language Models Are Poor Medical Coders\u0026mdash;Benchmarking of Medical Code Querying. NEJM AI. 2024;1(5):AIdbp2300040.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGlicksberg BS, Timsina P, Patel D, Sawant A. Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room. May 21 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHendelman W, Byszewski A. Formation of medical student professional identity: categorizing lapses of professionalism, and the learning environment. BMC medical education. 2014;14:1\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDecety J, Cowell JM. Friends or foes: Is empathy necessary for moral behavior? Perspectives on psychological science. 2014;9(5):525\u0026ndash;537.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eParviainen J, Rantala J. Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care. Medicine, Health Care and Philosophy. 2022;25(1):61\u0026ndash;71.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAyers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. Jun 1 2023;183(6):589\u0026ndash;596.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaldjian LC, Weir RF, Duffy TP. A clinician's approach to clinical ethical reasoning. J Gen Intern Med. Mar 2005;20(3):306\u0026ndash;311.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSulmasy LS, Bledsoe TA. American College of Physicians Ethics Manual: Seventh Edition. Annals of internal medicine. Jan 15 2019;170(2_Suppl):S1-s32.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDillion D, Mondal D, Tandon N, Gray K. Large Language Models as Moral Experts? GPT-4o Outperforms Expert Ethicist in Providing Moral Guidance.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRao A, Khandelwal A, Tanmay K, Agarwal U, Choudhury M. Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs. \u003cem\u003earXiv preprint arXiv:2310.07251.\u003c/em\u003e 2023.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAharoni E, Fernandes S, Brady DJ, et al. Attributions toward artificial agents in a modified Moral Turing Test. Scientific reports. 2024;14(1):8458.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGenuis SJ, Lipp C. Ethical diversity and the role of conscience in clinical medicine. International Journal of Family Medicine. 2013;2013(1):587541.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMashayekhi J, Mafinejad MK, Changiz T, et al. Exploring medical ethics\u0026rsquo; implementation challenges: A qualitative study. Journal of education and health promotion. 2021;10(1).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJames W. The moral philosopher and the moral life. The International Journal of Ethics. 1891;1(3):330\u0026ndash;354.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilliams B. Ethical consistency. Essays on moral realism. 1988:41\u0026ndash;58.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems. 2020;33:9459\u0026ndash;9474.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5382879/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5382879/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eImportance.\u003c/h2\u003e \u003cp\u003eMedical ethics is inherently complex, shaped by a broad spectrum of opinions, experiences, and cultural perspectives. The integration of large language models (LLMs) in healthcare is new and requires an understanding of their consistent adherence to ethical standards.\u003c/p\u003e\u003ch2\u003eObjective.\u003c/h2\u003e \u003cp\u003eTo compare the agreement rates in answering questions based on ethically ambiguous situations between three frontier LLMs (GPT-4, Gemini-pro-1.5, and Llama-3-70b) and a multi-disciplinary physician group.\u003c/p\u003e\u003ch2\u003eMethods.\u003c/h2\u003e \u003cp\u003eIn this cross-sectional study, three LLMs generated 1,248 medical ethics questions. These questions were derived based on the principles outlined in the American College of Physicians Ethics Manual. The topics spanned traditional, inclusive, interdisciplinary, and contemporary themes. Each model was then tasked in answering all generated questions. Twelve practicing physicians evaluated and responded to a randomly selected 10% subset of these questions. We compared agreement rates in question answering among the physicians, between the physicians and LLMs, and among LLMs.\u003c/p\u003e\u003ch2\u003eResults.\u003c/h2\u003e \u003cp\u003eThe models generated a total of 3,744 answers. Despite physicians perceiving the questions' complexity as moderate, with scores between 2 and 3 on a 5-point scale, their agreement rate was only 55.9%. The agreement between physicians and LLMs was also low at 57.9%. In contrast, the agreement rate among LLMs was notably higher at 76.8% (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), emphasizing the consistency in LLM responses compared to both physician-physician and physician-LLM agreement.\u003c/p\u003e\u003ch2\u003eConclusions.\u003c/h2\u003e \u003cp\u003eLLMs demonstrate higher agreement rates in ethically complex scenarios compared to physicians, suggesting their potential utility as consultants in ambiguous ethical situations. Future research should explore how LLMs can enhance consistency while adapting to the complexities of real-world ethical dilemmas.\u003c/p\u003e","manuscriptTitle":"Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-15 10:17:16","doi":"10.21203/rs.3.rs-5382879/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d31eb0e3-9d98-4434-9d48-e8267895cb95","owner":[],"postedDate":"November 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":40096207,"name":"Humanities/Complex networks"},{"id":40096208,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2025-06-27T00:08:20+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-15 10:17:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5382879","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5382879","identity":"rs-5382879","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}