A qualitative Interview Study Investigating Patient, Health Professional, and Developer Perspectives on Real-World Implementation of Patient-Centered AI Systems

A qualitative Interview Study Investigating Patient, Health Professional, and Developer Perspectives on Real-World Implementation of Patient-Centered AI Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A qualitative Interview Study Investigating Patient, Health Professional, and Developer Perspectives on Real-World Implementation of Patient-Centered AI Systems Natalie Benda, Pooja Desai, Zayan Reza, Victoria Winogora, Uday Suresh, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7908218/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 13 You are reading this latest preprint version Abstract Our objective was to triangulate patient, health professional, and developer perspectives for implementing patient-centered artificial intelligence (AI) systems. We conducted semi-structured interviews with patients (N = 18), health professionals (N = 8), and AI developers (N = 8). We created interview guides informed by frameworks in bioethics and health information informatics. We utilized a predictive algorithm for determining risk for postpartum depression as a use case to concretize our discussions. Our team analyzed transcripts from interview recordings using thematic, directed content analysis and the constant comparative process. Participants found mitigating potential harms caused by AI (e.g., bias, stigma, or patient anxiety) greatly important. They also believed that AI must provide clinical benefits by allowing health professionals and patients to easily take actions based on AI output. To take safe action, end users needed transparency to understand the AI’s accuracy and predictors driving risk. Patient participants wanted health professionals to interpret AI output, but health professionals did not always feel they had the time or training to do so. Participants also raised concerns regarding how data quality may affect AI accuracy, who may be responsible for inappropriate actions taken based on AI, and issues regarding data security, privacy, and accessibility. Our results support real-world implementation of more patient-centered AI tools by: providing health professionals with competencies for discussing AI-based risks; engaging patients and health professionals throughout the development process; inclusively communicating AI output to health professionals and patients; and implementing multi-layer systems of AI governance. Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Scientific community and society/Scientific community Artificial intelligence predictive algorithms bioethics patient-centered care Figures Figure 1 INTRODUCTION The transition to artificial intelligence (AI) in healthcare continues to be significant for patients and health professionals. 1 – 4 Despite AI’s predictive and generative power, many systems are never implemented or, when implemented, struggle to show improvements in patient outcomes compared to the standard of care. 5 This clinical translation gap between AI development and real-world, successful deployment is defined as the “last mile” problem. The “last mile” problem is the fact that AI models may show high accuracy and promise in simulated lab-based tasks, but (1) they are either never deployed in clinical environments, or (2) they cannot improve outcomes in real-life due to deficiencies in implementation. A critical gap lies in end users seeing, accepting, and acting upon AI-based recommendations. 6 Previous research in human-AI interaction has focused heavily on health professionals as the sole recipients of AI recommendations, with less attention to implementing patient-centered AI systems. 7 Our previous pre-implementation work for AI-based systems highlighted the importance of keeping patients informed regarding AI-informed recommendations to justify interventions and to support uptake. 8 Patients and the public have important perspectives regarding AI implementation and want to be part of the conversation. 9 – 11 However, their perspectives on centering patients in AI system implementation have been understudied compared to health professionals and AI developers. 7 , 8 , 12 – 17 Moreover, previous studies have independently surveyed patient, health professional, and developer preferences about AI implementation. No studies to our knowledge, however, have collected richer qualitative data and combined their collective insights. Consequently, we lack integrative, practical guidance for those seeking to ethically implement and evaluate patient-centered AI systems. It is also becoming more likely that patients may access AI output, even if not intended by AI developers. The U.S. 21st Century Cures Act, for example, prevents information blocking from patients, requiring organizations to give patients access to their electronic health information without delay or expense. 18 This may result in a patient seeing a predictive AI risk score before discussing it with their healthcare team. AI is also increasingly used in tools to support shared decision-making with patients. 19 While AI output may benefit patients, it may cause stress if not appropriately contextualized. AI can also be prone to hallucinations, the creation of false information, which may be challenging to differentiate from factual evidence. Our objective was to triangulate perspectives from patients, developers, and health professionals for real-world implementation of patient-centered AI systems. We interviewed these three stakeholder groups using interview guides informed by bioethics and informatics models. 20 To concretize discussions, we used an existing predictive AI tool developed by our team that proactively predicts a pregnant patient’s risk of postpartum depression (PPD) as a use case. Recruited patients and health professionals were potential users of the tool. PPD presents a complex bioethical case because of the sensitivity of the data and for the multi-layer autonomy; in pregnancy, the autonomy, harms, and benefits to the perinatal patient, newborn or fetus, and partner must be weighed simultaneously. 11 , 12 Our study provides novel, data-driven recommendations for how AI may be ethically implemented to support patient-centred care. RESULTS Participant Characteristics Table 1 provides demographic data from each stakeholder group (N = 36). Table 1 Participants' demographics provided at the aggregate level and broken out by stakeholder group. Demographic characteristic Stakeholder Group Overall (N = 36), n(%) Developers (n = 10), n (%) Health Professionals (n = 8), n (%) Patients (n = 18), n (%) Sex Female 29 (81) 5 (50) 7 (87.5) 17 (94.5) Male 4 (11) 4 (40) 0 (0) 0 (0) Missing 3 (8) 1 (10) 1 (12.5) 1 (5.5) Race White 13 (36) 5 (50) 2 (25) 6 (33.5) Asian 6 (17) 4 (40) 2 (25) 0 (0) Black or African American 9 (25) 0 (0) 2 (25) 7 (39) Native Hawaiian or Other Pacific Islander 1 (3) 0 (0) 0 (0) 1 (5.5) More than One Race 2 (5.5) 0 (0) 0 (0) 2 (11) Prefer Not to Answer 2 (5.5) 0 (0) 1 (12.5) 1 (5.5) Missing 3 (8) 1 (10) 1 (12.5) 1 (5.5) Age (generation) Generation Z (1997–2012) 6 (17) 3 (30) 0 (0) 3 (17) Millennials (1981–1996) 23 (64) 5 (50) 5 (62.5) 13 (72) Generation X (1965–1980) 3 (8) 0 (0) 2 (25) 1 (5.5) Boomers (1946–1964) 1 (3) 1 (10) 0 (0) 0 (0) Missing 3 (8) 1 (10) 1 (12.5) 1 (5.5) Education Some College or Associate's Degree 6 (17) 0 (0) 0 (0) 6 (33.5) Bachelor's Degree 11 (31) 1 (10) 2 (25) 8 (44) Master's Degree 8 (22) 2 (20) 3 (37.5) 3 (17) . Doctorate Degree 8 (22) 6 (60) 2 (25) 0 (0) Missing 3 (8) 1 (10) 1 (12.5) 1 (5.5) Emerging themes We inductively derived emerging themes for implementing patient-centered AI systems that cut across our overarching framework for analysis, Sittig and Singh’s Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems (Table 2 ) 20 . We describe the detailed emerging themes with the most illustrative quotes from developers (DV), health professionals (HP), and patients (PT). Table 2 Emerging themes from our analysis and their intersection with the dimensions of Sittig and Singh's Sociotechnical framework. Sittig and Singh Dimension Emerging theme Software, hardware, and computing infrastructure Clinical content Human- computer interface Workflow and communication People Internal organisational policies, procedures and culture External rules, regulations and pressures System measurement and monitoring Mitigating harm X X X X X X Clinical benefit and utility X X X X X Communicating AI models X X X X X Data quality X X X Data security, privacy, and access X X X X Responsible AI practices and liability X X X X Mitigating harm All participant roles stressed the importance of ensuring minimal harm to end users (e.g., patients, health professionals) related to AI. One primary source of harm discussed involved stigmas driven by AI-based predictions. For example, predicting risk for mental health conditions or addictions (e.g., opioid use disorder) could change patient care trajectories. Patients expressed fear that risk predictions of mental health issues could lead to child protective services (CPS) interventions. I may run a 100 score and it’s like, ‘oh i passed’. Like, why is the doctor coming to talk to me? And ACS (local CPS department) is coming to take my child? - PT12 Participants also raised the concern about model bias, noting the importance of utilizing representative training data and valuing fairness equally to accuracy. I’m sure they’d be curious if their ethnicity is put in there…But what if there is some type of bias in the tool? - HP02 Patients and health professionals viewed AI as potentially harmful due to the stress that high-risk predictions may cause patients. Participants described the importance of involving health professionals and patients in the early stages of AI development to identify and mitigate these harms. The patient is going to develop that insight and be on board… and lead to like a more … patient-centered treatment. - DV08 Clinical benefit and utility All groups expressed that AI must provide clinical benefits to patients, care partners, health professionals, or health systems. Participants agreed that for models to provide utility, a health professional or patient must be able to easily take action (e.g., making a referral, prescribing a medication, contacting their provider) based on AI’s output. Patient and health professional participants seemed most comfortable with AI that could “run in the background” and provide support to human decision-makers as needed. If it can explain its reasoning, I feel like that's more helpful…I like to think of AI like as a tool. So if it was actually like teaching me and…I can learn what those risk factors are and take the time, then I would find it way more useful. - PT16 For the tool discussed (predictive AI for PPD), patient participants were typically open to their providers using the tool as an additional data point and accepting treatment based on the AI and their doctor's input. However, one patient was very averse to using the tool, indicating the preference-sensitive nature of AI use in clinical care. I would be confused because I wouldn’t believe that they could figure that out…it seems so bizarre. I probably would lose a bit of trust. - PT09 Translating predictive models into action also requires a threshold for action. Patient participants, however, had challenges understanding what the numeric risk predictions meant practically and why there were certain thresholds for action (e.g., the tool recommended a mental health referral if the PPD risk was > 30%). 0 to 30%, you know, that's a closer gap. And then you have like this gap from 32 to 100, but it's like 40, 50, 60, 70. And I'm assuming the prevention treatment is starting at either 75 or 80. And I was like, what about those people in the middle? - PT12 To trust AI and promote relevant actions, most end users wanted to understand the accuracy of AI. Ideally, patients wanted to know how accurate it was for them individually. However, some patients did not need to see this information themselves but trusted health professionals to make judgments on when AI was safe for clinical use. I assume that [if] it’s in my patient portal, I’m thinking that it’s accurate for me. - PT01 Developers noted a tension between high performance and making understandable models that people wanted to use. Interviewer: Accuracy versus understandability and explainability, and if you could only choose one, which one would you choose and why? DV01: We need both, you can’t really do it without both, but when forced…the majority of the crowd choose understandability. Participants described different strategies for ensuring models provided benefits. First, health professionals need to support clinical validation and interrogate potential biases. Participants also highlighted the need to investigate and plan for how AI may alter existing workflows. Updates would then need to be implemented in line with institutional norms and policies. Developers viewed implementation and ensuring utility as a continuous process that would require updates over time, similar to any traditional software. Developers also acknowledged the importance of end-user feedback throughout AI model development. Although they did not always feel they had the skills to include this feedback, particularly related to patient users. I think one piece that's definitely missing in my research is patient perspectives about whether they think this kind of tool would be useful…I think that those questions have to be asked very carefully…But I don't know that developers should be the ones actually going in to ask those questions. - DV07 Communicating AI Models Several questions emerged regarding if, when, where, and how AI-related information should be communicated to patients. Patients had differing perspectives regarding how they would like to view AI output, with some preferring to see it before clinical encounters and others wanting someone to explain it to them. Multiple participants agreed that patients would need support in interpreting information and suggested that the provider may be the best person to do this. However, they also noted that the provider may need training and dedicated time to do this effectively. Who gets the alert, and what happens once you get an alert? If you're going through all that, then how much time are you spending to explain it? - HP06 Others highlighted that interfaces should be presented in lay terms, accessible to different audiences. Feature importance was discussed at length - participants suggested conveying predictors of risk (e.g., medications, history of past illness) as both positives and negatives contributing to the patient's total risk. Patients generally wanted to see all features, not only the modifiable predictors. Providing sensitive features (e.g., risks related to marital status, BMI) requires care but may stimulate educational conversations. It was also important to ensure the labels were understandable by patients, so using the EHR-based feature names (e.g., “thyroid preparations”, “hyperemesis”) may be insufficient. Patients also reported varying opinions on their desire to see AI-based risk information. Some felt they had a right to see these risks, how accurate they were, and what their individual risk factors were. Others trusted their provider to see the information and make the related decisions. Data quality Participants described challenges related to ensuring data quality, primarily related to bias, missingness, out-of-date information, and shortcomings in interpreting unstructured data (e.g., clinical notes) with insufficient context. These days...charts are not well updated… unless it's thoughtfully updated, which it rarely is because hospital systems are far too busy for that kind of meticulousness, then I think that people are going to feel like I could see people being like, ‘where the hell did that come from?’ - HP03 Missingness was noted to be particularly problematic in a perinatal context, where many may receive care for other specialties in different health systems. Data security, privacy, and access Participants agreed that the way the data is stored, the algorithm is computed, and displayed to end users must protect patient and health professional privacy. There was, however, a tension between making tools open source and protecting patient privacy, ensuring their autonomy over their data. Many viewed external organizations, such as the Centers for Medicare and Medicaid, as having a responsibility for ensuring privacy (i.e., data de-identification) and also governing privacy rules, although it was not always clear as to which entity was responsible. Health organizations were also mentioned as having a role in supporting data privacy and promoting data access. There were other external organizations whose access to data (e.g., CPS/ACS) was viewed as detrimental. Patients had different desires for accessing AI inputs and outputs, including through the patient portal, on mobile phones, via PDF, and secure email. Some also wanted the ability to share the information with chosen care partners, but this varied from person to person. Others highlighted how the combination of perinatal and psychiatric concerns made privacy issues more sensitive. Some patients, for example, may be reluctant to disclose sensitive information due to stigma or fear, leading to data accuracy issues. Whenever a pregnant woman talks about this type of information...that concerns like calling to Child Protective Services. That is a big privacy issue...clinicians worry about models raising severe alerts. - DV09 Participants working in pediatrics also noted that accessing maternal health information through a child’s chart can be challenging due to limitations in data linkage capabilities and due to confidentiality protections in the EHR for sensitive maternal data. 21 Responsible AI practices and liability Participants had various opinions about who was responsible for the safe development and implementation of AI. Participants mentioned responsible entities at various levels from insurers and device manufacturers to individual developers and health professionals. There needs to be some responsible party who's well informed, who's making the decisions about if and how these tools are actually used and up to date on how well they're actually being used in patient care as you introduce them - DV08 Many of the questions of responsibility stemmed from a desire to mitigate individual liability for health professionals. The prevailing perspective was that no single person or entity could be held responsible, and a structure of shared responsibility was necessary. Some perceived that determining safety for the use of an algorithm should fall to health professionals. Health professionals also highlighted the need for responsibility in terms of seeing and acting upon risk predictions. Developers, on the other hand, felt responsible for making systems with high predictive performance that also mitigated potential biases. So a clinician needs to make an informed decision. In which case…the clinic is liable, as they would be for any misdiagnosis or incorrect treatment decision. - DV08 If we implement it then…somebody has to have ownership to it that you are responsible for this. And if the patient is seeing this information, somebody has to interpret for them. - HP06 Responsible practices mentioned included checking for bias and accuracy prior to implementation, as well as continuous monitoring to identify model drift and retrain models. The conversation surrounding ensuring accuracy could be challenging, however, as the gold standard for comparison may not always be clear. DISCUSSION AI implementation in health systems requires coordination between patients, clinicians, developers, and health systems administrators and leaders. Through interviews with nearly 40stakeholders representing these perspectives, we identified requirements implementation and use of patient-centered AI systems. We summarize our findings in preliminary requirements derived directly from stakeholders, shown in Table 3 . Our results specific to end user perspectives on conveying AI-based risk of PPD overlap with Williams et. al’s recent qualitative study on a similar topic. 22 However, our work extends upon their findings, incorporating developer perspectives and using PPD as a use case to make broader recommendations for patient-centered AI regardless of clinical context. Our findings also resonate with the recently released AI Code of Conduct for Health and Medicine from the National Academy of Medicine, related to: advancing benefit for humanity, ensuring equity, engaging impacted individuals, ensuring workforce wellbeing, monitoring performance, and supporting continuous learning. 23 We describe practical next steps with data-driven insights for achieving these requirements across key stakeholder groups. Table 3 Stakeholder-driven requirements for patient-centered AI. Topic Requirement description Related themes Health professional competencies related to AI Provide health professionals with the time and training to deliver AI-based risk information Mitigating harm, communicating AI, responsible AI practices Ethical creation of AI systems Engage health professionals and patient stakeholders throughout the development process Mitigating harm, clinical benefit and utility Effective communication of AI output for all stakeholders Promote actionability by displaying options for viewing model accuracy, predictors, and thresholds for action Mitigating harm, communicating AI, responsible AI practices Patient-centered AI communication Include flexible options for communicating AI output to patients to manage cultural preferences and varying literacy levels Mitigating harm, communicating AI, responsible AI practices AI governance Making ethical decisions related to governance issues, including privacy, access, liability, and regulation, will require multiple levels of stakeholder engagement Data security, privacy, and access; responsible AI practices and liability; AI regulation First, health professionals will require competencies to use AI effectively and safely in healthcare. 24 Most patients reported that they wanted their health professionals to deliver predictive AI results, whereas most clinicians do not feel qualified to do so. Health professionals also highlighted the difficulty in interpreting AI output and the limited time during encounters. While some efforts have attempted to integrate AI competencies into medical and nursing education, 24 far more work is needed to educate the existing workforce of health professionals. Second, ur findings highlight how decisions made during the development process can have downstream impacts, for example, balancing explainability with accuracy, or addressing fairness. Model developers, at times, work in relative isolation from the health system leadership, clinicians, and patients who may ultimately adopt their model. Technical development is viewed as a foundational step with little input needed from other stakeholders. Rather, our findings suggest that because of the downstream implications of decisions made at the modeling stage, health professional and patient stakeholders should be included in the early stages of development and evaluation to ensure models prioritize the right values, are useful, and ethically sound. Participants also raised issues related to how data completeness and timeliness may affect model accuracy. Similar to displaying significant model features, visualization may support end users in understanding when key data elements may be missing or outdated. A systematic review of 44 implemented predictive AI models highlighted the lack of attention to user workflow and providing predictive risks in a way that enhances rather than degrades cognitive processing. 12 Future work may investigate generalizable best practices for displaying predictive AI models to work collaboratively with human end users. Third, to provide clinical utility, predictive AI output needs to be actionable, as we have previously reported. 8 For example, perinatal providers need a way to refer appropriate patients for psychiatric or social work evaluation. Participants noted several needs to promote actionability, including providing reasoning (i.e., risk factors) and the level of model accuracy. This information would allow them to weigh their expertise in conjunction with the model, and choose from multiple possible actions based on the patient's risk factors. Patients wanted to understand individual risk factors, which could support shared decision-making conversations with providers regarding risk reduction. Participants also noted that meaningful thresholds for action based on the numeric risk prediction were needed. This is not done consistently in practice, and there is a lack of robust methods for determining these thresholds for actionability. Fourth, patients in our study expressed a wide range of preferences and information needs related to interpretation of the model. Compared to clinician end users, less attention has been given to patients as end users of models. There are many scenarios in which patients may view model output, making their perspectives vital in considering how to display output. The harms of failing to incorporate their perspectives may include elevated worry and anxiety, increased outreach to care teams, and even unnecessary care visits. Lastly, several other publications have provided detailed needs assessments for enhancing privacy, navigating liabilities, and devising regulations related to AI. 23,25–28 Our findings reinforce that layers of shared governance, involving multiple stakeholder groups, will be necessary to address these challenging questions. Our findings also suggest multiple important areas of future work. First, our study primarily focused on predictive AI models. Generative AI is being rapidly adopted in research and in healthcare settings. A recent systematic review identified over 200 studies that have used large language models (LLMs) for healthcare purposes, including patient education, recommendations, and guidance, customizing care plans, interpreting medical information, and facilitating patient-clinician communication. 29 They reported positive findings relating to patient satisfaction but concerns about readability, accuracy, and bias. Thus, investigating patients’ needs and ability to use LLMs safely will be an important part of ensuring appropriate use in healthcare. Additionally, the science for both predictive and generative AI has weighed heavily towards model development and validation, with significantly fewer studies evaluating real-world clinical implementation and health outcomes. Multiple perspective pieces have called for more implementation science studies, 30 along with clinical trials that have thoughtful comparator arms and benchmarks representing the current state of the healthcare system. 31 While improving clinical outcomes is the ultimate goal for most predictive AI systems, recent studies have highlighted that only measuring these outcomes may produce an incomplete picture, particularly when AI does not have the intended effect. 32 We advocate for detailed tracking (e.g., via EHR log data) that may ascertain how AI impacts decisions, how it is accessed by patients, and the downstream impacts this may have on patient care. All of these metrics may not be assessed in a detailed manner, but having them available supports investigating “last mile” problems when highly accurate AI fails to improve clinical outcomes. This evidence is critically needed to inform continued surveillance and refinement of AI models and to guide strategies for operationalizing the broad guidelines and frameworks that professional bodies such as the National Academy of Medicine have published. 23 Finally, there are notable gaps in patients’ expressed interest in understanding more details about model predictions, and the strategies that we presently have to provide this information in a clear and comprehensible way. Despite substantial interest in the explainability of AI, very little explainability work has considered patients as end users. 33 Clinicians themselves may also struggle to determine how and when to follow AI recommendations, let alone how to deliver this information to patients. 34 ,356 These findings point to the need to both equip health professionals, including nurses and providers, with AI-related competencies or “literacy.” 34,3524 Limitations of this study include that we based part of our interview guide on an existing algorithm predicting postpartum depression; thus, findings may not generalize to other clinical contexts or populations. Additionally, though we strived to invite a diverse range of stakeholders to participate in interviews, some perspectives may have been underrepresented. Thus, future work building upon these findings to further investigate patient-centered AI across a range of clinical contexts and populations, with additional stakeholder groups, is needed. CONCLUSION Patients, health professionals, and AI developers provided important insights on AI implementation that considers patients as active end users. Participants described how to mitigate potential harms related to AI, how AI may provide meaningful benefits, recommendations, and new questions for communicating AI models, as well as challenges related to data quality, privacy, security, access, and liability. Based on our findings, we provide recommendations for patient-centered AI implementation that include engaging stakeholders throughout development (not just at the end), supporting health professionals with competencies for interpreting and communicating AI risks, and inclusively communicating AI in ways that also support taking relevant clinical actions. METHODS Ethics approval All participants provided written informed consent. The Columbia University Institutional Review Board approved the study. Setting, Study Design, and Sample We conducted prospective qualitative interviews with AI developers, health professionals, and perinatal patients. Health professionals and patients were potential end users of the tool in our use case, and AI developers had relevant expertise but were not directly involved in development of the given tool. Table 4 describes the inclusion criteria and recruitment methods. Table 4 Inclusion criteria and recruitment sources by participant stakeholder group. Stakeholder group Inclusion criteria Recruitment sources AI developers - Involved in the AI technical development process - Had experience with AI applications in mental or reproductive health - English-speaking Professional organizations (e.g., the American Medical Informatics Association), and personal connections Health professionals - Provides care to perinatal persons (related to somatic care, mental health care, or both) in the U.S. - English-speaking Departmental listservs, personal connections Patients - Currently pregnant or within one year postpartum - Received/receiving perinatal care in the U.S. - English-speaking Social media (Instagram, Reddit, Facebook), Columbia University (RecruitMe) Data Collection Our team iteratively developed interview guides (Supplemental file), combining insights from MITRE’s Ethical Framework for the Use of Consumer-Generated Data in Health Care and Sittig and Singh’s Socio-technical model. 20 , 36 Experts in obstetrics, perinatal mental health, nursing, AI development, consumer informatics, and qualitative methods provided input. We piloted each guide before enrollment. We showed patient and health professional participants a mockup of how the algorithm from our use case (i.e., proactive prediction of postpartum depression) may be displayed to concretize the discussion (Fig. 1 ). The mockup was informed by a survey of over 500 perinatal participants. 37 We found that participants trusted AI-based risk scores that included a concrete risk number (instead of phrases like “above average”), and that participants preferred number line graphics over numbers alone. Three researchers trained in qualitative methods conducted all interviews via Zoom or telephone (NCB, PMD, ZR). Interviews lasted 30–60 minutes and were audio recorded. Participants were compensated $ 50. Following the interview, participants electronically completed a demographics survey. Data Analysis We conducted a directed content analysis using thematic analysis and the constant comparative process. 38 In this paper, we focus on insights derived from the dimensions of the Sittig and Singh Socio-technical Model, which we first used to deductively organize the data, but with adapted definitions to suit our application to predictive AI. 20 A group of three coders (NB, PD, ZR) first analyzed transcripts collaboratively. Two team members then coded each transcript independently. We then met to compare and resolve disagreements, using the third coder in cases where agreement could not be reached. We also reviewed transcripts for thematic saturation, ensuring our sample met this criterion based on the diminishing need to update code definitions. Once all transcripts had been coded, we reviewed the data code by code to mitigate code drift. One of the original coders (NB) and another team member (MRT) then reviewed each parent code to derive emerging themes that cut across the domains of the Sittig and Singh framework. Declarations AUTHOR CONTRIBUTIONS MRT and NB jointly conceptualized the study and worked with JP to obtain supplementary funding to an existing grant. NB, PD, ZR, AH, YZ, RJ, and JP participated in reviewing and iterating upon the interview guides and study design. NB, PD, and ZR collected and analyzed the data along with MRT. US and VW contributed to interpretation of the analysis along with other authors. NB, US, and MRT jointly drafted the manuscript with significant input from all other authors. MRT takes responsibility for the manuscript as a whole. HUMAN ETHICS, ETHICAL APPROVAL, AND CONSENT TO PARTICIPATE This study was conducted in accordance with the Declaration of Helsinki. All procedures were approved by the Columbia University Institutional Review Board (#AAAU9647). All participants provided written informed consent electronically via Qualtrics after discussion with a study team member but before commencing data collection. CLINICAL TRIAL NUMBER Not applicable. DATA AVAILABILITY All raw data is available for sharing upon reasonable request to the corresponding author. COMPETING INTERESTS Drs. Reading Turchioe, Pathak, and Hermann are co-founders and have equity in Iris OB Health, a postpartum depression-focused startup. Dr. Hermann has also been a consultant for Progyny and Uptodate.com. No other authors have conflicts to disclose. FUNDING/SUPPORT This work was supported by the National Institute of Mental Health (R41MH124581-S1; PI: Pathak). PD and US were supported by the National Library of Medicine (NLM; T15LM007079 and T15LM007450). MRT and VW have been supported by the National Institute of Nursing Research (NINR; R00NR019124 and T32NR014205). NB is supported by the National Institute on Minority Health and Health Disparities (NIMHD: R00MD015781). ROLE OF FUNDING/SPONSOR The funders did not play any role in developing the content of this manuscript. References Science N, Technology Council. Preparing for the Future of Artificial Intelligence . Executive Office of the President; 2016. Accessed January 3, 2023. https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf Topol E. Deep medicine: how artificial intelligence can make healthcare human again. Deep medicine: how artificial intelligence can make healthcare human again . Published online 2019. Triantafyllidis AK, Tsanas A. Applications of Machine Learning in Real-Life Digital Health Interventions: Review of the Literature. J Med Internet Res . 2019;21(4):e12286. Ting DSW, Liu Y, Burlina P, Xu X, Bressler NM, Wong TY. AI for medical imaging goes deep. Nat Med . 2018;24(5):539–540. Zhou Q, Chen ZH, Cao YH, Peng S. Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review. npj Digital Medicine . 2021;4(1):154. Walker SC, French B, Moore RP, et al. Model-guided decision-making for thromboprophylaxis and hospital-acquired thromboembolic events among hospitalized children and adolescents: The CLOT randomized clinical trial. JAMA Netw Open . 2023;6(10):e2337789. Walsh CG, McKillop MM, Lee P, Harris JW, Simpson C, Novak LL. Risky business: a scoping review for communicating results of predictive models between providers and patients. JAMIA open . 2021;4(4):ooab092. Benda NC, Das LT, Abramson EL, et al. “How did you get to this number?” Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study. J Am Med Inform Assoc . 2020;27(5):709–716. Dai T, Tayur S. Designing AI-augmented healthcare delivery systems for physician buy‐in and patient acceptance. Prod Oper Manag . Published online September 9, 2022. doi: 10.1111/poms.13850 Jeyakumar T, Younus S, Zhang M, et al. Preparing for an Artificial Intelligence–Enabled Future: Patient Perspectives on Engagement and Health Care Professional Training for Adopting Artificial Intelligence Technologies in Health Care Settings. JMIR AI . 2023;2(1):e40973. Wu C, Xu H, Bai D, Chen X, Gao J, Jiang X. Public perceptions on the application of artificial intelligence in healthcare: a qualitative meta-synthesis. BMJ Open . 2023;13(1):e066322. Lee TC, Shah NU, Haack A, Baxter SL. Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review. Informatics (MDPI) . 2020;7(3). doi: 10.3390/informatics7030025 Reale C, Novak LL, Robinson K, et al. User-Centered Design of a Machine Learning Intervention for Suicide Risk Prediction in a Military Setting. AMIA Annu Symp Proc . 2020;2020:1050–1058. Matthiesen S, Diederichsen SZ, Hansen MKH, et al. Clinician Preimplementation Perspectives of a Decision-Support Tool for the Prediction of Cardiac Arrhythmia Based on Machine Learning: Near-Live Feasibility and Qualitative Study. JMIR human factors . 2021;8(4):e26964. Baxter SL, Bass JS, Sitapati AM. Barriers to implementing an artificial intelligence model for unplanned readmissions. ACI Open . 2020;4(2):e108-e113. Nong P, Adler-Milstein J. Socially situated risk: challenges and strategies for implementing algorithmic risk scoring for care management. JAMIA open . 2021;4(3):ooab076. Salwei ME, Anders S, Novak L, et al. Preventing clinical deterioration in cancer outpatients: Human centered design of a predictive model and response system. J Clin Orthod . 2022;40(16_suppl):e13567-e13567. Act CC. Public Law 114–255. 114 th Congress. Published online 2021. Hassan N, Slight R, Bimpong K, et al. Systematic review to understand users perspectives on AI-enabled decision aids to inform shared decision making. NPJ Digit Med . 2024;7(1):332. Sittig DF, Singh H. A New Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems. In: Patel VL, Kannampallil TG, Kaufman DR, eds. Cognitive Informatics for Biomedicine . Health Informatics. Springer International Publishing; 2015:59–80. Bourgeois FC, Sinha A, Tuli G, et al. The substance-exposed birthing person-infant/child dyad and health information exchange in the United States. J Am Med Inform Assoc . 2025;32(3):417–425. Williams K, Nikolajski C, Rodriguez S, et al. Incorporating end-user perspectives into the development of a machine learning algorithm for first time perinatal depression prediction. J Am Med Inform Assoc . 2025;32(7):1186–1198. National Academy of Medicine. An Artificial Intelligence Code of Conduct for Health and Medicine: Essential Guidance for Aligned Action . National Academies Press; 2025. Russell RG, Lovett Novak L, Patel M, et al. Competencies for the Use of Artificial Intelligence-Based Tools by Health Care Professionals. Acad Med . 2023;98(3):348–356. Ratwani RM, Booker E. We need A federal framework for health care AI. Forefront Group . Published online January 16, 2025. doi: 10.1377/forefront.20250114.858960 Current Use And Evaluation Of Artificial Intelligence And Predictive Models In US Hospitals: Article Examines Uses and Evaluation of Artificial Intelligence and . Abulibdeh R, Celi LA, Sejdić E. The illusion of safety: A report to the FDA on AI healthcare product approvals. PLOS Digit Health . 2025;4(6):e0000866. Ong JCL, Ning Y, Collins GS, et al. International partnership for governing generative artificial intelligence models in medicine. Nat Med . Published online June 30, 2025. doi: 10.1038/s41591-025-03787-4 Aydin S, Karabacak M, Vlachos V, Margetis K. Large language models in patient education: a scoping review of applications in medicine. Front Med (Lausanne) . 2024;11:1477898. Longhurst CA, Singh K, Chopra A, Atreja A, Brownstein JS. A call for artificial intelligence implementation science centers to evaluate clinical effectiveness. NEJM AI . 2024;1(8). doi: 10.1056/aip2400223 Kohane IS. Compared with what? Measuring AI against the health care we have. N Engl J Med . 2024;391(17):1564–1566. Chen JG. Reducing Pediatric Hospital-Acquired Venous Thromboembolism—Overcoming Challenges Using Quality Improvement. JAMA Netw Open . 2023;6(10):e2338275-e2338275. Markus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform . 2021;113:103655. Lomis K, Jeffries P, Palatta A, et al. Artificial intelligence for health professions educators. NAM Perspect . 2021;2021. doi: 10.31478/202109a Kleib M, Nagle LM, Furlong KE, Paul P, Duarte Wisnesky U, Ali S. Are Future Nurses Ready for Digital Health?: Informatics Competency Baseline Assessment. Nurse Educ . 2022;47(5):E98-E104. Granger E, Skopac J. An ethical framework for the use of consumer generated data in health care. In: 2021 Health Datapalooza and National Health Policy Conference . AcademyHealth; 2021. https://academyhealth.confex.com/academyhealth/2021hdpnhpc/meetingapp.cgi/Paper/44315 Desai PM, Harkins S, Rahman S, et al. Visualizing machine learning-based predictions of postpartum depression risk for lay audiences. J Am Med Inform Assoc . 2024;31(2):289–297. Saldana J. The Coding Manual For Qualitative Researchers . Sage; 2012:328. Additional Declarations Competing interest reported. Drs. Reading Turchioe, Pathak, and Hermann are co-founders and have equity in Iris OB Health, a postpartum depression-focused startup. Dr. Hermann has also been a consultant for Progyny and Uptodate.com. No other authors have conflicts to disclose. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 12 Nov, 2025 Reviews received at journal 12 Nov, 2025 Reviews received at journal 11 Nov, 2025 Reviews received at journal 10 Nov, 2025 Reviewers agreed at journal 01 Nov, 2025 Reviewers agreed at journal 29 Oct, 2025 Reviewers agreed at journal 27 Oct, 2025 Reviewers agreed at journal 27 Oct, 2025 Reviewers agreed at journal 27 Oct, 2025 Reviewers invited by journal 27 Oct, 2025 Editor assigned by journal 24 Oct, 2025 Submission checks completed at journal 24 Oct, 2025 First submitted to journal 20 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7908218","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":538588417,"identity":"ce8cad2e-57a0-4a31-9c86-d798ec641053","order_by":0,"name":"Natalie Benda","email":"","orcid":"","institution":"Columbia University Irving Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Natalie","middleName":"","lastName":"Benda","suffix":""},{"id":538588418,"identity":"19a05e95-3676-42c9-9740-2737f4f8dc8f","order_by":1,"name":"Pooja Desai","email":"","orcid":"","institution":"Columbia University Irving Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Pooja","middleName":"","lastName":"Desai","suffix":""},{"id":538588419,"identity":"68801b0c-1c47-4579-bca6-462d8a3b5920","order_by":2,"name":"Zayan Reza","email":"","orcid":"","institution":"Columbia University Irving Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Zayan","middleName":"","lastName":"Reza","suffix":""},{"id":538588420,"identity":"64e799e2-ac57-41a7-ad65-fb2ba89800d3","order_by":3,"name":"Victoria Winogora","email":"","orcid":"","institution":"Columbia University Irving Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Victoria","middleName":"","lastName":"Winogora","suffix":""},{"id":538588421,"identity":"d08bb170-3a24-4789-b9aa-5b98a9381b8b","order_by":4,"name":"Uday Suresh","email":"","orcid":"","institution":"Vanderbilt University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Uday","middleName":"","lastName":"Suresh","suffix":""},{"id":538588422,"identity":"4742c6a6-fde5-4b7a-82d9-be92bb4c2757","order_by":5,"name":"Yiye Zhang","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yiye","middleName":"","lastName":"Zhang","suffix":""},{"id":538588423,"identity":"34518eec-5fdd-46ed-9066-56e01a620930","order_by":6,"name":"Alison Hermann","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Alison","middleName":"","lastName":"Hermann","suffix":""},{"id":538588424,"identity":"d5adc259-53fb-468a-955a-e5fe58c437fd","order_by":7,"name":"Rochelle Joly","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Rochelle","middleName":"","lastName":"Joly","suffix":""},{"id":538588425,"identity":"918452c7-1f50-4d11-a11a-92b85c66cafa","order_by":8,"name":"Jyotishman Pathak","email":"","orcid":"","institution":"Arizona State University","correspondingAuthor":false,"prefix":"","firstName":"Jyotishman","middleName":"","lastName":"Pathak","suffix":""},{"id":538588426,"identity":"7f2d41d4-103b-492a-b3cf-b7ec90409b1e","order_by":9,"name":"Meghan Reading Turchioe","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8klEQVRIiWNgGAWjYFAC9uOff/DYwHgWDAwSjA1ABjMeLTxpzAwyaTCeBDFaGMyYGWwOI2sBM3Br0Z19IO1xQc55eYNrhx8w/KiQSFw7u7nxA0OFdWIDLivOJR43nnHmtuGG22kGjD1nJBK33TnYLMFwJh23ljMMCRK8PbcTDG7nMDAztgG13EhskGBsO4xPi4EE779zKFqafzD+w6vFTJqH5wCKljZgoOHTwpNsOAOIZwL9chDoF2OgX9osEo6lG+PWwn7wwQceO3m+28kPH/yosJHddrv98Y0PNdayuLSggANwVgIxykfBKBgFo2AU4AQACNBfka0r1IAAAAAASUVORK5CYII=","orcid":"","institution":"Columbia University Irving Medical Center","correspondingAuthor":true,"prefix":"","firstName":"Meghan","middleName":"Reading","lastName":"Turchioe","suffix":""}],"badges":[],"createdAt":"2025-10-20 18:38:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7908218/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7908218/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95340415,"identity":"7259c55f-8fdc-4cca-a551-ac3e79b4039e","added_by":"auto","created_at":"2025-11-07 01:28:03","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1930367,"visible":true,"origin":"","legend":"","description":"","filename":"npjRealWorldEval.SpecialIssue.docx","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/7ee9f92621d5bff0fe191ce8.docx"},{"id":95340418,"identity":"5ff05041-820f-4c1c-a841-4f1723bbc00a","added_by":"auto","created_at":"2025-11-07 01:28:03","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":11317,"visible":true,"origin":"","legend":"","description":"","filename":"edd6c8bf45cc4a31bfd688db27bb689f.json","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/3a2ed3b606ad1f4811d22543.json"},{"id":95525116,"identity":"be69b024-855c-433e-bd49-6afaca0a0849","added_by":"auto","created_at":"2025-11-10 10:04:16","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":116150,"visible":true,"origin":"","legend":"","description":"","filename":"edd6c8bf45cc4a31bfd688db27bb689f1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/ca44ac91b0f33a78f62dc44d.xml"},{"id":95525168,"identity":"2188fdff-265c-4ce4-a0fb-66a4b5215760","added_by":"auto","created_at":"2025-11-10 10:04:24","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":11936,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/4f31d401d3260c47b4da6ed4.png"},{"id":95525081,"identity":"1f7f417d-3012-4d9b-b017-44d7874928d8","added_by":"auto","created_at":"2025-11-10 10:04:13","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":114497,"visible":true,"origin":"","legend":"","description":"","filename":"edd6c8bf45cc4a31bfd688db27bb689f1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/252a3976e3334e106b43b440.xml"},{"id":95340413,"identity":"e630fd05-25c8-4f84-8bd2-ec4d5d97c81f","added_by":"auto","created_at":"2025-11-07 01:28:03","extension":"html","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":126876,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/28a6e9036347f5a72fe2fe19.html"},{"id":95340411,"identity":"1c19075f-f727-4768-a364-a467046ce4ee","added_by":"auto","created_at":"2025-11-07 01:28:03","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":47681,"visible":true,"origin":"","legend":"\u003cp\u003eSample output of postpartum risk prediction AI tool shown to patient and heath professional participants.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/d9ca7baf5075d318ee64ab41.png"},{"id":95530750,"identity":"09c58b58-3860-43d3-b8ac-99c554bb2ab9","added_by":"auto","created_at":"2025-11-10 10:21:44","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1065026,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7908218/v1/2ad7e715-abf8-4b00-bfc2-e137f2ad4a19.pdf"}],"financialInterests":"Competing interest reported. Drs. Reading Turchioe, Pathak, and Hermann are co-founders and have equity in Iris OB Health, a postpartum depression-focused startup. Dr. Hermann has also been a consultant for Progyny and Uptodate.com. No other authors have conflicts to disclose.","formattedTitle":"A qualitative Interview Study Investigating Patient, Health Professional, and Developer Perspectives on Real-World Implementation of Patient-Centered AI Systems","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eThe transition to artificial intelligence (AI) in healthcare continues to be significant for patients and health professionals.\u003csup\u003e\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Despite AI\u0026rsquo;s predictive and generative power, many systems are never implemented or, when implemented, struggle to show improvements in patient outcomes compared to the standard of care.\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e This clinical translation gap between AI development and real-world, successful deployment is defined as the \u0026ldquo;last mile\u0026rdquo; problem. The \u0026ldquo;last mile\u0026rdquo; problem is the fact that AI models may show high accuracy and promise in simulated lab-based tasks, but (1) they are either never deployed in clinical environments, or (2) they cannot improve outcomes in real-life due to deficiencies in implementation. A critical gap lies in end users seeing, accepting, and acting upon AI-based recommendations.\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e Previous research in human-AI interaction has focused heavily on health professionals as the sole recipients of AI recommendations, with less attention to implementing patient-centered AI systems.\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e Our previous pre-implementation work for AI-based systems highlighted the importance of keeping patients informed regarding AI-informed recommendations to justify interventions and to support uptake.\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003ePatients and the public have important perspectives regarding AI implementation and want to be part of the conversation.\u003csup\u003e\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e However, their perspectives on centering patients in AI system implementation have been understudied compared to health professionals and AI developers.\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e,\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e,\u003cspan additionalcitationids=\"CR13 CR14 CR15 CR16\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e Moreover, previous studies have independently surveyed patient, health professional, and developer preferences about AI implementation. No studies to our knowledge, however, have collected richer qualitative data and combined their collective insights. Consequently, we lack integrative, practical guidance for those seeking to ethically implement and evaluate patient-centered AI systems.\u003c/p\u003e\u003cp\u003eIt is also becoming more likely that patients may access AI output, even if not intended by AI developers. The U.S. 21st Century Cures Act, for example, prevents information blocking from patients, requiring organizations to give patients access to their electronic health information without delay or expense.\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e This may result in a patient seeing a predictive AI risk score before discussing it with their healthcare team. AI is also increasingly used in tools to support shared decision-making with patients.\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e While AI output may benefit patients, it may cause stress if not appropriately contextualized. AI can also be prone to hallucinations, the creation of false information, which may be challenging to differentiate from factual evidence.\u003c/p\u003e\u003cp\u003eOur objective was to triangulate perspectives from patients, developers, and health professionals for real-world implementation of patient-centered AI systems. We interviewed these three stakeholder groups using interview guides informed by bioethics and informatics models.\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e To concretize discussions, we used an existing predictive AI tool developed by our team that proactively predicts a pregnant patient\u0026rsquo;s risk of postpartum depression (PPD) as a use case. Recruited patients and health professionals were potential users of the tool. PPD presents a complex bioethical case because of the sensitivity of the data and for the multi-layer autonomy; in pregnancy, the autonomy, harms, and benefits to the perinatal patient, newborn or fetus, and partner must be weighed simultaneously.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e,\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e Our study provides novel, data-driven recommendations for how AI may be ethically implemented to support patient-centred care.\u003c/p\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eParticipant Characteristics\u003c/h2\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e provides demographic data from each stakeholder group (N\u0026thinsp;=\u0026thinsp;36).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eParticipants' demographics provided at the aggregate level and broken out by stakeholder group.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colspan=\"2\" morerows=\"1\" nameend=\"c2\" namest=\"c1\" rowspan=\"2\"\u003e\u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eDemographic characteristic\u003c/span\u003e\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colspan=\"4\" nameend=\"c6\" namest=\"c3\"\u003e\u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eStakeholder Group\u003c/span\u003e\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003e\u003cb\u003eOverall\u003c/b\u003e (N\u0026thinsp;=\u0026thinsp;36), n(%)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003e\u003cb\u003eDevelopers\u003c/b\u003e (n\u0026thinsp;=\u0026thinsp;10), n (%)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003e\u003cb\u003eHealth Professionals\u003c/b\u003e (n\u0026thinsp;=\u0026thinsp;8), n (%)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003e\u003cb\u003ePatients\u003c/b\u003e (n\u0026thinsp;=\u0026thinsp;18), n (%)\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eSex\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eFemale\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e29 (81)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e5 (50)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e7 (87.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e17 (94.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMale\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e4 (11)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e4 (40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMissing\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e3 (8)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e1 (12.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eRace\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eWhite\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e13 (36)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e5 (50)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e6 (33.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAsian\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e6 (17)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e4 (40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eBlack or African American\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e9 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e7 (39)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eNative Hawaiian or Other Pacific Islander\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e1 (3)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMore than One Race\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e2 (5.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e2 (11)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003ePrefer Not to Answer\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e2 (5.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e1 (12.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMissing\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e3 (8)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e1 (12.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eAge (generation)\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eGeneration Z (1997\u0026ndash;2012)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e6 (17)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e3 (30)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e3 (17)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMillennials (1981\u0026ndash;1996)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e23 (64)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e5 (50)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e5 (62.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e13 (72)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eGeneration X (1965\u0026ndash;1980)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e3 (8)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eBoomers (1946\u0026ndash;1964)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e1 (3)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMissing\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e3 (8)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e1 (12.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eEducation\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eSome College or Associate's Degree\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e6 (17)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e6 (33.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eBachelor's Degree\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e11 (31)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e8 (44)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMaster's Degree\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e8 (22)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e2 (20)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e3 (37.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e3 (17)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eDoctorate Degree\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e8 (22)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e6 (60)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e2 (25)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0 (0)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMissing\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e3 (8)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1 (10)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e1 (12.5)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1 (5.5)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eEmerging themes\u003c/h3\u003e\n\u003cp\u003eWe inductively derived emerging themes for implementing patient-centered AI systems that cut across our overarching framework for analysis, Sittig and Singh\u0026rsquo;s Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e)\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. We describe the detailed emerging themes with the most illustrative quotes from developers (DV), health professionals (HP), and patients (PT).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eEmerging themes from our analysis and their intersection with the dimensions of Sittig and Singh's Sociotechnical framework.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"9\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e\u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eSittig and Singh Dimension\u003c/span\u003e\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cspan type=\"BoldUnderline\" class=\"BoldUnderline\" name=\"Emphasis\"\u003eEmerging theme\u003c/span\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e\u003cb\u003eSoftware, hardware, and computing infrastructure\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e\u003cb\u003eClinical content\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e\u003cb\u003eHuman- computer interface\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e\u003cb\u003eWorkflow and communication\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e\u003cb\u003ePeople\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e\u003cb\u003eInternal organisational policies, procedures and culture\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u003cp\u003e\u003cb\u003eExternal rules, regulations and pressures\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u003cp\u003e\u003cb\u003eSystem measurement and monitoring\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eMitigating harm\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eClinical benefit and utility\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eCommunicating AI models\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eData quality\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eData security, privacy, and access\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eResponsible AI practices and liability\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c8\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c9\"\u003e\u003cp\u003eX\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\n\u003ch3\u003eMitigating harm\u003c/h3\u003e\n\u003cp\u003eAll participant roles stressed the importance of ensuring minimal harm to end users (e.g., patients, health professionals) related to AI. One primary source of harm discussed involved stigmas driven by AI-based predictions. For example, predicting risk for mental health conditions or addictions (e.g., opioid use disorder) could change patient care trajectories. Patients expressed fear that risk predictions of mental health issues could lead to child protective services (CPS) interventions.\u003c/p\u003e\u003cp\u003e\u003cem\u003eI may run a 100 score and it\u0026rsquo;s like, \u0026lsquo;oh i passed\u0026rsquo;. Like, why is the doctor coming to talk to me? And ACS\u003c/em\u003e (local CPS department) \u003cem\u003eis coming to take my child?\u003c/em\u003e - PT12\u003c/p\u003e\u003cp\u003eParticipants also raised the concern about model bias, noting the importance of utilizing representative training data and valuing fairness equally to accuracy.\u003c/p\u003e\u003cp\u003e\u003cem\u003eI\u0026rsquo;m sure they\u0026rsquo;d be curious if their ethnicity is put in there\u0026hellip;But what if there is some type of bias in the tool?\u003c/em\u003e - HP02\u003c/p\u003e\u003cp\u003ePatients and health professionals viewed AI as potentially harmful due to the stress that high-risk predictions may cause patients. Participants described the importance of involving health professionals and patients in the early stages of AI development to identify and mitigate these harms.\u003c/p\u003e\u003cp\u003e\u003cem\u003eThe patient is going to develop that insight and be on board\u0026hellip; and lead to like a more \u0026hellip; patient-centered treatment.\u003c/em\u003e - DV08\u003c/p\u003e\n\u003ch3\u003eClinical benefit and utility\u003c/h3\u003e\n\u003cp\u003eAll groups expressed that AI must provide clinical benefits to patients, care partners, health professionals, or health systems. Participants agreed that for models to provide utility, a health professional or patient must be able to easily take action (e.g., making a referral, prescribing a medication, contacting their provider) based on AI\u0026rsquo;s output. Patient and health professional participants seemed most comfortable with AI that could \u0026ldquo;run in the background\u0026rdquo; and provide support to human decision-makers as needed.\u003c/p\u003e\u003cp\u003e\u003cem\u003eIf it can explain its reasoning, I feel like that's more helpful\u0026hellip;I like to think of AI like as a tool. So if it was actually like teaching me and\u0026hellip;I can learn what those risk factors are and take the time, then I would find it way more useful.\u003c/em\u003e - PT16\u003c/p\u003e\u003cp\u003eFor the tool discussed (predictive AI for PPD), patient participants were typically open to their providers using the tool as an additional data point and accepting treatment based on the AI and their doctor's input. However, one patient was very averse to using the tool, indicating the preference-sensitive nature of AI use in clinical care.\u003c/p\u003e\u003cp\u003e\u003cem\u003eI would be confused because I wouldn\u0026rsquo;t believe that they could figure that out\u0026hellip;it seems so bizarre. I probably would lose a bit of trust.\u003c/em\u003e - PT09\u003c/p\u003e\u003cp\u003eTranslating predictive models into action also requires a threshold for action. Patient participants, however, had challenges understanding what the numeric risk predictions meant practically and why there were certain thresholds for action (e.g., the tool recommended a mental health referral if the PPD risk was \u0026gt;\u0026thinsp;30%).\u003c/p\u003e\u003cp\u003e\u003cem\u003e0 to 30%, you know, that's a closer gap. And then you have like this gap from 32 to 100, but it's like 40, 50, 60, 70. And I'm assuming the prevention treatment is starting at either 75 or 80. And I was like, what about those people in the middle?\u003c/em\u003e - PT12\u003c/p\u003e\u003cp\u003eTo trust AI and promote relevant actions, most end users wanted to understand the accuracy of AI. Ideally, patients wanted to know how accurate it was for them individually. However, some patients did not need to see this information themselves but trusted health professionals to make judgments on when AI was safe for clinical use.\u003c/p\u003e\u003cp\u003e\u003cem\u003eI assume that [if] it\u0026rsquo;s in my patient portal, I\u0026rsquo;m thinking that it\u0026rsquo;s accurate for me.\u003c/em\u003e - PT01\u003c/p\u003e\u003cp\u003eDevelopers noted a tension between high performance and making understandable models that people wanted to use.\u003c/p\u003e\u003cp\u003eInterviewer: \u003cem\u003eAccuracy versus understandability and explainability, and if you could only choose one, which one would you choose and why?\u003c/em\u003e\u003c/p\u003e\u003cp\u003eDV01: \u003cem\u003eWe need both, you can\u0026rsquo;t really do it without both, but when forced\u0026hellip;the majority of the crowd choose understandability.\u003c/em\u003e\u003c/p\u003e\u003cp\u003eParticipants described different strategies for ensuring models provided benefits. First, health professionals need to support clinical validation and interrogate potential biases. Participants also highlighted the need to investigate and plan for how AI may alter existing workflows. Updates would then need to be implemented in line with institutional norms and policies.\u003c/p\u003e\u003cp\u003eDevelopers viewed implementation and ensuring utility as a continuous process that would require updates over time, similar to any traditional software. Developers also acknowledged the importance of end-user feedback throughout AI model development. Although they did not always feel they had the skills to include this feedback, particularly related to patient users.\u003c/p\u003e\u003cp\u003e\u003cem\u003eI think one piece that's definitely missing in my research is patient perspectives about whether they think this kind of tool would be useful\u0026hellip;I think that those questions have to be asked very carefully\u0026hellip;But I don't know that developers should be the ones actually going in to ask those questions.\u003c/em\u003e - DV07\u003c/p\u003e\n\u003ch3\u003eCommunicating AI Models\u003c/h3\u003e\n\u003cp\u003eSeveral questions emerged regarding if, when, where, and how AI-related information should be communicated to patients. Patients had differing perspectives regarding how they would like to view AI output, with some preferring to see it before clinical encounters and others wanting someone to explain it to them. Multiple participants agreed that patients would need support in interpreting information and suggested that the provider may be the best person to do this. However, they also noted that the provider may need training and dedicated time to do this effectively.\u003c/p\u003e\u003cp\u003e\u003cem\u003eWho gets the alert, and what happens once you get an alert? If you're going through all that, then how much time are you spending to explain it?\u003c/em\u003e - HP06\u003c/p\u003e\u003cp\u003eOthers highlighted that interfaces should be presented in lay terms, accessible to different audiences. Feature importance was discussed at length - participants suggested conveying predictors of risk (e.g., medications, history of past illness) as both positives and negatives contributing to the patient's total risk. Patients generally wanted to see all features, not only the modifiable predictors. Providing sensitive features (e.g., risks related to marital status, BMI) requires care but may stimulate educational conversations. It was also important to ensure the labels were understandable by patients, so using the EHR-based feature names (e.g., \u0026ldquo;thyroid preparations\u0026rdquo;, \u0026ldquo;hyperemesis\u0026rdquo;) may be insufficient.\u003c/p\u003e\u003cp\u003ePatients also reported varying opinions on their desire to see AI-based risk information. Some felt they had a right to see these risks, how accurate they were, and what their individual risk factors were. Others trusted their provider to see the information and make the related decisions.\u003c/p\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003eData quality\u003c/h2\u003e\u003cp\u003eParticipants described challenges related to ensuring data quality, primarily related to bias, missingness, out-of-date information, and shortcomings in interpreting unstructured data (e.g., clinical notes) with insufficient context.\u003c/p\u003e\u003cp\u003e\u003cem\u003eThese days...charts are not well updated\u0026hellip; unless it's thoughtfully updated, which it rarely is because hospital systems are far too busy for that kind of meticulousness, then I think that people are going to feel like I could see people being like, \u0026lsquo;where the hell did that come from?\u0026rsquo;\u003c/em\u003e - HP03\u003c/p\u003e\u003cp\u003eMissingness was noted to be particularly problematic in a perinatal context, where many may receive care for other specialties in different health systems.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eData security, privacy, and access\u003c/h3\u003e\n\u003cp\u003eParticipants agreed that the way the data is stored, the algorithm is computed, and displayed to end users must protect patient and health professional privacy. There was, however, a tension between making tools open source and protecting patient privacy, ensuring their autonomy over their data.\u003c/p\u003e\u003cp\u003eMany viewed external organizations, such as the Centers for Medicare and Medicaid, as having a responsibility for ensuring privacy (i.e., data de-identification) and also governing privacy rules, although it was not always clear as to which entity was responsible. Health organizations were also mentioned as having a role in supporting data privacy and promoting data access. There were other external organizations whose access to data (e.g., CPS/ACS) was viewed as detrimental.\u003c/p\u003e\u003cp\u003ePatients had different desires for accessing AI inputs and outputs, including through the patient portal, on mobile phones, via PDF, and secure email. Some also wanted the ability to share the information with chosen care partners, but this varied from person to person.\u003c/p\u003e\u003cp\u003eOthers highlighted how the combination of perinatal and psychiatric concerns made privacy issues more sensitive. Some patients, for example, may be reluctant to disclose sensitive information due to stigma or fear, leading to data accuracy issues.\u003c/p\u003e\u003cp\u003e\u003cem\u003eWhenever a pregnant woman talks about this type of information...that concerns like calling to Child Protective Services. That is a big privacy issue...clinicians worry about models raising severe alerts.\u003c/em\u003e - DV09\u003c/p\u003e\u003cp\u003eParticipants working in pediatrics also noted that accessing maternal health information through a child\u0026rsquo;s chart can be challenging due to limitations in data linkage capabilities and due to confidentiality protections in the EHR for sensitive maternal data.\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003ch3\u003eResponsible AI practices and liability\u003c/h3\u003e\n\u003cp\u003eParticipants had various opinions about who was responsible for the safe development and implementation of AI. Participants mentioned responsible entities at various levels from insurers and device manufacturers to individual developers and health professionals.\u003c/p\u003e\u003cp\u003e\u003cem\u003eThere needs to be some responsible party who's well informed, who's making the decisions about if and how these tools are actually used and up to date on how well they're actually being used in patient care as you introduce them\u003c/em\u003e - DV08\u003c/p\u003e\u003cp\u003eMany of the questions of responsibility stemmed from a desire to mitigate individual liability for health professionals. The prevailing perspective was that no single person or entity could be held responsible, and a structure of shared responsibility was necessary. Some perceived that determining safety for the use of an algorithm should fall to health professionals. Health professionals also highlighted the need for responsibility in terms of seeing and acting upon risk predictions. Developers, on the other hand, felt responsible for making systems with high predictive performance that also mitigated potential biases.\u003c/p\u003e\u003cp\u003e\u003cem\u003eSo a clinician needs to make an informed decision. In which case\u0026hellip;the clinic is liable, as they would be for any misdiagnosis or incorrect treatment decision.\u003c/em\u003e - DV08\u003c/p\u003e\u003cp\u003e\u003cem\u003eIf we implement it then\u0026hellip;somebody has to have ownership to it that you are responsible for this. And if the patient is seeing this information, somebody has to interpret for them.\u003c/em\u003e - HP06\u003c/p\u003e\u003cp\u003eResponsible practices mentioned included checking for bias and accuracy prior to implementation, as well as continuous monitoring to identify model drift and retrain models. The conversation surrounding ensuring accuracy could be challenging, however, as the gold standard for comparison may not always be clear.\u003c/p\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eAI implementation in health systems requires coordination between patients, clinicians, developers, and health systems administrators and leaders. Through interviews with nearly 40stakeholders representing these perspectives, we identified requirements implementation and use of patient-centered AI systems. We summarize our findings in preliminary requirements derived directly from stakeholders, shown in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. Our results specific to end user perspectives on conveying AI-based risk of PPD overlap with Williams et. al\u0026rsquo;s recent qualitative study on a similar topic.\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e However, our work extends upon their findings, incorporating developer perspectives and using PPD as a use case to make broader recommendations for patient-centered AI regardless of clinical context. Our findings also resonate with the recently released AI Code of Conduct for Health and Medicine from the National Academy of Medicine, related to: advancing benefit for humanity, ensuring equity, engaging impacted individuals, ensuring workforce wellbeing, monitoring performance, and supporting continuous learning.\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e We describe practical next steps with data-driven insights for achieving these requirements across key stakeholder groups.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eStakeholder-driven requirements for patient-centered AI.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTopic\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eRequirement description\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eRelated themes\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eHealth professional competencies related to AI\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eProvide health professionals with the time and training to deliver AI-based risk information\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMitigating harm, communicating AI, responsible AI practices\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEthical creation of AI systems\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eEngage health professionals and patient stakeholders \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ethroughout\u003c/span\u003e the development process\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMitigating harm, clinical benefit and utility\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEffective communication of AI output for all stakeholders\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003ePromote actionability by displaying options for viewing model accuracy, predictors, and thresholds for action\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMitigating harm, communicating AI, responsible AI practices\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePatient-centered AI communication\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eInclude flexible options for communicating AI output to patients to manage cultural preferences and varying literacy levels\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMitigating harm, communicating AI, responsible AI practices\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAI governance\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMaking ethical decisions related to governance issues, including privacy, access, liability, and regulation, will require multiple levels of stakeholder engagement\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eData security, privacy, and access; responsible AI practices and liability; AI regulation\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eFirst, health professionals will require competencies to use AI effectively and safely in healthcare.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e Most patients reported that they wanted their health professionals to deliver predictive AI results, whereas most clinicians do not feel qualified to do so. Health professionals also highlighted the difficulty in interpreting AI output and the limited time during encounters. While some efforts have attempted to integrate AI competencies into medical and nursing education,\u003csup\u003e24\u003c/sup\u003e far more work is needed to educate the existing workforce of health professionals.\u003c/p\u003e\u003cp\u003eSecond, ur findings highlight how decisions made during the development process can have downstream impacts, for example, balancing explainability with accuracy, or addressing fairness. Model developers, at times, work in relative isolation from the health system leadership, clinicians, and patients who may ultimately adopt their model. Technical development is viewed as a foundational step with little input needed from other stakeholders. Rather, our findings suggest that because of the downstream implications of decisions made at the modeling stage, health professional and patient stakeholders should be included in the early stages of development and evaluation to ensure models prioritize the right values, are useful, and ethically sound.\u003c/p\u003e\u003cp\u003eParticipants also raised issues related to how data completeness and timeliness may affect model accuracy. Similar to displaying significant model features, visualization may support end users in understanding when key data elements may be missing or outdated. A systematic review of 44 implemented predictive AI models highlighted the lack of attention to user workflow and providing predictive risks in a way that enhances rather than degrades cognitive processing.\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e Future work may investigate generalizable best practices for displaying predictive AI models to work collaboratively with human end users.\u003c/p\u003e\u003cp\u003eThird, to provide clinical utility, predictive AI output needs to be actionable, as we have previously reported.\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e For example, perinatal providers need a way to refer appropriate patients for psychiatric or social work evaluation. Participants noted several needs to promote actionability, including providing reasoning (i.e., risk factors) and the level of model accuracy. This information would allow them to weigh their expertise in conjunction with the model, and choose from multiple possible actions based on the patient's risk factors. Patients wanted to understand individual risk factors, which could support shared decision-making conversations with providers regarding risk reduction. Participants also noted that meaningful thresholds for action based on the numeric risk prediction were needed. This is not done consistently in practice, and there is a lack of robust methods for determining these thresholds for actionability.\u003c/p\u003e\u003cp\u003eFourth, patients in our study expressed a wide range of preferences and information needs related to interpretation of the model. Compared to clinician end users, less attention has been given to patients as end users of models. There are many scenarios in which patients may view model output, making their perspectives vital in considering how to display output. The harms of failing to incorporate their perspectives may include elevated worry and anxiety, increased outreach to care teams, and even unnecessary care visits.\u003c/p\u003e\u003cp\u003eLastly, several other publications have provided detailed needs assessments for enhancing privacy, navigating liabilities, and devising regulations related to AI.\u003csup\u003e23,25\u0026ndash;28\u003c/sup\u003e Our findings reinforce that layers of shared governance, involving multiple stakeholder groups, will be necessary to address these challenging questions.\u003c/p\u003e\u003cp\u003eOur findings also suggest multiple important areas of future work. First, our study primarily focused on predictive AI models. Generative AI is being rapidly adopted in research and in healthcare settings. A recent systematic review identified over 200 studies that have used large language models (LLMs) for healthcare purposes, including patient education, recommendations, and guidance, customizing care plans, interpreting medical information, and facilitating patient-clinician communication.\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e They reported positive findings relating to patient satisfaction but concerns about readability, accuracy, and bias. Thus, investigating patients\u0026rsquo; needs and ability to use LLMs safely will be an important part of ensuring appropriate use in healthcare.\u003c/p\u003e\u003cp\u003eAdditionally, the science for both predictive and generative AI has weighed heavily towards model development and validation, with significantly fewer studies evaluating real-world clinical implementation and health outcomes. Multiple perspective pieces have called for more implementation science studies,\u003csup\u003e30\u003c/sup\u003e along with clinical trials that have thoughtful comparator arms and benchmarks representing the current state of the healthcare system.\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e While improving clinical outcomes is the ultimate goal for most predictive AI systems, recent studies have highlighted that only measuring these outcomes may produce an incomplete picture, particularly when AI does not have the intended effect.\u003csup\u003e\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e We advocate for detailed tracking (e.g., via EHR log data) that may ascertain how AI impacts decisions, how it is accessed by patients, and the downstream impacts this may have on patient care. All of these metrics may not be assessed in a detailed manner, but having them available supports investigating \u0026ldquo;last mile\u0026rdquo; problems when highly accurate AI fails to improve clinical outcomes. This evidence is critically needed to inform continued surveillance and refinement of AI models and to guide strategies for operationalizing the broad guidelines and frameworks that professional bodies such as the National Academy of Medicine have published.\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eFinally, there are notable gaps in patients\u0026rsquo; expressed interest in understanding more details about model predictions, and the strategies that we presently have to provide this information in a clear and comprehensible way. Despite substantial interest in the explainability of AI, very little explainability work has considered patients as end users.\u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e Clinicians themselves may also struggle to determine how and when to follow AI recommendations, let alone how to deliver this information to patients.\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e,356\u003c/sup\u003e These findings point to the need to both equip health professionals, including nurses and providers, with AI-related competencies or \u0026ldquo;literacy.\u0026rdquo;\u003csup\u003e34,3524\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eLimitations of this study include that we based part of our interview guide on an existing algorithm predicting postpartum depression; thus, findings may not generalize to other clinical contexts or populations. Additionally, though we strived to invite a diverse range of stakeholders to participate in interviews, some perspectives may have been underrepresented. Thus, future work building upon these findings to further investigate patient-centered AI across a range of clinical contexts and populations, with additional stakeholder groups, is needed.\u003c/p\u003e"},{"header":"CONCLUSION","content":"\u003cp\u003ePatients, health professionals, and AI developers provided important insights on AI implementation that considers patients as active end users. Participants described how to mitigate potential harms related to AI, how AI may provide meaningful benefits, recommendations, and new questions for communicating AI models, as well as challenges related to data quality, privacy, security, access, and liability. Based on our findings, we provide recommendations for patient-centered AI implementation that include engaging stakeholders throughout development (not just at the end), supporting health professionals with competencies for interpreting and communicating AI risks, and inclusively communicating AI in ways that also support taking relevant clinical actions.\u003c/p\u003e\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\u003cdiv id=\"Sec14\" class=\"Section3\"\u003e\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"METHODS","content":"\u003ch2\u003eEthics approval\u003c/h2\u003e\u003cp\u003e All participants provided written informed consent. The Columbia University Institutional Review Board approved the study.\u003c/p\u003e\u003ch2\u003eSetting, Study Design, and Sample\u003c/h2\u003e\u003cp\u003eWe conducted prospective qualitative interviews with AI developers, health professionals, and perinatal patients. Health professionals and patients were potential end users of the tool in our use case, and AI developers had relevant expertise but were not directly involved in development of the given tool. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e describes the inclusion criteria and recruitment methods.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eInclusion criteria and recruitment sources by participant stakeholder group.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eStakeholder group\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eInclusion criteria\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eRecruitment sources\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAI developers\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e- Involved in the AI technical development process\u003c/p\u003e\u003cp\u003e- Had experience with AI applications in mental or reproductive health\u003c/p\u003e\u003cp\u003e- English-speaking\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eProfessional organizations (e.g., the American Medical Informatics Association), and personal connections\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eHealth professionals\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e- Provides care to perinatal persons (related to somatic care, mental health care, or both) in the U.S.\u003c/p\u003e\u003cp\u003e- English-speaking\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eDepartmental listservs, personal connections\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePatients\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e- Currently pregnant or within one year postpartum\u003c/p\u003e\u003cp\u003e- Received/receiving perinatal care in the U.S.\u003c/p\u003e\u003cp\u003e- English-speaking\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eSocial media (Instagram, Reddit, Facebook), Columbia University (RecruitMe)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e\u003ch2\u003eData Collection\u003c/h2\u003e\u003cp\u003eOur team iteratively developed interview guides (Supplemental file), combining insights from MITRE’s Ethical Framework for the Use of Consumer-Generated Data in Health Care and Sittig and Singh’s Socio-technical model.\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e,\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e Experts in obstetrics, perinatal mental health, nursing, AI development, consumer informatics, and qualitative methods provided input. We piloted each guide before enrollment. We showed patient and health professional participants a mockup of how the algorithm from our use case (i.e., proactive prediction of postpartum depression) may be displayed to concretize the discussion (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The mockup was informed by a survey of over 500 perinatal participants.\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e We found that participants trusted AI-based risk scores that included a concrete risk number (instead of phrases like “above average”), and that participants preferred number line graphics over numbers alone.\u003c/p\u003e\u003cp\u003eThree researchers trained in qualitative methods conducted all interviews via Zoom or telephone (NCB, PMD, ZR). Interviews lasted 30–60 minutes and were audio recorded. Participants were compensated \u003cspan\u003e$\u003c/span\u003e50. Following the interview, participants electronically completed a demographics survey.\u003c/p\u003e\u003ch2\u003eData Analysis\u003c/h2\u003e\u003cp\u003eWe conducted a directed content analysis using thematic analysis and the constant comparative process.\u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e In this paper, we focus on insights derived from the dimensions of the Sittig and Singh Socio-technical Model, which we first used to deductively organize the data, but with adapted definitions to suit our application to predictive AI.\u003csup\u003e20\u003c/sup\u003e A group of three coders (NB, PD, ZR) first analyzed transcripts collaboratively. Two team members then coded each transcript independently. We then met to compare and resolve disagreements, using the third coder in cases where agreement could not be reached. We also reviewed transcripts for thematic saturation, ensuring our sample met this criterion based on the diminishing need to update code definitions. Once all transcripts had been coded, we reviewed the data code by code to mitigate code drift. One of the original coders (NB) and another team member (MRT) then reviewed each parent code to derive emerging themes that cut across the domains of the Sittig and Singh framework.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAUTHOR CONTRIBUTIONS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMRT and NB jointly conceptualized the study and worked with JP to obtain supplementary funding to an existing grant. NB, PD, ZR, AH, YZ, RJ, and JP participated in reviewing and iterating upon the interview guides and study design. NB, PD, and ZR collected and analyzed the data along with MRT. US and VW contributed to interpretation of the analysis along with other authors. NB, US, and MRT jointly drafted the manuscript with significant input from all other authors. MRT takes responsibility for the manuscript as a whole.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHUMAN ETHICS, ETHICAL APPROVAL, AND CONSENT TO PARTICIPATE\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was conducted in accordance with the Declaration of Helsinki. All procedures were approved by the Columbia University Institutional Review Board (#AAAU9647). All participants provided written informed consent electronically via Qualtrics after discussion with a study team member but before commencing data collection.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCLINICAL TRIAL NUMBER\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDATA AVAILABILITY\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll raw data is available for sharing upon reasonable request to the corresponding author.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCOMPETING INTERESTS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDrs. Reading Turchioe, Pathak, and Hermann are co-founders and have equity in Iris OB Health, a postpartum depression-focused startup. Dr. Hermann has also been a consultant for Progyny and Uptodate.com. No other authors have conflicts to disclose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFUNDING/SUPPORT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work was supported by the National Institute of Mental Health (R41MH124581-S1; PI: Pathak). PD and US were supported by the National Library of Medicine (NLM; T15LM007079 and T15LM007450). MRT and VW have been supported by the National Institute of Nursing Research (NINR; R00NR019124 and T32NR014205). NB is supported by the National Institute on Minority Health and Health Disparities (NIMHD: R00MD015781).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eROLE OF FUNDING/SPONSOR\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe funders did not play any role in developing the content of this manuscript.\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eScience N, Technology Council. \u003cem\u003ePreparing for the Future of Artificial Intelligence\u003c/em\u003e. Executive Office of the President; 2016. Accessed January 3, 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf\u003c/span\u003e\u003cspan address=\"https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTopol E. Deep medicine: how artificial intelligence can make healthcare human again. \u003cem\u003eDeep medicine: how artificial intelligence can make healthcare human again\u003c/em\u003e. Published online 2019.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTriantafyllidis AK, Tsanas A. Applications of Machine Learning in Real-Life Digital Health Interventions: Review of the Literature. \u003cem\u003eJ Med Internet Res\u003c/em\u003e. 2019;21(4):e12286.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTing DSW, Liu Y, Burlina P, Xu X, Bressler NM, Wong TY. AI for medical imaging goes deep. \u003cem\u003eNat Med\u003c/em\u003e. 2018;24(5):539\u0026ndash;540.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eZhou Q, Chen ZH, Cao YH, Peng S. Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review. \u003cem\u003enpj Digital Medicine\u003c/em\u003e. 2021;4(1):154.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWalker SC, French B, Moore RP, et al. Model-guided decision-making for thromboprophylaxis and hospital-acquired thromboembolic events among hospitalized children and adolescents: The CLOT randomized clinical trial. \u003cem\u003eJAMA Netw Open\u003c/em\u003e. 2023;6(10):e2337789.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWalsh CG, McKillop MM, Lee P, Harris JW, Simpson C, Novak LL. Risky business: a scoping review for communicating results of predictive models between providers and patients. \u003cem\u003eJAMIA open\u003c/em\u003e. 2021;4(4):ooab092.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBenda NC, Das LT, Abramson EL, et al. \u0026ldquo;How did you get to this number?\u0026rdquo; Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e. 2020;27(5):709\u0026ndash;716.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDai T, Tayur S. Designing AI-augmented healthcare delivery systems for physician buy‐in and patient acceptance. \u003cem\u003eProd Oper Manag\u003c/em\u003e. Published online September 9, 2022. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/poms.13850\u003c/span\u003e\u003cspan address=\"10.1111/poms.13850\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJeyakumar T, Younus S, Zhang M, et al. Preparing for an Artificial Intelligence\u0026ndash;Enabled Future: Patient Perspectives on Engagement and Health Care Professional Training for Adopting Artificial Intelligence Technologies in Health Care Settings. \u003cem\u003eJMIR AI\u003c/em\u003e. 2023;2(1):e40973.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWu C, Xu H, Bai D, Chen X, Gao J, Jiang X. Public perceptions on the application of artificial intelligence in healthcare: a qualitative meta-synthesis. \u003cem\u003eBMJ Open\u003c/em\u003e. 2023;13(1):e066322.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLee TC, Shah NU, Haack A, Baxter SL. Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review. \u003cem\u003eInformatics (MDPI)\u003c/em\u003e. 2020;7(3). doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/informatics7030025\u003c/span\u003e\u003cspan address=\"10.3390/informatics7030025\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eReale C, Novak LL, Robinson K, et al. User-Centered Design of a Machine Learning Intervention for Suicide Risk Prediction in a Military Setting. \u003cem\u003eAMIA Annu Symp Proc\u003c/em\u003e. 2020;2020:1050\u0026ndash;1058.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMatthiesen S, Diederichsen SZ, Hansen MKH, et al. Clinician Preimplementation Perspectives of a Decision-Support Tool for the Prediction of Cardiac Arrhythmia Based on Machine Learning: Near-Live Feasibility and Qualitative Study. \u003cem\u003eJMIR human factors\u003c/em\u003e. 2021;8(4):e26964.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBaxter SL, Bass JS, Sitapati AM. Barriers to implementing an artificial intelligence model for unplanned readmissions. \u003cem\u003eACI Open\u003c/em\u003e. 2020;4(2):e108-e113.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNong P, Adler-Milstein J. Socially situated risk: challenges and strategies for implementing algorithmic risk scoring for care management. \u003cem\u003eJAMIA open\u003c/em\u003e. 2021;4(3):ooab076.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSalwei ME, Anders S, Novak L, et al. Preventing clinical deterioration in cancer outpatients: Human centered design of a predictive model and response system. \u003cem\u003eJ Clin Orthod\u003c/em\u003e. 2022;40(16_suppl):e13567-e13567.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAct CC. Public Law 114\u0026ndash;255. 114 th Congress. Published online 2021.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHassan N, Slight R, Bimpong K, et al. Systematic review to understand users perspectives on AI-enabled decision aids to inform shared decision making. \u003cem\u003eNPJ Digit Med\u003c/em\u003e. 2024;7(1):332.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSittig DF, Singh H. A New Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems. In: Patel VL, Kannampallil TG, Kaufman DR, eds. \u003cem\u003eCognitive Informatics for Biomedicine\u003c/em\u003e. Health Informatics. Springer International Publishing; 2015:59\u0026ndash;80.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBourgeois FC, Sinha A, Tuli G, et al. The substance-exposed birthing person-infant/child dyad and health information exchange in the United States. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e. 2025;32(3):417\u0026ndash;425.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWilliams K, Nikolajski C, Rodriguez S, et al. Incorporating end-user perspectives into the development of a machine learning algorithm for first time perinatal depression prediction. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e. 2025;32(7):1186\u0026ndash;1198.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNational Academy of Medicine. \u003cem\u003eAn Artificial Intelligence Code of Conduct for Health and Medicine: Essential Guidance for Aligned Action\u003c/em\u003e. National Academies Press; 2025.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRussell RG, Lovett Novak L, Patel M, et al. Competencies for the Use of Artificial Intelligence-Based Tools by Health Care Professionals. \u003cem\u003eAcad Med\u003c/em\u003e. 2023;98(3):348\u0026ndash;356.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRatwani RM, Booker E. We need A federal framework for health care AI. \u003cem\u003eForefront Group\u003c/em\u003e. Published online January 16, 2025. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1377/forefront.20250114.858960\u003c/span\u003e\u003cspan address=\"10.1377/forefront.20250114.858960\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e\u003cem\u003eCurrent Use And Evaluation Of Artificial Intelligence And Predictive Models In US Hospitals: Article Examines Uses and Evaluation of Artificial Intelligence and\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAbulibdeh R, Celi LA, Sejdić E. The illusion of safety: A report to the FDA on AI healthcare product approvals. \u003cem\u003ePLOS Digit Health\u003c/em\u003e. 2025;4(6):e0000866.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOng JCL, Ning Y, Collins GS, et al. International partnership for governing generative artificial intelligence models in medicine. \u003cem\u003eNat Med\u003c/em\u003e. Published online June 30, 2025. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41591-025-03787-4\u003c/span\u003e\u003cspan address=\"10.1038/s41591-025-03787-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAydin S, Karabacak M, Vlachos V, Margetis K. Large language models in patient education: a scoping review of applications in medicine. \u003cem\u003eFront Med (Lausanne)\u003c/em\u003e. 2024;11:1477898.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLonghurst CA, Singh K, Chopra A, Atreja A, Brownstein JS. A call for artificial intelligence implementation science centers to evaluate clinical effectiveness. \u003cem\u003eNEJM AI\u003c/em\u003e. 2024;1(8). doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1056/aip2400223\u003c/span\u003e\u003cspan address=\"10.1056/aip2400223\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKohane IS. Compared with what? Measuring AI against the health care we have. \u003cem\u003eN Engl J Med\u003c/em\u003e. 2024;391(17):1564\u0026ndash;1566.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eChen JG. Reducing Pediatric Hospital-Acquired Venous Thromboembolism\u0026mdash;Overcoming Challenges Using Quality Improvement. \u003cem\u003eJAMA Netw Open\u003c/em\u003e. 2023;6(10):e2338275-e2338275.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMarkus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. \u003cem\u003eJ Biomed Inform\u003c/em\u003e. 2021;113:103655.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLomis K, Jeffries P, Palatta A, et al. Artificial intelligence for health professions educators. \u003cem\u003eNAM Perspect\u003c/em\u003e. 2021;2021. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.31478/202109a\u003c/span\u003e\u003cspan address=\"10.31478/202109a\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKleib M, Nagle LM, Furlong KE, Paul P, Duarte Wisnesky U, Ali S. Are Future Nurses Ready for Digital Health?: Informatics Competency Baseline Assessment. \u003cem\u003eNurse Educ\u003c/em\u003e. 2022;47(5):E98-E104.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGranger E, Skopac J. An ethical framework for the use of consumer generated data in health care. In: \u003cem\u003e2021 Health Datapalooza and National Health Policy Conference\u003c/em\u003e. AcademyHealth; 2021. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://academyhealth.confex.com/academyhealth/2021hdpnhpc/meetingapp.cgi/Paper/44315\u003c/span\u003e\u003cspan address=\"https://academyhealth.confex.com/academyhealth/2021hdpnhpc/meetingapp.cgi/Paper/44315\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDesai PM, Harkins S, Rahman S, et al. Visualizing machine learning-based predictions of postpartum depression risk for lay audiences. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e. 2024;31(2):289\u0026ndash;297.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSaldana J. \u003cem\u003eThe Coding Manual For Qualitative Researchers\u003c/em\u003e. Sage; 2012:328.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Artificial intelligence, predictive algorithms, bioethics, patient-centered care","lastPublishedDoi":"10.21203/rs.3.rs-7908218/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7908218/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eOur objective was to triangulate patient, health professional, and developer perspectives for implementing patient-centered artificial intelligence (AI) systems. We conducted semi-structured interviews with patients (N\u0026thinsp;=\u0026thinsp;18), health professionals (N\u0026thinsp;=\u0026thinsp;8), and AI developers (N\u0026thinsp;=\u0026thinsp;8). We created interview guides informed by frameworks in bioethics and health information informatics. We utilized a predictive algorithm for determining risk for postpartum depression as a use case to concretize our discussions. Our team analyzed transcripts from interview recordings using thematic, directed content analysis and the constant comparative process. Participants found mitigating potential harms caused by AI (e.g., bias, stigma, or patient anxiety) greatly important. They also believed that AI must provide clinical benefits by allowing health professionals and patients to easily take actions based on AI output. To take safe action, end users needed transparency to understand the AI\u0026rsquo;s accuracy and predictors driving risk. Patient participants wanted health professionals to interpret AI output, but health professionals did not always feel they had the time or training to do so. Participants also raised concerns regarding how data quality may affect AI accuracy, who may be responsible for inappropriate actions taken based on AI, and issues regarding data security, privacy, and accessibility. Our results support real-world implementation of more patient-centered AI tools by: providing health professionals with competencies for discussing AI-based risks; engaging patients and health professionals throughout the development process; inclusively communicating AI output to health professionals and patients; and implementing multi-layer systems of AI governance.\u003c/p\u003e","manuscriptTitle":"A qualitative Interview Study Investigating Patient, Health Professional, and Developer Perspectives on Real-World Implementation of Patient-Centered AI Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-07 01:27:58","doi":"10.21203/rs.3.rs-7908218/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-11-12T23:50:54+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-12T22:58:10+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-12T04:28:55+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-11T02:56:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"86526464486493408612793515357643955971","date":"2025-11-02T03:13:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"81647487088440100135400016919118537117","date":"2025-10-29T16:55:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"29503205805502483763985757420189394881","date":"2025-10-27T22:52:55+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"160254558392547431716965648614231600262","date":"2025-10-27T20:07:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"55553382035018973582289857109931963120","date":"2025-10-27T16:48:01+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-10-27T16:16:04+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-10-24T19:59:30+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-10-24T14:39:49+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2025-10-20T18:25:54+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d9d5319f-f3bd-4ee5-b276-8610ffe84866","owner":[],"postedDate":"November 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":57283204,"name":"Health sciences/Health care"},{"id":57283205,"name":"Physical sciences/Mathematics and computing"},{"id":57283206,"name":"Health sciences/Medical research"},{"id":57283207,"name":"Scientific community and society/Scientific community"}],"tags":[],"updatedAt":"2026-03-17T16:54:01+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-07 01:27:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7908218","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7908218","identity":"rs-7908218","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}