Electronic Health Record Developer Adherence to Federal AI Risk Transparency Requirements

Electronic Health Record Developer Adherence to Federal AI Risk Transparency Requirements | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report Electronic Health Record Developer Adherence to Federal AI Risk Transparency Requirements Nidhi Manikkoth, Ngan Vo, Jessica Handley, Josh Biro, Jordan Everson, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8255773/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 13 You are reading this latest preprint version Abstract We evaluated electronic health record developer compliance with federal transparency requirements for AI-enabled predictive decision support interventions. Developers are required to publicly report on risk analysis, mitigation, governance, and on eight other attributes. Across 35 developers, reporting was inconsistent and often incomplete, with substantial variation in structure and detail, yet all developers were certified as meeting the requirements. Results suggest a need for standardized reporting and more rigorous certification review. Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Scientific community and society/Scientific community Introduction Artificial intelligence (AI) algorithms are increasingly embedded in electronic health records (EHRs) to support diagnostic reasoning, risk prediction, clinical decision-making, workflow automation, and treatment recommendations. 1 These tools hold promise for enhancing patient safety by improving diagnostic consistency, reducing preventable errors, and providing timely clinical insights. 2 Beyond clinical decision support, emerging high-value uses of AI within EHRs include reimbursement optimization, operational management, and quality and safety monitoring. 3 However, concerns have emerged regarding model bias, data quality, explainability, inappropriate generalization, and their potential to introduce new forms of patient safety risks. 4 Studies have documented that AI-enabled decision support can propagate or amplify inequities when underlying training data are unrepresentative, lack important clinical context, or reflect structural biases in care delivery. 5 – 7 Additional risks—including inadequate validation, lack of transparency in model provenance, variable performance across demographic groups, and unclear lines of accountability—pose challenges for clinicians, patients, regulators, and developers. 8 – 10 To begin to address some of these concerns, the U.S. Department of Health and Human Services Assistant Secretary for Technology Policy (ASTP)/Office of the National Coordinator for Health Information Technology (ONC) sought to bring greater transparency around EHR developer practices related to AI algorithms embedded in their products. 11 These regulations recognize that EHR-integrated algorithms, particularly those offering predictive or diagnostic decision support, require clear communication about their development, intended use, evaluation, and known limitations to support safe and trustworthy adoption. 12 Under the Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing (HTI-1) Final Rule, certified health information technology (CHIT) developers (e.g. EHR developers) must provide publicly accessible Risk Management Summaries (RMS) for AI algorithms, which are broadly considered predictive decision support interventions (DSIs). 11 These summaries are required to describe the developer’s risk analysis process, mitigation strategies, and governance oversight. They should also address eight core attributes recommended for evaluating DSIs: validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. The intent of this requirement is to enable greater transparency for clinicians, health systems, purchasers, and regulators, and to promote accountability for responsible AI development. 13 Despite these expectations, little is known about how EHR developers interpret and operationalize the HTI-1 RMS requirements or how consistently they report information in practice. Understanding how developers interpret these requirements is essential because health systems increasingly rely on these tools without visibility into their safety or governance processes. To address this gap, we systematically analyzed all available EHR developer RMS to determine alignment with HTI-1 requirements and identify opportunities for policy refinement and improvements in EHR developer reporting practices. Methods Data Source and Sample Identification Using the publicly available HTI-1 certified health IT product list, we identified all products that included an AI-enabled predictive DSI. For each certified product, the corresponding RMS was downloaded for review. All documents were compiled, cataloged, and prepared for systematic review. Coding Framework and Procedures A structured coding framework was developed based on the requirements outlined in ONC’s HTI-1 Final Rule. The framework assessed whether each RMS (1) described its risk analysis process, (2) identified specific risks, (3) outlined risk mitigation strategies, (4) described governance structures, and (5) addressed any of the eight recommended attributes for evaluating DSIs: validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. Conceptual synonyms or closely related terminology (e.g., “bias mitigation” for fairness) were accepted as evidence that an attribute was addressed. Two reviewers were trained on the coding framework and independently coded all RMS documents. Coding included both binary indicators of attribute presence and qualitative notes describing the level of detail provided. Discrepancies were reviewed jointly and resolved through consensus. Analytic Approach We conducted descriptive quantitative analyses to summarize the frequency and distribution of reported risk management components across developers. Qualitative synthesis of reviewer notes was used to characterize patterns in reporting practices, variation in completeness and specificity, and common gaps relative to HTI-1 expectations. Because all data were publicly available and contained no identifiable information, this study did not require human subjects review. Results A total of 35 developers disclosed at least one AI-enabled DSI in their certified products. Among these, 71% (n = 25) described their overall risk analysis process, though the specificity of these descriptions varied. Only 57% (n = 20) explicitly identified the individual risks they monitor as part of this process. In contrast, most developers (91%, n = 32) outlined one or more mitigation strategies intended to address identified risks, and 80% (n = 28) reported having a formal governance structure in place to oversee risk management activities. When examining alignment with the eight DSI attributes required by ONC/ASTP, 77% (n = 27) referenced all eight attributes at least once within their RMS. However, the depth and clarity of these references differed considerably across developers, with some providing only brief or high-level statements and others offering more detailed explanations of how the attributes were assessed or addressed. Discussion These results demonstrate considerable variability in how EHR developers report their risk management practices for AI enabled predictive DSIs. While most developers attempted to address the elements required by the HTI-1 Final Rule, the scope and specificity of their reporting differed substantially. Some developers focused primarily on listing the risks they had identified while others described portions of the broader lifecycle, such as how risks are identified, how they are analyzed, or how mitigation strategies are implemented. In several cases, RMS documents provided only high-level statements without supporting detail. Despite these incomplete or uneven summaries, all products were certified as meeting the reporting requirements. The RMS documents varied widely in structure, terminology, and formatting. Developers used different organizational approaches and the level of explanation for required elements ranged from brief references to more extensive narrative descriptions. This inconsistency limits transparency and makes it difficult to compare practices across products or evaluate the robustness of individual developers’ risk management approaches. Even when information was provided, it was not always usable by a broad audience limiting the practical utility of these reports in supporting responsible adoption of AI-enabled decision support. To improve accessibility, ASTP/ONC should provide a standardized reporting template or structure for RMS submissions. Likewise, certification bodies responsible for verifying adherence to HTI-1 reporting requirements should ensure that developers provide sufficient and appropriately detailed information. Historically, some certification organizations have not consistently conducted rigorous reviews of developer submissions, which may contribute to the variability seen across RMS documents. 14 EHR developers themselves also have an opportunity to strengthen transparency by providing more complete descriptions of their risk management practices. Our findings echo concerns raised in recent literature about the current lack of uniformity in CDS governance frameworks and risk-management practices despite growing adoption of AI-enabled CDS in health systems. A recent review article calls for standardized reporting and local oversight to ensure equitable, safe, and transparent deployment; our observed inconsistencies in publicly available RMS documents indicate these recommendations are not yet realized. 15 This study has limitations, including reliance solely on publicly available RMS documents without access to proprietary or internal developer materials. As a result, the analysis reflects only what developers chose to disclose, not the full extent of their practices. It is possible that some variability reflects early-stage adoption of the RMS requirement, but even so, the inconsistencies highlight a need for clearer expectations and oversight. Nonetheless, the intent of the HTI-1 Final Rule is to foster responsible and safe integration of AI into healthcare by increasing visibility into developer risk management processes. 12 Achieving this goal will require greater standardization in reporting and more rigorous evaluation of RMS submissions to support transparency, accountability, and trust in predictive DSIs. Declarations Competing Interests The authors declare no competing interests. Funding Funding: Not applicable. Author Contribution NM, NV, RR and KM developed the coding framework. NM and NV performed the coding. NM, NV, and KM adjudicated discrepancies. JB, JH, JE, RR and KM conceptualized the study and reviewed the analyses. NM, NV, JB, JH, JE, RR and KM drafted, revised, and approved the final manuscript. References He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nature medicine . 2019;25(1):30-36. Bates DW, Levine D, Syrowatka A, et al. The potential of artificial intelligence to improve patient safety: a scoping review. NPJ digital medicine . 2021;4(1):54. Sahni NR, Carrus B. Artificial intelligence in US health care delivery. New England Journal of Medicine . 2023;389(4):348-358. Chustecki M. Benefits and risks of AI in health care: Narrative review. Interactive Journal of Medical Research . 2024;13(1):e53616. Hussain SA, Bresnahan M, Zhuang J. The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities–a scoping review. Ethnicity & Health . 2025;30(2):197-214. Chin MH, Afsar-Manesh N, Bierman AS, et al. Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. JAMA Network Open . 2023;6(12):e2345050-e2345050. Chen RJ, Wang JJ, Williamson DF, et al. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nature biomedical engineering . 2023;7(6):719-742. Reddy S, Rogers W, Makinen V-P, et al. Evaluation framework to guide implementation of AI systems into healthcare settings. BMJ health & care informatics . 2021;28(1):e100444. Fehr J, Citro B, Malpani R, Lippert C, Madai VI. A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare. Frontiers in Digital Health . 2024;6:1267290. Daneshjou R, Smith MP, Sun MD, Rotemberg V, Zou J. Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA dermatology . 2021;157(11):1362-1369. Services. USDoHaH. Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing. . 2024. Everson J, Smith J, Marchesini K, Tripathi M. A regulation to promote responsible AI in health care. Health Affairs Forefront . 2024; AI N. Artificial intelligence risk management framework (AI RMF 1.0). URL: https://nvlpubs nist gov/nistpubs/ai/nist ai . 2023:100-1. Ratwani RM, Benda NC, Hettinger AZ, Fairbanks RJ. Electronic health record vendor adherence to usability certification requirements and testing standards. Jama . 2015;314(10):1070-1071. Lin AL, Parrish AB, Cary M, et al. Algorithm-Based Clinical Decision Support: Evolving Regulatory Landscape and Best Practices for Local Oversight. Annual Review of Biomedical Data Science . 2025;8 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 07 Jan, 2026 Reviews received at journal 03 Jan, 2026 Reviews received at journal 02 Jan, 2026 Reviews received at journal 02 Jan, 2026 Reviews received at journal 27 Dec, 2025 Reviewers agreed at journal 23 Dec, 2025 Reviewers agreed at journal 18 Dec, 2025 Reviewers agreed at journal 11 Dec, 2025 Reviewers agreed at journal 11 Dec, 2025 Reviewers invited by journal 09 Dec, 2025 Editor assigned by journal 08 Dec, 2025 Submission checks completed at journal 08 Dec, 2025 First submitted to journal 01 Dec, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8255773","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":558838320,"identity":"29f65d68-50cf-42a2-931e-3258f181b7fb","order_by":0,"name":"Nidhi Manikkoth","email":"","orcid":"","institution":"Georgetown University","correspondingAuthor":false,"prefix":"","firstName":"Nidhi","middleName":"","lastName":"Manikkoth","suffix":""},{"id":558838321,"identity":"57863438-cbba-48bb-8369-647bd2ed46a8","order_by":1,"name":"Ngan Vo","email":"","orcid":"","institution":"Georgetown University","correspondingAuthor":false,"prefix":"","firstName":"Ngan","middleName":"","lastName":"Vo","suffix":""},{"id":558838322,"identity":"50ba6acc-1417-42bf-b217-278dc84ac9b2","order_by":2,"name":"Jessica Handley","email":"","orcid":"","institution":"MedStar Health National Center for Human Factors in Healthcare","correspondingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"","lastName":"Handley","suffix":""},{"id":558838323,"identity":"0d528d7c-4144-4c45-a4ca-ae674612a872","order_by":3,"name":"Josh Biro","email":"","orcid":"","institution":"MedStar Health National Center for Human Factors in Healthcare","correspondingAuthor":false,"prefix":"","firstName":"Josh","middleName":"","lastName":"Biro","suffix":""},{"id":558838324,"identity":"b20f63b3-27c3-497a-bedf-3c384a1d4fac","order_by":4,"name":"Jordan Everson","email":"","orcid":"","institution":"Georgetown University","correspondingAuthor":false,"prefix":"","firstName":"Jordan","middleName":"","lastName":"Everson","suffix":""},{"id":558838325,"identity":"e2f9ce39-a16a-44b2-8e6d-2606f5b2ef8b","order_by":5,"name":"Raj M. Ratwani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGklEQVRIie3QsUrDQBzH8X85sMtR14RA8woXAi0Omlf5HwFdMksGoTfpcg+gUHyGTpmvHLRLIGskgpFCJzdBKpTipaWTHDp2uO8QCMknPy4ALtcJlghAQmF3uMPcXNQfhClDpqCB7knZEXJ4RO0EyPZIevf/IUvVkgx0mHjpun1/LoaDqpp/5vA6TGxnkYiG7CLpXY8ZL5rYr1Pil7CObStXsCebngxw5BnCZzUBX4Dm0kLi87b7Y2Uig5svj0+byazS5NuQiY1Ej9iRBZdBZlZEg0ylZ90KWo9ft6gp06l8+7j1cNFET3U6uhBMR1JZSJXxFc315YNcFv7mrgkH1Xz1InId9oVlBigqYL8/ZXu9q2/Zd7lcLtexHzr0ZGfe+s97AAAAAElFTkSuQmCC","orcid":"","institution":"MedStar Health National Center for Human Factors in Healthcare","correspondingAuthor":true,"prefix":"","firstName":"Raj","middleName":"M.","lastName":"Ratwani","suffix":""},{"id":558838326,"identity":"284bae8a-0569-4414-a85d-0815c28eba63","order_by":6,"name":"Kristen Miller","email":"","orcid":"","institution":"MedStar Health National Center for Human Factors in Healthcare","correspondingAuthor":false,"prefix":"","firstName":"Kristen","middleName":"","lastName":"Miller","suffix":""}],"badges":[],"createdAt":"2025-12-02 04:23:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8255773/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8255773/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98058858,"identity":"cc11eb7f-a1b7-4e60-86d8-68f96774e560","added_by":"auto","created_at":"2025-12-12 10:26:08","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":28179,"visible":true,"origin":"","legend":"","description":"","filename":"ONCASTPNPJDigitalMedicine7Dec2025.docx","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/3607256507bafa8b74b6296e.docx"},{"id":98058857,"identity":"a64756a4-88b8-41be-a2d4-cb446608d487","added_by":"auto","created_at":"2025-12-12 10:26:08","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7158,"visible":true,"origin":"","legend":"","description":"","filename":"4a9648138cac417fb8365d52d6c57014.json","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/05c7ad73d6415c847363bc63.json"},{"id":98058861,"identity":"0160da3e-6c3c-4793-969e-35aad56a078b","added_by":"auto","created_at":"2025-12-12 10:26:08","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":36582,"visible":true,"origin":"","legend":"","description":"","filename":"4a9648138cac417fb8365d52d6c570141enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/d018a6a341730c5dd0042156.xml"},{"id":98058859,"identity":"e55169d2-7c39-4fe9-9087-6a9eadda66b2","added_by":"auto","created_at":"2025-12-12 10:26:08","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":35415,"visible":true,"origin":"","legend":"","description":"","filename":"4a9648138cac417fb8365d52d6c570141structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/f5d11876a50a5054e590a36f.xml"},{"id":98428431,"identity":"ab2ae07d-36b6-4ac5-b6d1-d45bae578338","added_by":"auto","created_at":"2025-12-17 16:42:01","extension":"html","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":41687,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/888c878693693451889fbf97.html"},{"id":98444679,"identity":"6c6ec850-8b44-4633-be49-b23db695b40f","added_by":"auto","created_at":"2025-12-17 17:16:51","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":302467,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8255773/v1/593d7e63-5af1-44ed-9334-af2e6e6ff2ed.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Electronic Health Record Developer Adherence to Federal AI Risk Transparency Requirements","fulltext":[{"header":"Introduction","content":"\u003cp\u003eArtificial intelligence (AI) algorithms are increasingly embedded in electronic health records (EHRs) to support diagnostic reasoning, risk prediction, clinical decision-making, workflow automation, and treatment recommendations.\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e These tools hold promise for enhancing patient safety by improving diagnostic consistency, reducing preventable errors, and providing timely clinical insights.\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e Beyond clinical decision support, emerging high-value uses of AI within EHRs include reimbursement optimization, operational management, and quality and safety monitoring.\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e However, concerns have emerged regarding model bias, data quality, explainability, inappropriate generalization, and their potential to introduce new forms of patient safety risks.\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Studies have documented that AI-enabled decision support can propagate or amplify inequities when underlying training data are unrepresentative, lack important clinical context, or reflect structural biases in care delivery.\u003csup\u003e\u003cspan additionalcitationids=\"CR6\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e Additional risks\u0026mdash;including inadequate validation, lack of transparency in model provenance, variable performance across demographic groups, and unclear lines of accountability\u0026mdash;pose challenges for clinicians, patients, regulators, and developers.\u003csup\u003e\u003cspan additionalcitationids=\"CR9\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eTo begin to address some of these concerns, the U.S. Department of Health and Human Services Assistant Secretary for Technology Policy (ASTP)/Office of the National Coordinator for Health Information Technology (ONC) sought to bring greater transparency around EHR developer practices related to AI algorithms embedded in their products.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e These regulations recognize that EHR-integrated algorithms, particularly those offering predictive or diagnostic decision support, require clear communication about their development, intended use, evaluation, and known limitations to support safe and trustworthy adoption.\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eUnder the Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing (HTI-1) Final Rule, certified health information technology (CHIT) developers (e.g. EHR developers) must provide publicly accessible Risk Management Summaries (RMS) for AI algorithms, which are broadly considered predictive decision support interventions (DSIs).\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e These summaries are required to describe the developer\u0026rsquo;s risk analysis process, mitigation strategies, and governance oversight. They should also address eight core attributes recommended for evaluating DSIs: validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. The intent of this requirement is to enable greater transparency for clinicians, health systems, purchasers, and regulators, and to promote accountability for responsible AI development.\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eDespite these expectations, little is known about how EHR developers interpret and operationalize the HTI-1 RMS requirements or how consistently they report information in practice. Understanding how developers interpret these requirements is essential because health systems increasingly rely on these tools without visibility into their safety or governance processes. To address this gap, we systematically analyzed all available EHR developer RMS to determine alignment with HTI-1 requirements and identify opportunities for policy refinement and improvements in EHR developer reporting practices.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eData Source and Sample Identification\u003c/h2\u003e\u003cp\u003eUsing the publicly available HTI-1 certified health IT product list, we identified all products that included an AI-enabled predictive DSI. For each certified product, the corresponding RMS was downloaded for review. All documents were compiled, cataloged, and prepared for systematic review.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eCoding Framework and Procedures\u003c/h3\u003e\n\u003cp\u003eA structured coding framework was developed based on the requirements outlined in ONC\u0026rsquo;s HTI-1 Final Rule. The framework assessed whether each RMS (1) described its risk analysis process, (2) identified specific risks, (3) outlined risk mitigation strategies, (4) described governance structures, and (5) addressed any of the eight recommended attributes for evaluating DSIs: validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. Conceptual synonyms or closely related terminology (e.g., \u0026ldquo;bias mitigation\u0026rdquo; for fairness) were accepted as evidence that an attribute was addressed. Two reviewers were trained on the coding framework and independently coded all RMS documents. Coding included both binary indicators of attribute presence and qualitative notes describing the level of detail provided. Discrepancies were reviewed jointly and resolved through consensus.\u003c/p\u003e\n\u003ch3\u003eAnalytic Approach\u003c/h3\u003e\n\u003cp\u003eWe conducted descriptive quantitative analyses to summarize the frequency and distribution of reported risk management components across developers. Qualitative synthesis of reviewer notes was used to characterize patterns in reporting practices, variation in completeness and specificity, and common gaps relative to HTI-1 expectations. Because all data were publicly available and contained no identifiable information, this study did not require human subjects review.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eA total of 35 developers disclosed at least one AI-enabled DSI in their certified products. Among these, 71% (n\u0026thinsp;=\u0026thinsp;25) described their overall risk analysis process, though the specificity of these descriptions varied. Only 57% (n\u0026thinsp;=\u0026thinsp;20) explicitly identified the individual risks they monitor as part of this process. In contrast, most developers (91%, n\u0026thinsp;=\u0026thinsp;32) outlined one or more mitigation strategies intended to address identified risks, and 80% (n\u0026thinsp;=\u0026thinsp;28) reported having a formal governance structure in place to oversee risk management activities.\u003c/p\u003e\u003cp\u003eWhen examining alignment with the eight DSI attributes required by ONC/ASTP, 77% (n\u0026thinsp;=\u0026thinsp;27) referenced all eight attributes at least once within their RMS. However, the depth and clarity of these references differed considerably across developers, with some providing only brief or high-level statements and others offering more detailed explanations of how the attributes were assessed or addressed.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThese results demonstrate considerable variability in how EHR developers report their risk management practices for AI enabled predictive DSIs. While most developers attempted to address the elements required by the HTI-1 Final Rule, the scope and specificity of their reporting differed substantially. Some developers focused primarily on listing the risks they had identified while others described portions of the broader lifecycle, such as how risks are identified, how they are analyzed, or how mitigation strategies are implemented. In several cases, RMS documents provided only high-level statements without supporting detail. Despite these incomplete or uneven summaries, all products were certified as meeting the reporting requirements.\u003c/p\u003e\u003cp\u003eThe RMS documents varied widely in structure, terminology, and formatting. Developers used different organizational approaches and the level of explanation for required elements ranged from brief references to more extensive narrative descriptions. This inconsistency limits transparency and makes it difficult to compare practices across products or evaluate the robustness of individual developers\u0026rsquo; risk management approaches. Even when information was provided, it was not always usable by a broad audience limiting the practical utility of these reports in supporting responsible adoption of AI-enabled decision support. To improve accessibility, ASTP/ONC should provide a standardized reporting template or structure for RMS submissions. Likewise, certification bodies responsible for verifying adherence to HTI-1 reporting requirements should ensure that developers provide sufficient and appropriately detailed information. Historically, some certification organizations have not consistently conducted rigorous reviews of developer submissions, which may contribute to the variability seen across RMS documents.\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e EHR developers themselves also have an opportunity to strengthen transparency by providing more complete descriptions of their risk management practices.\u003c/p\u003e\u003cp\u003eOur findings echo concerns raised in recent literature about the current lack of uniformity in CDS governance frameworks and risk-management practices despite growing adoption of AI-enabled CDS in health systems. A recent review article calls for standardized reporting and local oversight to ensure equitable, safe, and transparent deployment; our observed inconsistencies in publicly available RMS documents indicate these recommendations are not yet realized.\u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eThis study has limitations, including reliance solely on publicly available RMS documents without access to proprietary or internal developer materials. As a result, the analysis reflects only what developers chose to disclose, not the full extent of their practices. It is possible that some variability reflects early-stage adoption of the RMS requirement, but even so, the inconsistencies highlight a need for clearer expectations and oversight. Nonetheless, the intent of the HTI-1 Final Rule is to foster responsible and safe integration of AI into healthcare by increasing visibility into developer risk management processes.\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e Achieving this goal will require greater standardization in reporting and more rigorous evaluation of RMS submissions to support transparency, accountability, and trust in predictive DSIs.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003ch2\u003eCompeting Interests\u003c/h2\u003e\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\u003c/p\u003e\u003cp\u003e\u003ch2\u003eFunding\u003c/h2\u003e\u003cp\u003eFunding: Not applicable.\u003c/p\u003e\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eNM, NV, RR and KM developed the coding framework. NM and NV performed the coding. NM, NV, and KM adjudicated discrepancies. JB, JH, JE, RR and KM conceptualized the study and reviewed the analyses. NM, NV, JB, JH, JE, RR and KM drafted, revised, and approved the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eHe J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. \u003cem\u003eNature medicine\u003c/em\u003e. 2019;25(1):30-36.\u003c/li\u003e\n\u003cli\u003eBates DW, Levine D, Syrowatka A, et al. The potential of artificial intelligence to improve patient safety: a scoping review. \u003cem\u003eNPJ digital medicine\u003c/em\u003e. 2021;4(1):54.\u003c/li\u003e\n\u003cli\u003eSahni NR, Carrus B. Artificial intelligence in US health care delivery. \u003cem\u003eNew England Journal of Medicine\u003c/em\u003e. 2023;389(4):348-358.\u003c/li\u003e\n\u003cli\u003eChustecki M. Benefits and risks of AI in health care: Narrative review. \u003cem\u003eInteractive Journal of Medical Research\u003c/em\u003e. 2024;13(1):e53616.\u003c/li\u003e\n\u003cli\u003eHussain SA, Bresnahan M, Zhuang J. The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities\u0026ndash;a scoping review. \u003cem\u003eEthnicity \u0026amp; Health\u003c/em\u003e. 2025;30(2):197-214.\u003c/li\u003e\n\u003cli\u003eChin MH, Afsar-Manesh N, Bierman AS, et al. Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. \u003cem\u003eJAMA Network Open\u003c/em\u003e. 2023;6(12):e2345050-e2345050.\u003c/li\u003e\n\u003cli\u003eChen RJ, Wang JJ, Williamson DF, et al. Algorithmic fairness in artificial intelligence for medicine and healthcare. \u003cem\u003eNature biomedical engineering\u003c/em\u003e. 2023;7(6):719-742.\u003c/li\u003e\n\u003cli\u003eReddy S, Rogers W, Makinen V-P, et al. Evaluation framework to guide implementation of AI systems into healthcare settings. \u003cem\u003eBMJ health \u0026amp; care informatics\u003c/em\u003e. 2021;28(1):e100444.\u003c/li\u003e\n\u003cli\u003eFehr J, Citro B, Malpani R, Lippert C, Madai VI. A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare. \u003cem\u003eFrontiers in Digital Health\u003c/em\u003e. 2024;6:1267290.\u003c/li\u003e\n\u003cli\u003eDaneshjou R, Smith MP, Sun MD, Rotemberg V, Zou J. Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review. \u003cem\u003eJAMA dermatology\u003c/em\u003e. 2021;157(11):1362-1369.\u003c/li\u003e\n\u003cli\u003eServices. USDoHaH. Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing. . 2024.\u003c/li\u003e\n\u003cli\u003eEverson J, Smith J, Marchesini K, Tripathi M. A regulation to promote responsible AI in health care. \u003cem\u003eHealth Affairs Forefront\u003c/em\u003e. 2024;\u003c/li\u003e\n\u003cli\u003eAI N. Artificial intelligence risk management framework (AI RMF 1.0). \u003cem\u003eURL: \u003c/em\u003e\u003cem\u003ehttps://nvlpubs\u003c/em\u003e\u003cem\u003e nist gov/nistpubs/ai/nist ai\u003c/em\u003e. 2023:100-1.\u003c/li\u003e\n\u003cli\u003eRatwani RM, Benda NC, Hettinger AZ, Fairbanks RJ. Electronic health record vendor adherence to usability certification requirements and testing standards. \u003cem\u003eJama\u003c/em\u003e. 2015;314(10):1070-1071.\u003c/li\u003e\n\u003cli\u003eLin AL, Parrish AB, Cary M, et al. Algorithm-Based Clinical Decision Support: Evolving Regulatory Landscape and Best Practices for Local Oversight. \u003cem\u003eAnnual Review of Biomedical Data Science\u003c/em\u003e. 2025;8\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8255773/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8255773/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWe evaluated electronic health record developer compliance with federal transparency requirements for AI-enabled predictive decision support interventions. Developers are required to publicly report on risk analysis, mitigation, governance, and on eight other attributes. Across 35 developers, reporting was inconsistent and often incomplete, with substantial variation in structure and detail, yet all developers were certified as meeting the requirements. Results suggest a need for standardized reporting and more rigorous certification review.\u003c/p\u003e","manuscriptTitle":"Electronic Health Record Developer Adherence to Federal AI Risk Transparency Requirements","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-12 10:25:59","doi":"10.21203/rs.3.rs-8255773/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-01-08T03:58:18+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-04T04:45:48+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-02T15:43:44+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-02T08:32:05+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-27T13:06:06+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"31620296167239618628899617512436709215","date":"2025-12-23T15:15:35+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"44567598857121117169485825510877767837","date":"2025-12-18T12:20:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"228792391090547738500524491368137190557","date":"2025-12-11T12:44:55+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"32131471546993702306250879222129582690","date":"2025-12-11T09:50:32+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-09T08:25:04+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-12-09T01:25:55+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-12-08T18:24:02+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2025-12-02T04:11:38+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7994653f-68a0-4cc1-9a1b-1848d2bf36f6","owner":[],"postedDate":"December 12th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":59491162,"name":"Health sciences/Health care"},{"id":59491163,"name":"Physical sciences/Mathematics and computing"},{"id":59491164,"name":"Health sciences/Medical research"},{"id":59491165,"name":"Scientific community and society/Scientific community"}],"tags":[],"updatedAt":"2026-05-09T17:53:29+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-12 10:25:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8255773","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8255773","identity":"rs-8255773","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}