The Clinical Sustainability Assessment Tool: Measuring Organizational Capacity to Promote Sustainability in Healthcare

The Clinical Sustainability Assessment Tool: Measuring Organizational Capacity to Promote Sustainability in Healthcare | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Methodology The Clinical Sustainability Assessment Tool: Measuring Organizational Capacity to Promote Sustainability in Healthcare Sara Malone, Kim Prewitt, Rachel Hackett, John C. Lin, Virginia McKay, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-58012/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 17 Jul, 2021 Read the published version in Implementation Science Communications → Version 1 posted You are reading this latest preprint version Abstract Background Few validated assessment tools are available to support sustainable implementation, an increasingly recognized need among clinicians. We describe the development of the Clinical Sustainability Assessment Tool, or CSAT, designed to assess factors that contribute to sustainable practices in clinical settings. Methods Fifty participants from clinical and research fields were recruited to brainstorm factors that lead to sustained clinical practices. Once repeated factors were removed, participants sorted remaining factors based on similarity. Using concept mapping analyses, we grouped these factors into meaningful domains and developed an initial tool. We then recruited 126 practicing clinicians to pilot and evaluate the tool. Individuals were recruited from clinical settings across pediatric (53%) and adult (47%) medical and surgical subspecialties, and averaged 11 years in their position. The data were analyzed using confirmatory factor analysis (CFA) to test hypothesized subscale structure in the instrument. We used root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) to assess fit and thus the ability of CSAT to measure the identified domains. Results The concept mapping resulted in seven domains and 47 items. The pilot and CFA resulted in 35 items, five per domain. The RMSEA of 0.084 and the SRMR of 0.075 indicated good fit. The final domains were: engaged staff and leadership, engaged stakeholders, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness. Conclusions The CSAT is a new reliable assessment tool which allows for greater practical and scientific understanding of contextual factors that enable sustainable clinical practices over time. The full CSAT instrument is available at https://sustaintool.org . Health Policy sustainability healthcare clinical science measurement development reliability Figures Figure 1 Figure 2 Figure 3 Contributions To The Literature Sustainability of evidence-based practices is important to deliver intended health outcomes to individuals and populations. However, there has been relatively less focus on sustainability theory and methods in implementation science. We introduce the Clinical Sustainability Assessment Tool (CSAT), a new instrument for measuring sustainability capacity in clinical and healthcare settings. The CSAT is designed to be easy to use—it is short, requires little training, and can be used by researchers, evaluators, and frontline clinical staff. The CSAT has excellent reliability, and preliminary validation data suggest that the CSAT is able to distinguish among different types of clinical settings. Background Sustainability is the degree to which an evidence-based program or intervention can deliver its intended benefits over an extended period of time [ 1 ]. Programs need to be sustained and clinical interventions continuously delivered to achieve their desired public health and clinical outcomes and impacts [ 2 ]. Sustainability can be challenging, and in fact many initially successful programs fail to remain in place over time. Evidence suggests that interventions that are not sustained could result in decreased care quality and worsened patient outcomes and quality of life [ 3 , 4 ]. Programs that are not kept up after an initial investment can waste money and non-monetary resources [ 5 , 6 ]. Having a better understanding of what factors affect sustainability can inform strategies to improve the likelihood that interventions will successfully continue after implementation [ 7 ]. Studies within implementation science often focus on initial program or intervention adoption and implementation, placing less emphasis on sustainability [ 8 ]. In two recent reviews of sustainability, the authors found relatively few studies focusing on sustainability, most of which were of limited quality and methodological rigor [ 9 , 4 , 10 ]. Research on sustainability is limited in part because data collection for grant funded studies is typically not carried out past the funding end date and is beyond the scope of many public health and health services studies [ 11 ]. Consequently, there have been recent calls for increased attention to sustainability research, theory, and methods [ 4 , 10 ]. Partly to address this critical gap in the field, we developed the Program Sustainability Assessment Tool (PSAT), which was designed to measure capacity for program sustainability [ 12 ], particularly for public health evidence-based programs. The PSAT has been widely used around the world for local, regional, and national programming [ 13 ]. Although it was developed for public health programs, it has also been used by social service, clinical care, and education programs to assess sustainability of a variety of programs focused on transitional services, obesity prevention, behavioral health, health systems strengthening, and tobacco control. The PSAT has been used to examine sustainability within clinical and healthcare settings, but early reports from users highlighted a number of limitations. Most essentially, clinical settings differ from public health settings in a number of ways which change how sustainability should be conceptualized and measured. For example, funding in clinical settings tends to be tied primarily to public and private insurance funding, rather than public grants and/or private donations for public health initiatives [ 14 ]. Thus, the sustainability of a practice is often tied to its ability to be reimbursed by payors [ 15 , 16 ]. Furthermore, community outreach, a key driver of sustainability in public health settings, is conducted less in clinical settings as services are mainly delivered to identified in-services patients [ 17 ]. Finally, although data are somewhat thin, current evidence suggests that sustaining evidence-based practices are challenging in clinical settings (as they are in public health settings). For example, a review of practices in the National Health Service (NHS) in the UK found that 33% of quality improvement projects are not sustained for one year after initial implementation [ 18 , 19 ]. Recently, a changing context in clinical healthcare has also become relevant to the implementation and integration of evidence-based practices. Clinical healthcare has widely adopted electronic medical records, dramatically changing how practitioners chart, build workflows, and communicate with other team members. This change has altered the structure in different health settings and how professionals orient themselves to medical decisions as well as other team members. New evidence-based clinical practices need to be integrated into this complex information and work system to be able to sustain them over time. Furthermore, sustainability outcomes in clinical settings play out over shorter time periods than for public health programming that is focused more on population health outcomes. Finally, the rapid cycle improvement that has historically been promoted in clinical settings and quality improvement initiatives has failed to focus on or understand the role of long term sustainability and organizational capacity within this realm [ 20 – 22 ]. To address need for additional research focused on sustainability and to support clinicians interested in planning for sustainable implementation of innovations, we present a new instrument for measuring sustainability capacity in clinical and healthcare settings: the Clinical Sustainability Assessment Tool (CSAT). In this paper, we describe the conceptual and measurement development of the CSAT, and present pilot data demonstrating its usability, reliability, and preliminary validation with a sample of healthcare practitioners and researchers. We also outline next steps for the CSAT, including broader dissemination and validation of the tool. Methods This is a measurement development study of a tool to assess capacity for sustainability in clinical healthcare settings. The study included two major components: 1) conceptual development guided by a literature review and expert opinion (via concept mapping); and 2) a pilot study of an initial version of the CSAT to assess its reliability and produce a final version of the instrument ready for dissemination. The CSAT was designed to be easy to use by researchers, evaluators, and practitioners, and applicable to a wide variety of healthcare settings. Conceptual Development After completing a review of existing literature and measures, a concept mapping approach was utilized to develop the CSAT [23]. Concept mapping is a mixed methods approach to reveal the structure of a complex conceptual domain, and is a useful tool for measurement development [24, 25]. We used concept mapping to conceptualize sustainability of practices in clinical settings, which resulted in the production and refinement of the domain structure for the measure. More than 60 participants from multiple professions and different healthcare settings participated in an initial brainstorming activity. Respondents included: Researchers in areas of clinical care, sustainability, and dissemination and implementation science (n=10) Clinical care administrators (n=9) Clinical care service providers (e.g., physicians, nurses, rehabilitation specialists; n=43) These participants represented various specialties and clinical environments (i.e., inpatient, outpatient, adult, and pediatric settings) These participants responded to the prompt, “In order for clinical practices to continue over time, they need…” and generated a list of 230 statements. Researchers reviewed the list of statements and removed any duplicate or nonsensical items and edited for spelling and grammar. The final list included 97 statements. In a subsequent activity, participants then grouped similar the statements together and rated each statement on dimensions of feasibility and importance. Figure 1 shows the concept mapping results after sorting the 97 statements. Seven domains emerged based on the underlying items (points in the figure) and literature on sustainability. The resulting conceptual domains were: Engaged Staff & Leadership, Engaged Stakeholders, Monitoring & Evaluation, Organizational Context & Capacity, Workflow Integration, Planning & Implementation, and Outcomes & Effectiveness . CSAT Instrument Development & Pilot Testing CSAT initial instrument development . For each conceptual domain, items were selected from the concept mapping responses that best reflected the meaning of its domain. A total of seven statements were selected for each of the domains for a total of 49 indicators. Each statement was edited so that CSAT respondents would be asked to assess the extent to which each existed in their clinic or setting on a Likert-type scale from 1 (no extent) to 7 (great extent). An example item was, “The practice has adequate staff to achieve its goals.” Participants & Recruitment. The Center for Public Health Systems Science research team invited individuals for a pilot test of the instrument (Table 1). This convenience sample was made up of participants selected from different clinical work environments as well as from different healthcare professions. Recruitment efforts used a snowball sampling approach, and included identifying and contacting stakeholders that could potentially benefit from using the tool, as well as promoting the CSAT at local and national dissemination and implementation conferences. Respondents had the option to forward the link to peers or nominate individuals to complete the CSAT. To incentivize participation, all respondents were offered an optional tailored sustainability results report and the opportunity to enter a drawing for one of five $50 gift cards. The final sample size was 120 participants (Table 1). Data Collection & Instrument . Data collection was administered through an electronic survey on Qualtrics [26]. The assessment consisted of 64 questions total, including the 49 on the pilot measure, demographic questions, and tool usability questions. Demographic items included questions about the individual, their profession, and their practice setting. Usability questions were adapted from the System Usability Scale [27] and were used to understand participant reaction to completing the CSAT and give insight into any concerns with functionality of the tool. Data Management & Analyses. Data were downloaded and managed in R. A variety of descriptive, visualization, and psychometric analyses were conducted to explore the item, domain, and instrument characteristics of the CSAT, with particular emphasis on reliability. The classical test theory package (CTT, Version 2.3.3) in R was used to calculate measures of internal consistency, particularly Cronbach’s alpha. Confirmatory factor analyses were done using the lavaan package (Version 0.6-5) [28]. Confirmatory factor analysis (CFA) is a powerful and appropriate tool for testing a hypothesized subscale structure in a measurement instrument [29, 30]. Initially CFA was applied to the pilot CSAT (with 7 items per subscale) to identify poorly performing items and test our hypothesized sustainability domain structure. Poor items were those that had low variability and/or poor fit with the intended subscale. Once the final structure was determined, we reran the CFA on the smaller, final version of the CSAT (5 items per subscale). For both tests, we used three measures of model fit to assess model adequacy: the comparative fit index (CFI), root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) [31]. Demographics were further assessed to understanding potential differences in responses and understanding how these variables might influence sustainability capacity. The Guidelines for Reporting Reliability and Agreement Studies were utilized for reporting [32]. Results Instrument improvement and domain structure Item and confirmatory factor analyses were used to determine the final structure of the CSAT. Table 2 shows the improved psychometrics during the instrument development process. The baseline model is used as a comparison for the pilot and final models. The pilot model includes all initial 49 items contained in seven subscales. After psychometric analyses, 14 items were dropped from the pilot version of the framework and tool. Items were dropped if they had lower loadings in the latent factors, had poor variance ( i.e., restricted range), or had excessive missing data. The final CSAT is comprised of 35 items organized within 7 subscale domains. Each domain has five items. This simple and balanced structure facilitates training and scoring with practices and groups. The domain structure and final items of the CSAT can be seen in Additional file 1. The first domain, Engaged Staff & Leadership, includes items assessing the extent to which the clinical practice has the support of internal frontline staff and management within the organization. Engaged Stakeholders assesses the extent to which the practice has support among external stakeholders. Organizational Readiness measures whether the organization has the internal supports and resources needed to effectively manage the practice. Workflow Integration refers to whether the practice has been designed to fit into existing workplace processes, policies, and technologies (e.g., EMR systems). Implementation & Training reflects whether the organization promotes processes and learning that appropriately guide the direction, goals, and strategies of the clinical practice. Monitoring & Evaluation assesses the extent to which the organization monitors the clinical practice and uses data to inform quality improvement. Finally, Outcomes & Effectiveness refers to whether and how the organization measures practice outcomes and impacts. Each subscale is scored separately, and an overall CSAT score can be obtained, ranging from 1 to 7. Higher scores indicate greater organizational capacity for clinical sustainability. Table 2 also shows the relatively good fit of the 7 factor CFA model to the data—that is, the 35 item CSAT does a credible job of measuring seven important clinical sustainability domains that were identified in the literature and in the concept mapping phase of the study. Although the CFI could be larger, the RMSEA of 0.084 is just above the target for an acceptable fit (0.08), and the SRMR of 0.075 is smaller than 0.08, indicating a good fit [33, 34]. The most important pattern in the CFA results presented in Table 2 is the improvement in model fit as we move from a single factor model, to a seven subscale model with all items, and finally the seven subscale model with the reduced number of items. The overall results suggest that the CSAT measures distinctive aspects of clinical sustainability with a relatively modest number of items that promotes ease of use. Subscale reliability Table 3 presents the subscale reliabilities (internal consistency) for the CSAT. The average internal consistency of the seven subscales is 0.88, and ranges from 0.82 to 0.94. These indicate excellent scale reliability, especially given the small size of each subscale (5 items) [35]. (Internal consistency always goes up with more items, so the most desirable goal is the fewest items that still maintains high reliability [33].) Furthermore, the item loadings show consistently high intercorrelations with their respective subscales, indicating good fit of individual items with overall subscale scores (full results available from authors). Preliminary CSAT results and validation Subdomain scores are the simple means of the five items in each domain. The total CSAT score is then the mean of the seven subdomain scores. Table 4 presents descriptive statistics for the seven subscales and the total scores. Organizational Readiness has the lowest average score, while Outcomes & Effectiveness is rated the highest. The standard deviations, ranges, and the density plot shown in Figure 2 show that there is good variability of scores, and no major problems with restricted range. The CSAT total scores range from 2.3 to 6.9. The standard deviation for the total scores is lower than for the subdomain scores, which is expected given that the total score is the mean of the seven subdomain scores. In addition to the CSAT scores, basic characteristics of the clinical setting were collected (i.e., patient, setting, and environment type) as well as two characteristics of the respondent (i.e., job position, and training profession). As part of a set of preliminary validational analyses, CSAT total and subscale scores were examined across these five setting and respondent characteristics. In terms of setting characteristics, total CSAT scores varied significantly by setting type ( F = 3.16, p = .047) and environment ( F = 2.93, p = .038), but not by patient type ( F = 1.09, p = .299). CSAT scores did not vary by respondent’s profession ( F = 0.93, p = .449) or job position ( F = 1.69, p = .175). Figure 3 shows the CSAT subscale score profile plots for the three setting (top row) and two respondent (bottom row) characteristics. Domain scores were very similar based on patient age category but showed significant differences based on outpatient vs. inpatient clinical setting and academic vs. community vs. private practice environment. Outpatient settings report consistently lower sustainability scores across all seven domains than inpatient settings. Academic medical centers and community hospitals were assessed as having higher sustainability capacities than community health centers and private practices. Type of environment is a little more complicated, but it appears that community hospitals and academic medical centers report higher sustainability than community health centers and private practice settings. Usability testing We also collected data regarding ease of use and asked participants to report on their experience using the CSAT. On average, it took participants just under 20 minutes to complete the longer initial 49 item version of the CSAT and just under 10 minutes to complete the final 35 item tool. Participants also rated the experience of completing the CSAT in a positive manner: 85% of participants rated the tool as easy to use; 75% felt very confident about their ability to use the tool; 90% thought that most other people would be able to learn quickly how to use the tool. Importantly, only 35% thought that they would need external support to use the tool effectively. In addition to the usability data from our pilot participants, we assessed the final 35 item version of the CSAT using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS)[36]. The CSAT rates good or excellent in all five of the PAPERS pragmatic categories: brevity, cost, ease of training, ease of interpretation, and language. The CSAT rates good in brevity with less than 40 items and language at less than a 10 th grade reading level. It rates excellent in cost, ease of training, and ease of interpretation being free, requiring no training to use, and automated calculations of scores. Discussion We introduce here the Clinical Sustainability Assessment Tool (CSAT) as a reliable instrument to assess organizational capacity for sustainability in clinical settings. The final CSAT includes seven distinct subscale domains with high internal consistency within each domain. The CSAT also demonstrates good variability with respect to clinical setting and practice, but maintained consistency across respondents and patients. More specifically, these instrument development results provide important initial validational support for the CSAT. The CSAT is designed to assess sustainability of specific clinical practices in a clinical setting that should be associated with different practice and environmental characteristics. Because sustainability requires a team effort, assessment of sustainability should not be different based on the training and/or job title of the people providing the assessment. Here with the small pilot sample we can see that the CSAT is able to detect perceived sustainability differences among different types of clinical settings and environments. This instrument is of potential value to researchers in implementation science, as well as evaluators of clinical practices and outcomes. Our measure aligns with existing frameworks conceptualizing organizational capacity, and emphasizes specific concepts known to be relevant to clinical settings [ 8 , 37 ]. For example, workflow is known to have a significant impact on implementation of new interventions in clinical settings[ 38 ]. Similarly, we would hypothesize that new interventions that are poorly integrated into existing clinical workflow will be less likely to be sustained. Other existing instruments exist that assess organizational context and have been well-validated and widely used: Organizational Readiness for Change Assessment (ORCA), Alberta Context Tool, and the Implementation Leadership Scale are frequently used tools [ 39 – 41 ]. The CSAT adds to this toolbox in important ways. Unlike the ORCA, which focuses on initial implementation, the CSAT specifically focuses on sustainability but can be used in all phases of implementation. Indeed, planning for sustainability in the early implementation phases improves likelihood of long-term sustainment [ 4 ]. In addition, unlike the Alberta Context Tool and the Implementation Leadership Scale, which assess general aspects of the organization, the CSAT focuses on sustaining a specific intervention rather than a general measurement of context. This allows users to focus their sustainability assessment on specific interventions. The CSAT’s ability to identify concrete interventions and activities will allow researchers to understand aspects of capacity that are essential to sustaining specific interventions, to recognize patterns across different types of clinical settings attempting to sustain the same intervention, and to describe the impact of organizational capacity on different kinds of interventions [ 42 ]. Our instrument also pairs well with the goals of health systems and quality improvement research. It aims to assist with sustainment of an identified clinical practice, which would support overall quality care. Further, this applies across the context of specific practice improvement, a traditional focus of quality improvement work, while also focusing on broader contextual and process factors that are highlighted in implementation science work. We intended this tool not only to be reliable and valid, but also to be brief and easy to use in busy clinical settings, making the CSAT of equal value to clinicians. To allow equal and open access, the CSAT and the companion assessment for public health settings, the PSAT, are both available at https://sustaintool.org . On this website, clinicians may fill out an online version of the tool and receive a report summarizing their CSAT score and recommendations for improving organizational capacity for sustaining clinical interventions. This simple structure, that may be used by one individual or a group, facilitates training and implementation planning. The complete CSAT instrument, along with instructions, is also available in Additional file 1. Future CSAT research and evaluation work is planned to enhance our knowledge of how best to measure capacity for sustainability in clinical settings, to determine how it operates within broader implementation studies, and to assess its predictive validity for implementation outcomes in healthcare systems and clinical settings. Collectively, this work will address the major limitation of this study, which is the relatively small convenience sample. Conclusion The full benefit of evidence-based interventions will continue to go unrealized without sustained delivery of interventions over time. A critical precursor to sustaining intervention is having adequate organizational capacity. To date, researchers and practitioners have lacked adequate measures to assess organizational capacity for sustainability. The CSAT is a valid, reliable, and easy to use tool that may be used to assess capacity, support sustainability research, and promote sustainable healthcare service delivery. List of Abbreviations PSAT: Program Sustainability Assessment Tool NHS: National Health Service CSAT: Clinical Sustainability Assessment Tool CFI: comparative fit index RMSEA: root mean square error of approximation SRMR: standardized root mean square residual CFA: Confirmatory factor analysis ORCA: Organizational Readiness to Change Assessment PAPERS: Psychometric and Pragmatic Evidence Rating Scale Declarations Ethics Approval: This work was reviewed and declared exempt by the Washington University IRB #201706163. Consent for Publication: Not applicable. Availability of Data: The datasets generated and/or analyzed during the current study are not publicly available due their containing information which could compromise confidentiality, but are available from the corresponding author on reasonable request. Competing Interests: The authors declare that they have no competing interests. Funding: This work was funded by the Center for Dissemination and Implementation at Washington University in St. Louis and by the Center for Public Health Systems Science at Washington University in St. Louis. Research reported in this publication was supported by National Cancer Institute of the National Institutes of Health under award number P50CA244431 and the Washington University Institute of Clinical and Translational Sciences grant UL1TR002345 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The content is solely the responsibility of the authors and does not necessarily represent the official view of the NIH. Authors Contributions : SM participated in study development, participant recruitment, data management, and led the writing of the manuscript. KP participated in participant recruitment, data management, and writing of the manuscript. RH participated in pilot study management and writing of the manuscript. JL participated in pilot study development, recruitment, and writing of the manuscript. VM participated in data analysis and writing of the manuscript. CWB participated in writing and formatting of the manuscript. DL participated in pilot study development, data analysis, and preparation of the manuscript. Acknowledgements : We would like to acknowledge the Center for Dissemination and Implementation at Washington University in St. Louis as well as the WUNDIR network (Washington University Network for Dissemination & Implementation Research) for their assistance throughout the development of this study. References Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117-23. doi:10.1097/01.PHH.0000311888.06252.bb. Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB et al. Public health program capacity for sustainability: a new framework. Implement Sci. 2013;8:15. doi:10.1186/1748-5908-8-15. Scheirer M. Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability. American Journal of Evaluation. 2005;26(3):28p. doi:10.1177/1098214005278752. Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. doi:10.1186/1748-5908-7-17. Goodman RM, Steckler A. A model for the institutionalization of health promotion programs. Family & Community Health. 1989;11(4):63-78. Demski DB, Arabie K. Employee Competency Development and Engagement: - The surprising impact on sustainability, HSE performance and financial results. SPE International Conference on Health, Safety, and Environment in Oil and Gas Exploration and Production; 2008/1/1/; Nice, France. SPE: Society of Petroleum Engineers; 2008. p. 16. Campbell S, Pieters K, Mullen KA, Reece R, Reid RD. Examining sustainability in a hospital setting: case of smoking cessation. Implement Sci. 2011;6:108. doi:10.1186/1748-5908-6-108. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi:10.1186/1748-5908-8-117. Lennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci. 2018;13(1):27. doi:10.1186/s13012-017-0707-4. Braithwaite J, Ludlow K, Testa L, Herkes J, Augustsson H, Lamprell G et al. Built to last? The sustainability of healthcare system improvements, programmes and interventions: a systematic integrative review. BMJ Open. 2020;10(6):e036453. doi:10.1136/bmjopen-2019-036453. Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88. doi:10.1186/s13012-015-0274-5. Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184. doi:10.5888/pcd11.130184. Luke D, Malone, S., Prewitt, K., Hackett, R., Lin, J. , editor. The clinical sustainability assessment tool (CSAT): Assessing sustainability in clinical medicine settings. 11th Annual Conference on the Science of Dissemination and Implementation in Health; December 2018; Washington, D.C. Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. 2009;4:83. doi:10.1186/1748-5908-4-83. Chou WC, Tinetti ME, King MB, Irwin K, Fortinsky RH. Perceptions of physicians on the barriers and facilitators to integrating fall risk evaluation and management into practice. J Gen Intern Med. 2006;21(2):117-22. doi:10.1111/j.1525-1497.2005.00298.x. Hanbury A, Farley K, Thompson C, Wilson P, Chambers D. Challenges in identifying barriers to adoption in a theory-based implementation study: lessons for future implementation studies. BMC Health Serv Res. 2012;12:422. doi:10.1186/1472-6963-12-422. Hou SI, Roberson K. A systematic review on US-based community health navigator (CHN) interventions for cancer screening promotion--comparing community- versus clinic-based navigator models. J Cancer Educ. 2015;30(1):173-86. doi:10.1007/s13187-014-0723-x. Maher L, Gustafson D, Evans A. Sustainability Model and Guide. National Health Service; 2010. Silver SA, McQuillan R, Harel Z, Weizman AV, Thomas A, Nesrallah G et al. How to Sustain Change and Support Continuous Quality Improvement. Clin J Am Soc Nephrol. 2016;11(5):916-24. doi:10.2215/CJN.11501015. Beer M, Nohria N. Cracking the code of change. Harv Bus Rev. 2000;78(3):133-41, 216. Ham C, Kipping R, McLeod H. Redesigning work processes in health care: lessons from the National Health Service. Milbank Q. 2003;81(3):415-39. doi:10.1111/1468-0009.t01-3-00062. Geoffrion TR, Lynch IP, Hsu W, Phelps E, Minhajuddin A, Tsai E et al. An Implementation Science Approach to Handoff Redesign in a Cardiac Surgery Intensive Care Unit. Ann Thorac Surg. 2019. doi:10.1016/j.athoracsur.2019.09.047. Malone S, Prewitt K, Lin J, Luke D, editors. Using Concept Mapping for Measure Development: Examples from Sustainability. Conference on the Science of Dissemination and Implementation in Health; 2018; Washington, DC. Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health program capacity for sustainability: a new framework. Implementation Science. 2013;8(1):9p. doi:10.1186/1748-5908-8-15. Rosas SR, Ridings JW. The use of concept mapping in measurement development and evaluation: Application and future directions. Eval Program Plann. 2017;60:265-76. doi:10.1016/j.evalprogplan.2016.08.016. Qualtrics. Qualtrics Provo, Utah, USA2005. Brooke J. SUS-A quick and dirty usability scale. Usability evaluation in industry. 1996;189(194):4-7. Rosseel Y. lavaan: An R Package for Structural Equation Modeling. 2012. 2012;48(2):36. doi:10.18637/jss.v048.i02. Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment. 1995;7(3):286-99. doi:10.1037/1040-3590.7.3.286. Luke D, Ribisl KM, Walton MA, Davidson WS. Assessing the Diversity of Personal Beliefs About Addiction: Development of the Addiction Belief Inventory. Substance Use & Misuse. 2002;37(1):32p. doi:10.1081/ja-120001498. Hu L, Bentler P. Evaluating model fit. In: Hoyle RH, editor. Structural Equation Modeling: concepts, issues, and applications. Thousand Oaks, CA: Sage Publications; 1995. p. 76-99. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies. 2011;48(6):661-71. doi: https://doi.org/10.1016/j.ijnurstu.2011.01.016 . Hu L, PM B. Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. 1998;3:424-53. doi:10.1037/1082-989X.3.4.424. Kenny D, McCoach DB. Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling. Structural Equation Modeling. 2003;10(3):19p. doi:10.1207/s15328007sem1003_1. Schmitt N. Uses and abuses of coefficient alpha. Psychological Assessment. 1996;8(4):350-3. doi:10.1037/1040-3590.8.4.350. Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2019. doi:10.1093/tbm/ibz164. Meyer AM, Davis M, Mays GP. Defining organizational capacity for public health services and systems research. J Public Health Manag Pract. 2012;18(6):535-44. doi:10.1097/PHH.0b013e31825ce928. Cain C, Haque S. Organizational Workflow and Its Impact on Work Quality. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Advances in Patient Safety. Rockville (MD)2008. Aarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45. doi:10.1186/1748-5908-9-45. Estabrooks CA, Squires JE, Cummings GG, Birdsell JM, Norton PG. Development and assessment of the Alberta Context Tool. BMC Health Serv Res. 2009;9:234. doi:10.1186/1472-6963-9-234. Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4:38. doi:10.1186/1748-5908-4-38. Flaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Unpacking prevention capacity: an intersection of research-to-practice models and community-centered models. Am J Community Psychol. 2008;41(3-4):182-96. doi:10.1007/s10464-008-9162-3. Tables Table 1 CSAT pilot participant demographic characteristics. Characteristic N % Profession Nurse 18 16 Pharmacist 37 33 Physician 29 26 Admin/Support 7 6 Other 21 19 Position/Role Bedside Provider 44 40 Unit Level Management 7 6 System Leadership 6 5 Program Leader 27 24 Other 27 24 Environment Academic Medical Center 67 60 Private Practice 6 5 Community hospital 21 19 Community Health Center 6 5 Other 12 11 Setting Inpatient 56 55 Outpatient 26 26 Both 19 19 Patient Pediatric 54 53 Adult 47 47 Note: Total N = 120; Frequencies add up to less than 120 because of missing responses. Table 2 Confirmatory factor analysis (CFA) results of baseline, pilot, and final Clinical Sustainability Assessment Tool instruments. Phase Subscales Items Chi/df CFI RMSEA SRMR AIC Baseline 1 49 2.77 0.47 0.121 0.119 16039 Pilot 7 49 2.09 0.68 0.095 0.094 15189 Final 7 35 1.84 0.81 0.084 0.075 10453 Note: Total N = 120; CFA model fit with robust maximum-likelihood. Table 3 Subscale reliabilities (internal consistency) for the Clinical Sustainability Assessment Tool subscales. Subscale Cronbach’s α Engaged Staff & Leadership 0.85 Engaged Stakeholders 0.82 Organizational Readiness 0.87 Workflow Integration 0.89 Implementation & Training 0.90 Monitoring & Evaluation 0.94 Outcomes & Effectiveness 0.90 Note: Each subscale is made up of 5 items. Table 4 CSAT subscale and total score descriptive statistics. (N = 120) Subscale Mean SD Low High Engaged Staff & Leadership 5.50 1.03 1.60 7.00 Engaged Stakeholders 5.02 1.18 1.20 7.00 Organizational Readiness 4.97 1.23 2.40 7.00 Workflow Integration 5.49 1.13 1.60 7.00 Implementation & Training 5.14 1.25 2.00 7.00 Monitoring & Evaluation 5.12 1.41 1.00 7.00 Outcomes & Effectiveness 5.95 1.01 1.60 7.00 Total Score 5.29 0.90 2.31 6.86 Supplementary Files Additionalfiletworeportingchecklist.docx Additionalfile1.pdf Cite Share Download PDF Status: Published Journal Publication published 17 Jul, 2021 Read the published version in Implementation Science Communications → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-58012","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Methodology","associatedPublications":[],"authors":[{"id":1454191,"identity":"6ffcb61f-64af-4c41-9d5f-deabd0aba98d","order_by":0,"name":"Sara Malone","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYBADGSBmfABi8QGxBDFaeICY2QCkmo0ULWwSRGmRbz/78ANjjh0Pv3TzsWretro6Ngbmg7d58GgxOJNuLMG4LZlHcs6xtNu8bYeBtrAlW+PVwpDGANTCzGNwI8fsdm7bAaAWHjNpfFrk+58x/2DcVg/Ukv+tOLetDqiF/xteLQw30tiAthwG2cLGnNvGDLKFDa8WgxvP2CwStx3nkZyRZiz959xhyTZmNmPLOXgdlsZ84+O2ajl+ieSHH2eU1fHzszc/vPEGn8NAIAGFx0xI+SgYBaNgFIwCggAAYTI9oPDig9sAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-0361-8733","institution":"Washington University in Saint Louis","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Sara","middleName":"","lastName":"Malone","suffix":""},{"id":1454192,"identity":"35401a47-8ace-4c04-b73f-d4b186efd602","order_by":1,"name":"Kim Prewitt","email":"","orcid":"","institution":"Washington University in Saint Louis","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kim","middleName":"","lastName":"Prewitt","suffix":""},{"id":1454193,"identity":"8bf024d5-b8b6-45e8-aa11-3c5d9c4599e3","order_by":2,"name":"Rachel Hackett","email":"","orcid":"","institution":"Washington University in Saint Louis","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Rachel","middleName":"","lastName":"Hackett","suffix":""},{"id":1454194,"identity":"26a8daa8-36fb-4b71-8acf-8155abf513f5","order_by":3,"name":"John C. Lin","email":"","orcid":"","institution":"Washington University in Saint Louis School of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"John","middleName":"C.","lastName":"Lin","suffix":""},{"id":1454195,"identity":"f4b9dad2-1508-4117-8467-79361c3eb444","order_by":4,"name":"Virginia McKay","email":"","orcid":"","institution":"Washington University in Saint Louis","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Virginia","middleName":"","lastName":"McKay","suffix":""},{"id":1454196,"identity":"745e7c36-ed78-4e69-9126-2cf2d7662719","order_by":5,"name":"Callie Walsh-Bailey","email":"","orcid":"","institution":"Washington University in Saint Louis","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Callie","middleName":"","lastName":"Walsh-Bailey","suffix":""},{"id":1454197,"identity":"dd9bd232-7c21-41c3-89d6-f67b3e7cfd12","order_by":6,"name":"Douglas Luke","email":"","orcid":"","institution":"Washington University in Saint Louis","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Douglas","middleName":"","lastName":"Luke","suffix":""}],"badges":[],"createdAt":"2020-08-12 11:12:01","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-58012/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-58012/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s43058-021-00181-2","type":"published","date":"2021-07-17T20:57:47+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":1964071,"identity":"86247c3a-b310-43da-91fe-161eedca0b99","added_by":"auto","created_at":"2020-08-17 22:48:38","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":50642,"visible":true,"origin":"","legend":"Seven conceptual domains of clinical sustainability, suggested by the concept mapping results.","description":"","filename":"Fig1.JPG","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/Fig1.JPG"},{"id":1964072,"identity":"3d63f812-148e-4d20-a16b-26741f44057a","added_by":"auto","created_at":"2020-08-17 22:48:38","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":208364,"visible":true,"origin":"","legend":"Variability of CSAT total scores.","description":"","filename":"Fig2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/Fig2.jpg"},{"id":1964073,"identity":"f88a62ca-efc8-4077-ab2d-99f1c35c4944","added_by":"auto","created_at":"2020-08-17 22:48:39","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":280541,"visible":true,"origin":"","legend":"Profile plots for CSAT subscales by characteristics of the setting and respondents. ","description":"","filename":"Fig3.JPG","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/Fig3.JPG"},{"id":13580497,"identity":"763844eb-1274-4d91-8699-773cd96774fd","added_by":"auto","created_at":"2021-09-17 04:23:49","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":528248,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/9ce137b3-8f41-4ddf-93c0-5dc78a411fe2.pdf"},{"id":1964075,"identity":"e4bf4ffd-eea7-40b1-acfb-829b1ce314a6","added_by":"auto","created_at":"2020-08-17 22:48:39","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":18492,"visible":true,"origin":"","legend":"","description":"","filename":"Additionalfiletworeportingchecklist.docx","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/Additionalfiletworeportingchecklist.docx"},{"id":1964076,"identity":"ed1bef00-646b-4864-a9c9-47be16b80528","added_by":"auto","created_at":"2020-08-17 22:48:39","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":287324,"visible":true,"origin":"","legend":"","description":"","filename":"Additionalfile1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-58012/v1/Additionalfile1.pdf"}],"financialInterests":"","formattedTitle":"The Clinical Sustainability Assessment Tool: Measuring Organizational Capacity to Promote Sustainability in Healthcare","fulltext":[{"header":"Contributions To The Literature","content":" \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eSustainability of evidence-based practices is important to deliver intended health outcomes to individuals and populations. However, there has been relatively less focus on sustainability theory and methods in implementation science.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWe introduce the Clinical Sustainability Assessment Tool (CSAT), a new instrument for measuring sustainability capacity in clinical and healthcare settings.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe CSAT is designed to be easy to use\u0026mdash;it is short, requires little training, and can be used by researchers, evaluators, and frontline clinical staff.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe CSAT has excellent reliability, and preliminary validation data suggest that the CSAT is able to distinguish among different types of clinical settings.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e "},{"header":"Background","content":" \u003cp\u003eSustainability is the degree to which an evidence-based program or intervention can deliver its intended benefits over an extended period of time [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Programs need to be sustained and clinical interventions continuously delivered to achieve their desired public health and clinical outcomes and impacts [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Sustainability can be challenging, and in fact many initially successful programs fail to remain in place over time. Evidence suggests that interventions that are not sustained could result in decreased care quality and worsened patient outcomes and quality of life [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Programs that are not kept up after an initial investment can waste money and non-monetary resources [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Having a better understanding of what factors affect sustainability can inform strategies to improve the likelihood that interventions will successfully continue after implementation [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eStudies within implementation science often focus on initial program or intervention adoption and implementation, placing less emphasis on sustainability [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. In two recent reviews of sustainability, the authors found relatively few studies focusing on sustainability, most of which were of limited quality and methodological rigor [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Research on sustainability is limited in part because data collection for grant funded studies is typically not carried out past the funding end date and is beyond the scope of many public health and health services studies [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Consequently, there have been recent calls for increased attention to sustainability research, theory, and methods [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003ePartly to address this critical gap in the field, we developed the Program Sustainability Assessment Tool (PSAT), which was designed to measure capacity for program sustainability [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], particularly for public health evidence-based programs. The PSAT has been widely used around the world for local, regional, and national programming [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Although it was developed for public health programs, it has also been used by social service, clinical care, and education programs to assess sustainability of a variety of programs focused on transitional services, obesity prevention, behavioral health, health systems strengthening, and tobacco control.\u003c/p\u003e \u003cp\u003eThe PSAT has been used to examine sustainability within clinical and healthcare settings, but early reports from users highlighted a number of limitations. Most essentially, clinical settings differ from public health settings in a number of ways which change how sustainability should be conceptualized and measured. For example, funding in clinical settings tends to be tied primarily to public and private insurance funding, rather than public grants and/or private donations for public health initiatives [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Thus, the sustainability of a practice is often tied to its ability to be reimbursed by payors [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Furthermore, community outreach, a key driver of sustainability in public health settings, is conducted less in clinical settings as services are mainly delivered to identified in-services patients [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Finally, although data are somewhat thin, current evidence suggests that sustaining evidence-based practices are challenging in clinical settings (as they are in public health settings). For example, a review of practices in the National Health Service (NHS) in the UK found that 33% of quality improvement projects are not sustained for one year after initial implementation [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eRecently, a changing context in clinical healthcare has also become relevant to the implementation and integration of evidence-based practices. Clinical healthcare has widely adopted electronic medical records, dramatically changing how practitioners chart, build workflows, and communicate with other team members. This change has altered the structure in different health settings and how professionals orient themselves to medical decisions as well as other team members. New evidence-based clinical practices need to be integrated into this complex information and work system to be able to sustain them over time. Furthermore, sustainability outcomes in clinical settings play out over shorter time periods than for public health programming that is focused more on population health outcomes. Finally, the rapid cycle improvement that has historically been promoted in clinical settings and quality improvement initiatives has failed to focus on or understand the role of long term sustainability and organizational capacity within this realm [\u003cspan additionalcitationids=\"CR21\" citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTo address need for additional research focused on sustainability and to support clinicians interested in planning for sustainable implementation of innovations, we present a new instrument for measuring sustainability capacity in clinical and healthcare settings: the \u003cem\u003eClinical Sustainability Assessment Tool\u003c/em\u003e (CSAT). In this paper, we describe the conceptual and measurement development of the CSAT, and present pilot data demonstrating its usability, reliability, and preliminary validation with a sample of healthcare practitioners and researchers. We also outline next steps for the CSAT, including broader dissemination and validation of the tool.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eThis is a measurement development study of a tool to assess capacity for sustainability in clinical healthcare settings. The study included two major components: 1) conceptual development guided by a literature review and expert opinion (via concept mapping); and 2) a pilot study of an initial version of the CSAT to assess its reliability and produce a final version of the instrument ready for dissemination. The CSAT was designed to be easy to use by researchers, evaluators, and practitioners, and applicable to a wide variety of healthcare settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eConceptual Development\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAfter completing a review of existing literature and measures, a concept mapping approach was utilized to develop the CSAT [23]. Concept mapping is a mixed methods approach to reveal the structure of a complex conceptual domain, and is a useful tool for measurement development [24, 25].\u003c/p\u003e\n\u003cp\u003eWe used concept mapping to conceptualize sustainability of practices in clinical settings, which resulted in the production and refinement of the domain structure for the measure. More than 60 participants from multiple professions and different healthcare settings participated in an initial brainstorming activity.\u003c/p\u003e\n\u003cp\u003eRespondents included:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eResearchers in areas of clinical care, sustainability, and dissemination and implementation science (n=10)\u003c/li\u003e\n\u003cli\u003eClinical care administrators (n=9)\u003c/li\u003e\n\u003cli\u003eClinical care service providers (e.g., physicians, nurses, rehabilitation specialists; n=43)\u003c/li\u003e\n\u003cli\u003eThese participants represented various specialties and clinical environments (i.e., inpatient, outpatient, adult, and pediatric settings)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese participants responded to the prompt, \u0026ldquo;In order for clinical practices to continue over time, they need\u0026hellip;\u0026rdquo; and generated a list of 230 statements. Researchers reviewed the list of statements and removed any duplicate or nonsensical items and edited for spelling and grammar. The final list included 97 statements. In a subsequent activity, participants then grouped similar the statements together and rated each statement on dimensions of feasibility and importance.\u003c/p\u003e\n\u003cp\u003eFigure 1 shows the concept mapping results after sorting the 97 statements. Seven domains emerged based on the underlying items (points in the figure) and literature on sustainability. The resulting conceptual domains were: \u003cem\u003eEngaged Staff \u0026amp; Leadership, Engaged Stakeholders, Monitoring \u0026amp; Evaluation, Organizational Context \u0026amp; Capacity, Workflow Integration, Planning \u0026amp; Implementation, and Outcomes \u0026amp; Effectiveness\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eCSAT Instrument Development \u0026amp; Pilot Testing\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCSAT initial instrument development\u003c/em\u003e. For each conceptual domain, items were selected from the concept mapping responses that best reflected the meaning of its domain. A total of seven statements were selected for each of the domains for a total of 49 indicators. Each statement was edited so that CSAT respondents would be asked to assess the extent to which each existed in their clinic or setting on a Likert-type scale from 1 (no extent) to 7 (great extent). An example item was, \u0026ldquo;The practice has adequate staff to achieve its goals.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eParticipants \u0026amp; Recruitment. \u003c/em\u003eThe Center for Public Health Systems Science research team invited individuals for a pilot test of the instrument (Table 1). This convenience sample was made up of participants selected from different clinical work environments as well as from different healthcare professions. Recruitment efforts used a snowball sampling approach, and included identifying and contacting stakeholders that could potentially benefit from using the tool, as well as promoting the CSAT at local and national dissemination and implementation conferences. Respondents had the option to forward the link to peers or nominate individuals to complete the CSAT. To incentivize participation, all respondents were offered an optional tailored sustainability results report and the opportunity to enter a drawing for one of five $50 gift cards. The final sample size was 120 participants (Table 1).\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eData Collection \u0026amp; Instrument\u003c/em\u003e. Data collection was administered through an electronic survey on Qualtrics [26]. The assessment consisted of 64 questions total, including the 49 on the pilot measure, demographic questions, and tool usability questions. Demographic items included questions about the individual, their profession, and their practice setting. Usability questions were adapted from the System Usability Scale [27] and were used to understand participant reaction to completing the CSAT and give insight into any concerns with functionality of the tool.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eData Management \u0026amp; Analyses.\u003c/em\u003e Data were downloaded and managed in R. A variety of descriptive, visualization, and psychometric analyses were conducted to explore the item, domain, and instrument characteristics of the CSAT, with particular emphasis on reliability. The \u003cem\u003eclassical test theory\u003c/em\u003e package (CTT, Version 2.3.3) in R was used to calculate measures of internal consistency, particularly Cronbach\u0026rsquo;s alpha. Confirmatory factor analyses were done using the \u003cem\u003elavaan\u003c/em\u003e package (Version 0.6-5) [28]. Confirmatory factor analysis (CFA) is a powerful and appropriate tool for testing a hypothesized subscale structure in a measurement instrument [29, 30]. Initially CFA was applied to the pilot CSAT (with 7 items per subscale) to identify poorly performing items and test our hypothesized sustainability domain structure. Poor items were those that had low variability and/or poor fit with the intended subscale. Once the final structure was determined, we reran the CFA on the smaller, final version of the CSAT (5 items per subscale). For both tests, we used three measures of model fit to assess model adequacy: the comparative fit index (CFI), root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) [31]. Demographics were further assessed to understanding potential differences in responses and understanding how these variables might influence sustainability capacity. The Guidelines for Reporting Reliability and Agreement Studies were utilized for reporting [32].\u0026nbsp;\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cstrong\u003e\u003cem\u003eInstrument improvement and domain structure\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eItem and confirmatory factor analyses were used to determine the final structure of the CSAT. Table 2 shows the improved psychometrics during the instrument development process. The baseline model is used as a comparison for the pilot and final models. The pilot model includes all initial 49 items contained in seven subscales. After psychometric analyses, 14 items were dropped from the pilot version of the framework and tool. Items were dropped if they had lower loadings in the latent factors, had poor variance (\u003cem\u003ei.e., \u003c/em\u003erestricted range), or had excessive missing data. The final CSAT is comprised of 35 items organized within 7 subscale domains. Each domain has five items. This simple and balanced structure facilitates training and scoring with practices and groups.\u003c/p\u003e\n\u003cp\u003eThe domain structure and final items of the CSAT can be seen in Additional file 1. The first domain, \u003cem\u003eEngaged Staff \u0026amp; Leadership, \u003c/em\u003eincludes items assessing the extent to which the clinical practice has the support of internal frontline staff and management within the organization. \u003cem\u003eEngaged Stakeholders\u003c/em\u003e assesses the extent to which the practice has support among external stakeholders. \u003cem\u003eOrganizational Readiness\u003c/em\u003e measures whether the organization has the internal supports and resources needed to effectively manage the practice. \u003cem\u003eWorkflow Integration\u003c/em\u003e refers to whether the practice has been designed to fit into existing workplace processes, policies, and technologies (e.g., EMR systems). \u003cem\u003eImplementation \u0026amp; Training\u003c/em\u003e reflects whether the organization promotes processes and learning that appropriately guide the direction, goals, and strategies of the clinical practice. \u003cem\u003eMonitoring \u0026amp; Evaluation\u003c/em\u003e assesses the extent to which the organization monitors the clinical practice and uses data to inform quality improvement. Finally, \u003cem\u003eOutcomes \u0026amp; Effectiveness \u003c/em\u003erefers to whether and how the organization measures practice outcomes and impacts. Each subscale is scored separately, and an overall CSAT score can be obtained, ranging from 1 to 7. Higher scores indicate greater organizational capacity for clinical sustainability.\u003c/p\u003e\n\u003cp\u003eTable 2 also shows the relatively good fit of the 7 factor CFA model to the data\u0026mdash;that is, the 35 item CSAT does a credible job of measuring seven important clinical sustainability domains that were identified in the literature and in the concept mapping phase of the study. Although the CFI could be larger, the RMSEA of 0.084 is just above the target for an acceptable fit (0.08), and the SRMR of 0.075 is smaller than 0.08, indicating a good fit [33, 34]. The most important pattern in the CFA results presented in Table 2 is the improvement in model fit as we move from a single factor model, to a seven subscale model with all items, and finally the seven subscale model with the reduced number of items. The overall results suggest that the CSAT measures distinctive aspects of clinical sustainability with a relatively modest number of items that promotes ease of use.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSubscale reliability\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTable 3 presents the subscale reliabilities (internal consistency) for the CSAT. The average internal consistency of the seven subscales is 0.88, and ranges from 0.82 to 0.94. These indicate excellent scale reliability, especially given the small size of each subscale (5 items) [35]. (Internal consistency always goes up with more items, so the most desirable goal is the fewest items that still maintains high reliability [33].) Furthermore, the item loadings show consistently high intercorrelations with their respective subscales, indicating good fit of individual items with overall subscale scores (full results available from authors).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003ePreliminary CSAT results and validation\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSubdomain scores are the simple means of the five items in each domain. The total CSAT score is then the mean of the seven subdomain scores. Table 4 presents descriptive statistics for the seven subscales and the total scores. \u003cem\u003eOrganizational Readiness\u003c/em\u003e has the lowest average score, while \u003cem\u003eOutcomes \u0026amp; Effectiveness\u003c/em\u003e is rated the highest. The standard deviations, ranges, and the density plot shown in Figure 2 show that there is good variability of scores, and no major problems with restricted range. The CSAT total scores range from 2.3 to 6.9. The standard deviation for the total scores is lower than for the subdomain scores, which is expected given that the total score is the mean of the seven subdomain scores.\u003c/p\u003e\n\u003cp\u003eIn addition to the CSAT scores, basic characteristics of the clinical setting were collected (i.e., patient, setting, and environment type) as well as two characteristics of the respondent (i.e., job position, and training profession). As part of a set of preliminary validational analyses, CSAT total and subscale scores were examined across these five setting and respondent characteristics. In terms of setting characteristics, total CSAT scores varied significantly by setting type (\u003cem\u003eF\u003c/em\u003e = 3.16, \u003cem\u003ep\u003c/em\u003e = .047) and environment (\u003cem\u003eF\u003c/em\u003e = 2.93, \u003cem\u003ep\u003c/em\u003e = .038), but not by patient type (\u003cem\u003eF\u003c/em\u003e = 1.09, \u003cem\u003ep\u003c/em\u003e = .299). CSAT scores did not vary by respondent\u0026rsquo;s profession (\u003cem\u003eF\u003c/em\u003e = 0.93, \u003cem\u003ep\u003c/em\u003e = .449) or job position (\u003cem\u003eF\u003c/em\u003e = 1.69, \u003cem\u003ep\u003c/em\u003e = .175).\u003c/p\u003e\n\u003cp\u003eFigure 3 shows the CSAT subscale score profile plots for the three setting (top row) and two respondent (bottom row) characteristics. Domain scores were very similar based on patient age category but showed significant differences based on outpatient vs. inpatient clinical setting and academic vs. community vs. private practice environment. Outpatient settings report consistently lower sustainability scores across all seven domains than inpatient settings. Academic medical centers and community hospitals were assessed as having higher sustainability capacities than community health centers and private practices. Type of environment is a little more complicated, but it appears that community hospitals and academic medical centers report higher sustainability than community health centers and private practice settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eUsability testing\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe also collected data regarding ease of use and asked participants to report on their experience using the CSAT. On average, it took participants just under 20 minutes to complete the longer initial 49 item version of the CSAT and just under 10 minutes to complete the final 35 item tool. Participants also rated the experience of completing the CSAT in a positive manner: 85% of participants rated the tool as easy to use; 75% felt very confident about their ability to use the tool; 90% thought that most other people would be able to learn quickly how to use the tool. Importantly, only 35% thought that they would need external support to use the tool effectively.\u003c/p\u003e\n\u003cp\u003eIn addition to the usability data from our pilot participants, we assessed the final 35 item version of the CSAT using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS)[36]. The CSAT rates good or excellent in all five of the PAPERS pragmatic categories: brevity, cost, ease of training, ease of interpretation, and language. The CSAT rates good in brevity with less than 40 items and language at less than a 10\u003csup\u003eth\u003c/sup\u003e grade reading level. It rates excellent in cost, ease of training, and ease of interpretation being free, requiring no training to use, and automated calculations of scores.\u003c/p\u003e"},{"header":"Discussion","content":" \u003cp\u003eWe introduce here the Clinical Sustainability Assessment Tool (CSAT) as a reliable instrument to assess organizational capacity for sustainability in clinical settings. The final CSAT includes seven distinct subscale domains with high internal consistency within each domain. The CSAT also demonstrates good variability with respect to clinical setting and practice, but maintained consistency across respondents and patients.\u003c/p\u003e \u003cp\u003eMore specifically, these instrument development results provide important initial validational support for the CSAT. The CSAT is designed to assess sustainability of specific clinical practices in a clinical setting that should be associated with different practice and environmental characteristics. Because sustainability requires a team effort, assessment of sustainability should not be different based on the training and/or job title of the people providing the assessment. Here with the small pilot sample we can see that the CSAT is able to detect perceived sustainability differences among different types of clinical settings and environments.\u003c/p\u003e \u003cp\u003eThis instrument is of potential value to researchers in implementation science, as well as evaluators of clinical practices and outcomes. Our measure aligns with existing frameworks conceptualizing organizational capacity, and emphasizes specific concepts known to be relevant to clinical settings [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. For example, workflow is known to have a significant impact on implementation of new interventions in clinical settings[\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. Similarly, we would hypothesize that new interventions that are poorly integrated into existing clinical workflow will be less likely to be sustained.\u003c/p\u003e \u003cp\u003eOther existing instruments exist that assess organizational context and have been well-validated and widely used: Organizational Readiness for Change Assessment (ORCA), Alberta Context Tool, and the Implementation Leadership Scale are frequently used tools [\u003cspan additionalcitationids=\"CR40\" citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e]. The CSAT adds to this toolbox in important ways. Unlike the ORCA, which focuses on initial implementation, the CSAT specifically focuses on sustainability but can be used in all phases of implementation. Indeed, planning for sustainability in the early implementation phases improves likelihood of long-term sustainment [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. In addition, unlike the Alberta Context Tool and the Implementation Leadership Scale, which assess general aspects of the organization, the CSAT focuses on sustaining a specific intervention rather than a general measurement of context. This allows users to focus their sustainability assessment on specific interventions. The CSAT\u0026rsquo;s ability to identify concrete interventions and activities will allow researchers to understand aspects of capacity that are essential to sustaining specific interventions, to recognize patterns across different types of clinical settings attempting to sustain the same intervention, and to describe the impact of organizational capacity on different kinds of interventions [\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eOur instrument also pairs well with the goals of health systems and quality improvement research. It aims to assist with sustainment of an identified clinical practice, which would support overall quality care. Further, this applies across the context of specific practice improvement, a traditional focus of quality improvement work, while also focusing on broader contextual and process factors that are highlighted in implementation science work.\u003c/p\u003e \u003cp\u003eWe intended this tool not only to be reliable and valid, but also to be brief and easy to use in busy clinical settings, making the CSAT of equal value to clinicians. To allow equal and open access, the CSAT and the companion assessment for public health settings, the PSAT, are both available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://sustaintool.org\u003c/span\u003e\u003c/span\u003e. On this website, clinicians may fill out an online version of the tool and receive a report summarizing their CSAT score and recommendations for improving organizational capacity for sustaining clinical interventions. This simple structure, that may be used by one individual or a group, facilitates training and implementation planning. The complete CSAT instrument, along with instructions, is also available in Additional file 1.\u003c/p\u003e \u003cp\u003eFuture CSAT research and evaluation work is planned to enhance our knowledge of how best to measure capacity for sustainability in clinical settings, to determine how it operates within broader implementation studies, and to assess its predictive validity for implementation outcomes in healthcare systems and clinical settings. Collectively, this work will address the major limitation of this study, which is the relatively small convenience sample.\u003c/p\u003e "},{"header":"Conclusion","content":" \u003cp\u003eThe full benefit of evidence-based interventions will continue to go unrealized without sustained delivery of interventions over time. A critical precursor to sustaining intervention is having adequate organizational capacity. To date, researchers and practitioners have lacked adequate measures to assess organizational capacity for sustainability. The CSAT is a valid, reliable, and easy to use tool that may be used to assess capacity, support sustainability research, and promote sustainable healthcare service delivery.\u003c/p\u003e "},{"header":"List of Abbreviations","content":"\u003cp\u003ePSAT: Program Sustainability Assessment Tool\u003c/p\u003e\n\u003cp\u003eNHS: National Health Service\u003c/p\u003e\n\u003cp\u003eCSAT: Clinical Sustainability Assessment Tool\u003c/p\u003e\n\u003cp\u003eCFI: comparative fit index\u003c/p\u003e\n\u003cp\u003eRMSEA: root mean square error of approximation\u003c/p\u003e\n\u003cp\u003eSRMR: standardized root mean square residual\u003c/p\u003e\n\u003cp\u003eCFA: Confirmatory factor analysis\u003c/p\u003e\n\u003cp\u003eORCA: Organizational Readiness to Change Assessment\u003c/p\u003e\n\u003cp\u003ePAPERS: Psychometric and Pragmatic Evidence Rating Scale\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eEthics Approval:\u003c/strong\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis work was reviewed and declared exempt by the Washington University IRB #201706163.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eConsent for Publication:\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eAvailability of Data:\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and/or analyzed during the current study are not publicly available due their containing information which could compromise confidentiality, but are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eCompeting Interests:\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eFunding:\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work was funded by the Center for Dissemination and Implementation at Washington University in St. Louis and by the Center for Public Health Systems Science at Washington University in St. Louis. Research reported in this publication was supported by \u003cem\u003eNational Cancer Institute\u0026nbsp;\u003c/em\u003eof the National Institutes of Health under award number P50CA244431 and the Washington University Institute of Clinical and Translational Sciences grant UL1TR002345 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The content is solely the responsibility of the authors and does not necessarily represent the official view of the NIH.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eAuthors Contributions\u003c/em\u003e:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSM participated in study development, participant recruitment, data management, and led the writing of the manuscript. KP participated in participant recruitment, data management, and writing of the manuscript. RH participated in pilot study management and writing of the manuscript. JL participated in pilot study development, recruitment, and writing of the manuscript. VM participated in data analysis and writing of the manuscript. CWB participated in writing and formatting of the manuscript. DL participated in pilot study development, data analysis, and preparation of the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eAcknowledgements\u003c/em\u003e\u003c/strong\u003e:\u003c/p\u003e\n\u003cp\u003eWe would like to acknowledge the Center for Dissemination and Implementation at Washington University in St. Louis as well as the WUNDIR network (Washington University Network for Dissemination \u0026amp; Implementation Research) for their assistance throughout the development of this study.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eRabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117-23. doi:10.1097/01.PHH.0000311888.06252.bb.\u003c/li\u003e\n\u003cli\u003eSchell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB et al. Public health program capacity for sustainability: a new framework. Implement Sci. 2013;8:15. doi:10.1186/1748-5908-8-15.\u003c/li\u003e\n\u003cli\u003eScheirer M. Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability. American Journal of Evaluation. 2005;26(3):28p. doi:10.1177/1098214005278752.\u003c/li\u003e\n\u003cli\u003eWiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. doi:10.1186/1748-5908-7-17.\u003c/li\u003e\n\u003cli\u003eGoodman RM, Steckler A. A model for the institutionalization of health promotion programs. Family \u0026amp; Community Health. 1989;11(4):63-78.\u003c/li\u003e\n\u003cli\u003eDemski DB, Arabie K. Employee Competency Development and Engagement: - The surprising impact on sustainability, HSE performance and financial results. SPE International Conference on Health, Safety, and Environment in Oil and Gas Exploration and Production; 2008/1/1/; Nice, France. SPE: Society of Petroleum Engineers; 2008. p. 16.\u003c/li\u003e\n\u003cli\u003eCampbell S, Pieters K, Mullen KA, Reece R, Reid RD. Examining sustainability in a hospital setting: case of smoking cessation. Implement Sci. 2011;6:108. doi:10.1186/1748-5908-6-108.\u003c/li\u003e\n\u003cli\u003eChambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi:10.1186/1748-5908-8-117.\u003c/li\u003e\n\u003cli\u003eLennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci. 2018;13(1):27. doi:10.1186/s13012-017-0707-4.\u003c/li\u003e\n\u003cli\u003eBraithwaite J, Ludlow K, Testa L, Herkes J, Augustsson H, Lamprell G et al. Built to last? The sustainability of healthcare system improvements, programmes and interventions: a systematic integrative review. BMJ Open. 2020;10(6):e036453. doi:10.1136/bmjopen-2019-036453.\u003c/li\u003e\n\u003cli\u003eProctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88. doi:10.1186/s13012-015-0274-5.\u003c/li\u003e\n\u003cli\u003eLuke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184. doi:10.5888/pcd11.130184.\u003c/li\u003e\n\u003cli\u003eLuke D, Malone, S., Prewitt, K., Hackett, R., Lin, J. , editor. The clinical sustainability assessment tool (CSAT): Assessing sustainability in clinical medicine settings. 11th Annual Conference on the Science of Dissemination and Implementation in Health; December 2018; Washington, D.C.\u003c/li\u003e\n\u003cli\u003eAarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. 2009;4:83. doi:10.1186/1748-5908-4-83.\u003c/li\u003e\n\u003cli\u003eChou WC, Tinetti ME, King MB, Irwin K, Fortinsky RH. Perceptions of physicians on the barriers and facilitators to integrating fall risk evaluation and management into practice. J Gen Intern Med. 2006;21(2):117-22. doi:10.1111/j.1525-1497.2005.00298.x.\u003c/li\u003e\n\u003cli\u003eHanbury A, Farley K, Thompson C, Wilson P, Chambers D. Challenges in identifying barriers to adoption in a theory-based implementation study: lessons for future implementation studies. BMC Health Serv Res. 2012;12:422. doi:10.1186/1472-6963-12-422.\u003c/li\u003e\n\u003cli\u003eHou SI, Roberson K. A systematic review on US-based community health navigator (CHN) interventions for cancer screening promotion--comparing community- versus clinic-based navigator models. J Cancer Educ. 2015;30(1):173-86. doi:10.1007/s13187-014-0723-x.\u003c/li\u003e\n\u003cli\u003eMaher L, Gustafson D, Evans A. Sustainability Model and Guide. National Health Service; 2010.\u003c/li\u003e\n\u003cli\u003eSilver SA, McQuillan R, Harel Z, Weizman AV, Thomas A, Nesrallah G et al. How to Sustain Change and Support Continuous Quality Improvement. Clin J Am Soc Nephrol. 2016;11(5):916-24. doi:10.2215/CJN.11501015.\u003c/li\u003e\n\u003cli\u003eBeer M, Nohria N. Cracking the code of change. Harv Bus Rev. 2000;78(3):133-41, 216.\u003c/li\u003e\n\u003cli\u003eHam C, Kipping R, McLeod H. Redesigning work processes in health care: lessons from the National Health Service. Milbank Q. 2003;81(3):415-39. doi:10.1111/1468-0009.t01-3-00062.\u003c/li\u003e\n\u003cli\u003eGeoffrion TR, Lynch IP, Hsu W, Phelps E, Minhajuddin A, Tsai E et al. An Implementation Science Approach to Handoff Redesign in a Cardiac Surgery Intensive Care Unit. Ann Thorac Surg. 2019. doi:10.1016/j.athoracsur.2019.09.047.\u003c/li\u003e\n\u003cli\u003eMalone S, Prewitt K, Lin J, Luke D, editors. Using Concept Mapping for Measure Development: Examples from Sustainability. Conference on the Science of Dissemination and Implementation in Health; 2018; Washington, DC.\u003c/li\u003e\n\u003cli\u003eLuke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health program capacity for sustainability: a new framework. Implementation Science. 2013;8(1):9p. doi:10.1186/1748-5908-8-15.\u003c/li\u003e\n\u003cli\u003eRosas SR, Ridings JW. The use of concept mapping in measurement development and evaluation: Application and future directions. Eval Program Plann. 2017;60:265-76. doi:10.1016/j.evalprogplan.2016.08.016.\u003c/li\u003e\n\u003cli\u003eQualtrics. Qualtrics Provo, Utah, USA2005.\u003c/li\u003e\n\u003cli\u003eBrooke J. SUS-A quick and dirty usability scale. Usability evaluation in industry. 1996;189(194):4-7.\u003c/li\u003e\n\u003cli\u003eRosseel Y. lavaan: An R Package for Structural Equation Modeling. 2012. 2012;48(2):36. doi:10.18637/jss.v048.i02.\u003c/li\u003e\n\u003cli\u003eFloyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment. 1995;7(3):286-99. doi:10.1037/1040-3590.7.3.286.\u003c/li\u003e\n\u003cli\u003eLuke D, Ribisl KM, Walton MA, Davidson WS. Assessing the Diversity of Personal Beliefs About Addiction: Development of the Addiction Belief Inventory. Substance Use \u0026amp; Misuse. 2002;37(1):32p. doi:10.1081/ja-120001498.\u003c/li\u003e\n\u003cli\u003eHu L, Bentler P. Evaluating model fit. In: Hoyle RH, editor. Structural Equation Modeling: concepts, issues, and applications. Thousand Oaks, CA: Sage Publications; 1995. p. 76-99.\u003c/li\u003e\n\u003cli\u003eKottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hr\u0026oacute;bjartsson A et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies. 2011;48(6):661-71. doi:\u003ca href=\"https://doi.org/10.1016/j.ijnurstu.2011.01.016\"\u003ehttps://doi.org/10.1016/j.ijnurstu.2011.01.016\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eHu L, PM B. Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. 1998;3:424-53. doi:10.1037/1082-989X.3.4.424.\u003c/li\u003e\n\u003cli\u003eKenny D, McCoach DB. Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling. Structural Equation Modeling. 2003;10(3):19p. doi:10.1207/s15328007sem1003_1.\u003c/li\u003e\n\u003cli\u003eSchmitt N. Uses and abuses of coefficient alpha. Psychological Assessment. 1996;8(4):350-3. doi:10.1037/1040-3590.8.4.350.\u003c/li\u003e\n\u003cli\u003eStanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2019. doi:10.1093/tbm/ibz164.\u003c/li\u003e\n\u003cli\u003eMeyer AM, Davis M, Mays GP. Defining organizational capacity for public health services and systems research. J Public Health Manag Pract. 2012;18(6):535-44. doi:10.1097/PHH.0b013e31825ce928.\u003c/li\u003e\n\u003cli\u003eCain C, Haque S. Organizational Workflow and Its Impact on Work Quality. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Advances in Patient Safety. Rockville (MD)2008.\u003c/li\u003e\n\u003cli\u003eAarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45. doi:10.1186/1748-5908-9-45.\u003c/li\u003e\n\u003cli\u003eEstabrooks CA, Squires JE, Cummings GG, Birdsell JM, Norton PG. Development and assessment of the Alberta Context Tool. BMC Health Serv Res. 2009;9:234. doi:10.1186/1472-6963-9-234.\u003c/li\u003e\n\u003cli\u003eHelfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4:38. doi:10.1186/1748-5908-4-38.\u003c/li\u003e\n\u003cli\u003eFlaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Unpacking prevention capacity: an intersection of research-to-practice models and community-centered models. Am J Community Psychol. 2008;41(3-4):182-96. doi:10.1007/s10464-008-9162-3.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCSAT pilot participant demographic characteristics.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCharacteristic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e%\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eProfession\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNurse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePharmacist\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e33\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhysician\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdmin/Support\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003ePosition/Role\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBedside Provider\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUnit Level Management\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSystem Leadership\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProgram Leader\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eEnvironment\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAcademic Medical Center\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e60\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrivate Practice\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCommunity hospital\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCommunity Health Center\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eSetting\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInpatient\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e55\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOutpatient\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBoth\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003ePatient\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatric\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e53\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdult\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003eNote: Total N\u0026thinsp;=\u0026thinsp;120; Frequencies add up to less than 120 because of missing responses.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \n\u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eConfirmatory factor analysis (CFA) results of baseline, pilot, and final Clinical Sustainability Assessment Tool instruments.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhase\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSubscales\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eItems\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChi/df\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCFI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eRMSEA\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eSRMR\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eAIC\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBaseline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.121\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.119\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e16039\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePilot\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.09\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.095\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.094\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e15189\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFinal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.084\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.075\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e10453\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"8\"\u003eNote: Total N\u0026thinsp;=\u0026thinsp;120; CFA model fit with robust maximum-likelihood.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSubscale reliabilities (internal consistency) for the Clinical Sustainability Assessment Tool subscales.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSubscale\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCronbach\u0026rsquo;s α\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEngaged Staff \u0026amp; Leadership\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEngaged Stakeholders\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.82\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOrganizational Readiness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWorkflow Integration\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eImplementation \u0026amp; Training\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMonitoring \u0026amp; Evaluation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOutcomes \u0026amp; Effectiveness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003eNote: Each subscale is made up of 5 items.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCSAT subscale and total score descriptive statistics. (N\u0026thinsp;=\u0026thinsp;120)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSubscale\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLow\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEngaged Staff \u0026amp; Leadership\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEngaged Stakeholders\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOrganizational Readiness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4.97\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWorkflow Integration\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eImplementation \u0026amp; Training\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMonitoring \u0026amp; Evaluation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOutcomes \u0026amp; Effectiveness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.01\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal Score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e6.86\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e "}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"sustainability, healthcare, clinical science, measurement development, reliability","lastPublishedDoi":"10.21203/rs.3.rs-58012/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-58012/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground \u003c/strong\u003eFew validated assessment tools are available to support sustainable implementation, an increasingly recognized need among clinicians. We describe the development of the Clinical Sustainability Assessment Tool, or CSAT, designed to assess factors that contribute to sustainable practices in clinical settings. \u003c/p\u003e\u003cp\u003e\u003cstrong\u003eMethods \u003c/strong\u003eFifty participants from clinical and research fields were recruited to brainstorm factors that lead to sustained clinical practices. Once repeated factors were removed, participants sorted remaining factors based on similarity. Using concept mapping analyses, we grouped these factors into meaningful domains and developed an initial tool. We then recruited 126 practicing clinicians to pilot and evaluate the tool. Individuals were recruited from clinical settings across pediatric (53%) and adult (47%) medical and surgical subspecialties, and averaged 11 years in their position. The data were analyzed using confirmatory factor analysis (CFA) to test hypothesized subscale structure in the instrument. We used root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) to assess fit and thus the ability of CSAT to measure the identified domains.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eResults \u003c/strong\u003eThe concept mapping resulted in seven domains and 47 items. The pilot and CFA resulted in 35 items, five per domain. The RMSEA of 0.084 and the SRMR of 0.075 indicated good fit. The final domains were: engaged staff and leadership, engaged stakeholders, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness. \u003c/p\u003e\u003cp\u003e\u003cstrong\u003eConclusions \u003c/strong\u003eThe CSAT is a new reliable assessment tool which allows for greater practical and scientific understanding of contextual factors that enable sustainable clinical practices over time. The full CSAT instrument is available at \u003ca href=\"https://sustaintool.org/\" rel=\"noopener noreferrer\" target=\"_blank\"\u003ehttps://sustaintool.org\u003c/a\u003e.\u003c/p\u003e","manuscriptTitle":"The Clinical Sustainability Assessment Tool: Measuring Organizational Capacity to Promote Sustainability in Healthcare","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2020-08-17 22:48:38","doi":"10.21203/rs.3.rs-58012/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"509b8ee6-81d3-4bee-9709-24c4bb7b1258","owner":[],"postedDate":"August 17th, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":307948,"name":"Health Policy"}],"tags":[],"updatedAt":"2021-07-27T20:57:47+00:00","versionOfRecord":{"articleIdentity":"rs-58012","link":"https://doi.org/10.1186/s43058-021-00181-2","journal":{"identity":"implementation-science-communications","isVorOnly":false,"title":"Implementation Science Communications"},"publishedOn":"2021-07-17 20:57:47","publishedOnDateReadable":"July 17th, 2021"},"versionCreatedAt":"2020-08-17 22:48:38","video":"","vorDoi":"10.1186/s43058-021-00181-2","vorDoiUrl":"https://doi.org/10.1186/s43058-021-00181-2","workflowStages":[]},"version":"v1","identity":"rs-58012","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-58012","identity":"rs-58012","version":["v1"]},"buildId":"cBFmMYwuxLRRLfASyISRj","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}