Development and Testing of an Interactive Evaluation Tool: The Evaluating QUality and ImPlementation (EQUIP) Tool

Development and Testing of an Interactive Evaluation Tool: The Evaluating QUality and ImPlementation (EQUIP) Tool | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Development and Testing of an Interactive Evaluation Tool: The Evaluating QUality and ImPlementation (EQUIP) Tool Laura McAlpine, Candace Ramjohn, Erin L. Faught, Naomi Popeski, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4572444/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 31 Mar, 2025 Read the published version in Implementation Science Communications → Version 1 posted 5 You are reading this latest preprint version Abstract Background: This article describes the development of an interactive online tool that helps people build robust evaluations. Evaluating implementation outcomes is gaining momentum in health service delivery organizations. Teams are recognizing the importance of capturing and learning from their implementation efforts, and implementation scientists have published extensively on implementation outcomes. However, quality improvement approaches and tools are better known and routinely used in healthcare to improve processes and outcomes. An example of a widespread quality improvement tool is the Alberta Quality Matrix for Health, a planning and evaluation framework that guides most healthcare-related evaluations in Alberta, Canada. This matrix is missing measures of implementation, which are essential to produce robust evaluations of change initiatives. Despite requests to integrate implementation science and quality improvement concepts, there is a dearth of tools and training opportunities. This lack of integration limits improvement and implementation in the health system. Methods: This work aimed to co-design a product that helps users integrate quality and implementation outcomes and strengthen existing evaluation processes. Rapid, iterative co-design sessions and usability testing were used to develop an interactive evaluation tool. Feedback collected and analyzed from usability testing in the seven-month testing period refined the digital prototype into a comprehensive online product. Results: The result is the EQUIP (Evaluating QUality and ImPlementation) Tool. This online, interactive tool integrates quality measures from the Alberta Quality Matrix for Health and implementation measures from a commonly used outcomes framework developed by Proctor and colleagues. It allows users to explore implementation outcomes and quality dimensions from different perspectives and select questions and indicators relevant to their project. Conclusion: The EQUIP tool was co-designed and refined with end users to create an accessible and useful online tool. This work is an example of integrating quality and implementation science in ways that strengthen overall healthcare quality improvements. Evaluation quality improvement implementation science implementation outcomes usability testing Figures Figure 1 Figure 2 Figure 3 Figure 4 Background The Issue Evaluating implementation outcomes is gaining momentum in health service delivery organizations. 1 , 2 Teams are recognizing the importance of capturing and learning from their implementation efforts. 3 – 5 This is especially critical if an innovation (i.e., a new way of doing things) is successful. Those responsible for implementing the innovation need to understand all the factors, formal and informal, seen and unseen, that influence the outcome of an implementation process, so that it can be successfully replicated elsewhere. As Proctor and others have described, when health-system innovations fail, and they often do, it is essential to know if the failure occurred because the innovation was ineffective (innovation failure) or if a good innovation was poorly implemented (implementation failure). 6 , 7 Healthcare has largely adopted quality improvement approaches as a way to improve processes and outcomes. As a result, quality improvement methods and tools aim to improve local healthcare performance, 8 , 9 and are widely accepted in health service delivery organizations. 10 This is demonstrated by the extensive availability and uptake of quality improvement infrastructure, supports, and tools available across different care contexts. 11 , 12 A local example of this is the Alberta Quality Matrix for Health, a single quality improvement health planning and evaluation framework that guides the design and development of most healthcare-related evaluations taking place in the province. 11 The emerging field of implementation science, dedicated to understanding methods and strategies to move research evidence into healthcare practice and policy, 13 is gaining popularity as a source of guidance that can strengthen quality improvement approaches. The growing movement to evaluate implementation in a robust way is an important example of this development. 4 , 14 , 15 Despite numerous requests to improve effective innovation uptake and sustainment by integrating implementation science and quality improvement concepts 15 , there are few tools and training opportunities currently available to merge these areas. 4 , 14 The lead authors of this paper (LM, GZ) identified a need for such a tool in their health innovation implementation consultation services with the Alberta Strategy for Patient-Oriented Research SUPPORT Unit (AbSPORU). In their consultations, they repeatedly noted how the lack of integration between quality improvement and implementation science limits improvement and implementation. This results in requests from across the health system to help build evaluations that incorporate both implementation outcomes and quality improvement outcomes outlined in the Alberta Quality Matrix for Health. Thus, the authors undertook a research project to co-design a product that helps users integrate quality and implementation outcomes and strengthen existing evaluation processes in the provincial health system. Aims In this paper, we describe the EQUIP (Evaluating QUality and ImPlementation) Tool, an online evaluation tool that integrates quality and implementation outcomes. This tool was co-designed with a team of content experts and health system partners and then tested and refined with a broader group of potential end-users (i.e., people who would use the tool) to create the final product. The questions used to guide the research project were: 1. How might we co-design an evaluation tool that brings together the Alberta Quality Matrix for Health with Proctor et al.’s taxonomy of implementation outcomes? 2. How might we test and refine the tool so that it meets the needs of intended users? 3. How might we ensure the tool is accessible and valuable to users and the work they do? By answering these questions, we aimed to support researchers, funders, and practitioners working in health service delivery organizations who want to, directly or indirectly, strengthen implementation evaluation capacity and establish routine evaluation of implementation outcomes in health research studies and practice change initiatives. Context This project took place in Alberta, Canada, within the context of Alberta Health Services, the single health authority in the province that provides programs and services at more than 900 facilities throughout the province, including hospitals, clinics, continuing care facilities, cancer centres, mental health facilities, and community health sites. 16 The Alberta Quality Matrix for Health is very well recognized and frequently used in evaluations taking place in Alberta Health Services (Table 1 ). It is designed to assess patient outcomes and quality in a standardized way across the complex healthcare system. 11 However, the matrix is missing measures of implementation, which are essential to produce robust evaluations following implementation of initiatives. 2 , 6 , 17 Table 1 Definitions of quality dimensions and implementation outcomes Dimension or Outcome Definition Quality Dimensions (from the Alberta Quality Matrix for Health) Acceptability Health services are respectful and responsive to user needs, preferences and expectations Accessibility Health services are obtained in the most suitable setting in a reasonable time and distance Appropriateness Health services are relevant to user needs and are based on accepted or evidence-based practice Effectiveness Health services are based on scientific knowledge to achieve desired outcomes Efficiency Resources are optimally used in achieving desired outcomes Safety Mitigate risks to avoid unintended or harmful results Implementation Outcomes (from Proctor et al.’s Taxonomy) Acceptability Perception among implementation stakeholders that a treatment, service, practice or innovation is agreeable, palatable, or satisfactory Adoption Intention, initial decision, or action to try or employ an innovation or evidence-based practice Appropriateness Perceived fit, relevance, or compatibility of the innovation or evidence-based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation to address a particular issue or problem Feasibility Extent to which a new treatment, or an innovation, can be successfully used or carried out within a given agency or setting Fidelity Degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the program developers Cost Cost impact of an implementation effort Penetration Integration of a practice within a service setting and its subsystems Sustainability Extent to which a newly implemented treatment is maintained or institutionalized with a service setting’s ongoing, stable operations In the field of implementation science, a commonly used outcomes framework developed by Proctor and colleagues articulates eight implementation outcomes (Table 1 ), which precede but are interrelated with service and client outcomes. 6 The service outcomes, which are based on the six quality improvement aims from the Institute of Medicine, 18 complement the quality dimensions of the Alberta Quality Matrix for Health. Thus, integrating quality measures from the Alberta Quality Matrix for Health and implementation measures from Proctor et al. is a natural fit for the development of an evaluation tool that integrates strengths of both quality improvement and implementation science. Methods We report the development of the EQUIP tool in accordance with the 'Guidance for reporting intervention development studies in health research' (GUIDED) checklist. 19 See Additional File 1 for the completed checklist. The design and usability stages of the project took place over eighteen months, from January 2020 to June 2021, with monitoring and further refinement throughout that time. Creation of the Co-Design Team During an initial brainstorming session, AbSPORU staff identified potential co-design team members based on pre-existing relationships with stakeholders from Alberta Health Services. The aim was to have at least three members from different areas of Alberta Health Services and three members from AbSPORU. Team members (n = 6) consisted of end-users, provincial health system partners from Alberta Health Services, AbSPORU and specialists in evaluation, implementation science, and design. Some team members were connected to more than one of these roles, and all team members were based in Alberta, Canada. Design The tool was developed using a co-design approach to incorporate end users’ perspectives in the product design. A combination of the Successive Approximation Model and Design Thinking was used to guide the process. 20 – 22 The Successive Approximation Model, from the field of instructional design, outlines an iterative, participatory design and development process that focuses on end users’ experiences, engagement, and motivation. 20 Design Thinking is a human-centered, solution-based approach to design, which encourages teams to focus on the users and their contexts. It accomplishes this by drawing on methods targeted at gaining a deep understanding of users’ needs, experiences, and desires from their perspective to develop solutions that are effective and accessible. 21 , 22 The co-design team first developed a prototype through rapid, iterative, virtual design sessions held over a video conferencing platform, Zoom (Zoom Video Communications, Inc.). At the first design session, attendees discussed potential users and their contexts, formulated a “how might we” question to guide design and development, brainstormed possible solutions, and began prototyping. A voting exercise was held at the end of the session to select one prototype for further development. A second design session was held a few weeks later to revisit discussions held during the first session and confirm selection of the prototype. The team’s graphic designer (CR) then developed an online, interactive prototype that was further refined through usability testing with targeted user groups. Usability Usability testing was conducted using two different online questionnaires, administered using Google Forms: one on the tool’s functionality and another on the design and accessibility of the tool itself. See Additional File 2 for the complete questionnaires. The first questionnaire consisted of several tasks for users to work through to ensure they could access, navigate, and interact with the tool. The second questionnaire asked for feedback on the ease of use, relevance of the tool to their work, likelihood of recommending the tool, along with targeted questions on what to improve (and how), and what users liked about the tool. Data was collected from yes/no responses, multiple choice responses, Likert scales, and open-ended answers (Table 2 ). A group of potential end-users were recruited through existing relationships with co-design team members to complete the usability testing. Table 2 Usability testing questionnaire topics and example questions Questionnaire Topics Example Question Question Style Usability Questionnaire Accessibility Task 1 - Please go to the evaluation tool website by clicking on the link below. Are you able to access the website using your preferred web browser? Yes/No Navigation Task 3 - Please click on Use the Tool at the top of the page. Did the content on the Use the Tool page load properly? Yes/No Interaction Task 5 - Scroll down a little further to Healthcare Providers, find Feasibility (one of the blue boxes), and click on three buttons: Feasibility, Sample Implementation Question, and Sample Indicators. In the list of Sample Indicators, click on Feasibility of Intervention Measure and Technology Acceptance Survey. When you clicked on the Feasibility of Intervention Measure, did a new window open? Multiple choice Feedback Questionnaire Ease of use How difficult was it to use and understand the tool? 5-point Likert Relevance Do you think this tool could be helpful for your evaluations? Multiple choice General feedback Do you have any suggestions on how we can improve the instructions? Open-ended Likelihood of recommending How likely are you to recommend the tool to others? 5-point Likert Usability feedback was collected and analyzed using descriptive statistics in Microsoft Excel. Adjustments were made to the prototype in an ongoing manner. Regular, live meetings with two lead team members (LM and CR) were held over Zoom every six to eight weeks to discuss results, identify improvements, and update the tool prototype (and accompanying website), increasing to every two weeks near the end of testing. In-depth consultations were conducted with a subset of usability testing respondents (n = 8) to discuss their usability testing experience, review resulting updates to the prototype, and better understand the tool’s relevance to their work. All consultations were carried out over Zoom over 6 months (December 2020 to June 2021) and ran from 30 to 45 minutes in length. All respondents were asked the same set of three questions during the consultations: 1. What do you think of the updates to the website, especially the overall design and function? 2. How might the tool help your work or the work of others on your team? 3. What improvements could we make to the tool and website to support this? As with the questionnaire responses, key points from the consultations were compiled by two lead team members (LM and CR) and used to refine the prototype and accompanying website. Results Co-Design and Initial Prototype During the first design session, the team identified both researchers and practitioners in the health system as potential users of the tool, with practitioners encompassing community and acute care settings, clinical and support staff. It was felt that there was energy and interest among clinicians and managers around improvement, and that a tool like this could support improvement efforts, develop better evaluations, and demonstrate impact. It was also suggested that a tool like this could help groups applying for implementation science-focused funding opportunities. These often require applicants to incorporate implementation considerations in their studies, including implementation outcomes, which is often new to research teams. The tool needed to be user-friendly and easy to apply. It could not be too technical, theoretical or make peoples’ work more difficult. To achieve this, team members agreed early on that an online, web-based format that would enable users to select what they need and create a customized form would be ideal. The initial prototype after the first design session was a matrix that mirrored the Alberta Quality Matrix for Health, with the Dimensions of Health Service Quality across the top, and the different types of implementation research outcomes (client, service and implementation) from Enola Proctor’s paper listed down the side. However, team members recognized that this initial prototype was at odds with users’ needs. It took too much time to understand and navigate the sizeable table. After the second design session, the table was refined to include the six Dimensions of Health Service Quality and Enola Proctor’s Taxonomy of Implementation Outcomes across the top, and various key perspectives listed down the side. Two of the six Alberta Quality Matrix for Health dimensions overlap with Proctor et al.’s recommended taxonomy of implementation outcomes (in name only): Acceptability and Appropriateness. This overlap between the two frameworks helped to connect them because the importance of assessing Acceptability and Appropriateness is already well established in practice. However, Proctor et al.’s outcomes focus mostly on the perspectives of healthcare providers and the healthcare setting while the Alberta Quality Matrix for Health has a major focus on the patient experience. The co-design team felt it was important to consider several perspectives in an evaluation. The Interactive Systems Framework and Alberta Health Services’ use of the Quadruple Aim informed how to build the tool in light of this initial session and the question of which perspectives should be included in the final tool. 23 , 24 Team members also suggested that sample questions would be helpful to include. A frequent comment during the design sessions was that ‘people don’t know what they don’t know’. This started discussions around how this tool could prompt thinking and build capacity of the users. While the matrix demonstrated the integration of the Alberta Quality Matrix for Health dimensions and the implementation outcomes, due to coding complications it would not generate the form as intended. Therefore, the team’s graphic designer developed this concept into an online slide-box format. A comparison between the two layouts (matrix vs slide box) was included in usability testing (Fig. 1 ). The iterative development process allowed the team to discuss and consider other tools and frameworks to adapt the prototype to better fit the local context. These included the British Columbia Health Quality Matrix and the RE-AIM framework (RE-AIM stands for Reach, Effectiveness, Adoption, Implementation, and Maintenance). 12 , 25 A member of the co-design team highlighted that the British Columbia Health Quality Matrix includes the dimension ‘Equity’, which had not yet made its way into the Alberta Quality Matrix for Health but is increasingly important to consider. 12 The British Columbia Health Quality Matrix also does a good job of defining the quality dimensions and indicators in a straightforward way, which the team felt was important. The RE-AIM Framework was identified by the co-design team as being widely used and well understood in the Alberta health system, therefore some of the language, definitions and indicators for the implementation outcomes of the EQUIP tool were adapted from the RE-AIM framework (e.g., Reach instead of Penetration). 25 Usability Testing Usability testing of the prototype ran from November 2020 to May 2021. Improvements were made to the first testable prototype iteratively based on ongoing usability testing and feedback results. For example, sample questions and indicators were added, others were removed, new pages were added to the accompanying website, and existing pages were overhauled. Although user responses varied during testing, most asked that the tool and website be kept as straightforward and applicable as possible. Twenty-one people completed usability testing in three separate waves over the course of seven months. The majority of users came from either the healthcare system (57%) or academia (35%). Two users were evaluation specialists for community or health policy organizations (Fig. 2 ). All users had experience in either quality improvement, evaluation or implementation science and the majority had knowledge of the Alberta Quality Matrix for Health. All users (n = 21) completed all the tasks outlined in the usability questionnaire, and all users thought the tool could help with their work, while 81% (n = 17) agreed that the included perspectives (patients, healthcare providers, support teams, and organizations) were thorough enough. Three respondents thought key perspectives were missing, including that of funders and care partners. The remaining respondent thought too many perspectives were already included. Overall, users’ responses were positive and helpful for informing updates to the prototype (Fig. 3 ). Key suggestions for improvement included: tightening up the language, improving the layout of information on each page, and including an example evaluation. The overall structure and visuals as well as sample questions and indicators were features that users felt were done well and should not be changed. A majority of users (62%) said that working through the prototype prompted them to start thinking about including implementation outcomes in future evaluations. Initially the tool included five broad categories of perspectives and each perspective included all possible quality dimensions and implementation outcomes. Iterative feedback from the co-design team and usability testing respondents helped streamline which perspectives to include and which dimensions and outcomes were most relevant for each perspective. Figure 4 provides an example of how the tool changed after feedback from users and co-design team members was incorporated. Co-design and usability feedback helped to resolve inconsistencies in language in ways that help to integrate quality and implementation outcomes. For example, Appropriateness and Acceptability have different definitions in the Alberta Quality Matrix for Health and Proctor’s implementation outcomes, but the Quality definition was felt to suit the patient perspective, while the implementation definition suited providers. There was also some discussion about the difference between Appropriateness and Acceptability as they are conceptually similar which can cause confusion, but referring to the examples in Proctor’s paper provided clarity. 6 Usability testing significantly influenced the design and aesthetic of the tool. The initial prototype of the tool incorporated the colours and structure of the Alberta Quality Matrix for Health, which was originally included to help users by drawing on their familiarity with this matrix. Although some users resonated with the colours, others found it confusing and irrelevant. Content and design elements from the British Columbia Health Quality Matrix were then incorporated in response to this feedback. In further testing, it was found that these elements resonated better with users and made the tool feel simpler to use. Final Testing Eight consultations were completed with a subset of the respondents involved in usability testing to review updates and understand the relevance of the EQUIP Tool to their work. Of the eight, three were with implementation science networks within Alberta Health Services, three worked with other groups in Alberta Health Services, one was a health policy researcher, and one worked with the Health Quality Council of Alberta. Consultations were conducted over the course of 6 months with improvements made iteratively such that the conversations that happened later had fewer suggestions for improvement and specifically mentioned things like: Easy to navigate Clicking on the boxes was easy to understand but makes site clean… Text only when you want to see it Overall, thought tool was great, usability navigation, visuals, examples, easy to use Improvements included aesthetic updates to the tool prototype and accompanying website, simplifying the instructions for the tool, and adding links to validated measures directly within the tool. The overall response from users was very positive. Updates were well received, to both the tool and the accompanying website. Some users described how they were already using the tool to build evaluation capacity in their teams and inform evaluations of innovations in the health system. Users also shared how they already used the EQUIP tool in their work, including collaboratively designing evaluations, selecting standardized implementation measures, and informing an upcoming refresh of the Alberta Quality Matrix for Health. The Final Tool: The EQUIP (Evaluating QUality and ImPlementation) Tool (theequiptool.com) The final tool is housed on a website (theequiptool.com) that includes information about what the tool is, who the intended users are, and why it is important. There are interactive visuals that provide definitions of the dimensions of health service quality outlined in the Alberta Quality Matrix for Health, Proctor’s recommended list of implementation outcomes, and the different perspectives included. A resource page includes a link to a PDF version of the tool as well as an example. The tool itself is interactive, allowing users to explore the implementation outcomes and quality dimensions from the different perspectives and select those questions and indicators that are relevant to their project. After completing selections, users are able to save or print a hard copy record of them. Each outcome is defined and includes sample questions from either a quality perspective or implementation science perspective or both (for those domains represented by both Acceptability and Appropriateness). Examples of indicators and validated measures are included in the tool. Sample indicators and links to measures were pulled from several different evaluation frameworks used within Alberta Health Services as well as systematic reviews and scoping reviews that consider Proctor’s list of implementation outcomes. 26 – 28 The tool is meant to stimulate thinking and discussions. Therefore, any of the questions or indicators may need to be further developed by the user to suit their needs. Discussion Through rapid, iterative design sessions and usability testing, an interactive evaluation tool was developed that incorporates both quality and implementation outcomes. A co-design approach was used to ensure that the tool developed would be relevant and practical for end-users. 29(p) The result is an interactive evaluation tool called EQUIP (Evaluating QUality and ImPlementation), available at theequiptool.com. An important feature of the tool is that it highlights the different perspectives that should be considered when evaluating implementation efforts. While improving patient outcomes is the ultimate goal of most quality improvement and implementation efforts, it is also important to consider the impact of a given implementation on the people and systems involved. In fact, all of these perspectives are needed to understand why implementation efforts succeed or fail and to determine what constitutes success in the first place. 6 , 23 , 24 The perspectives included in the EQUIP Tool take into account the various roles involved in implementation and the broader context. They are based on the Interactive Systems Framework and Alberta Health Services use of the Quadruple Aim and refined by feedback from users. 23 , 24 The Interactive Systems Framework centers on the people, support and organizations that are needed to carry out the activities required for successful implementation. 23 The Quadruple Aim highlights not only patient experience but provider satisfaction in how they are supported in providing care. 24 Providers are especially important as they hold a high level of influence in the system. 30 They are largely responsible for delivering products and services to patients while also being accountable to the systems within which they work. In fact, all of the implementation outcomes and several quality dimensions are considered from the provider perspective in the EQUIP tool. Without considering the whole context within which an intervention is being implemented, important evaluation questions could be missed. The EQUIP tool encourages users to consider important quality and implementation questions from each of these key perspectives. While the level of analysis for implementation outcomes has not been well studied, 6 , 31 the EQUIP tool provides an opportunity to explore if a given level or perspective is appropriate for a particular implementation outcome. The consultations provided an opportunity to delve deeper into different frameworks and consider other definitions and perspectives. As Reilly and colleagues point out, there is a great deal of consistency between RE-AIM and Proctor’s implementation outcomes. 31 However, until recently RE-AIM did not include Acceptability, Appropriateness, and Feasibility (which they include as antecedents of several implementation outcomes in their proposed expansion to RE-AIM indicators). 31 Different frameworks have different definitions for the various outcomes, therefore, ongoing discussions with users helped to find the right fit for the different perspectives in the tool. Similarly, Weiner et al 32 noted that measures are rarely defined or distinguishable from one another, resulting in confusion around what is being assessed. Therefore, having sample questions and indicators in the tool helps with understanding the different dimensions and outcomes and having links to validated measures improves certainty. The engagement with the co-design team and potential end-users was an important part of ensuring the tool was relevant and responsive to users’ needs. The result is a tool that some stakeholders are already using for training (e.g., to better understand the quality dimensions and implementation outcomes), to stimulate thinking (e.g., about the different perspectives involved), and develop evaluations. The tool has also since been included as a resource in the updated Alberta Health Services Innovation Pipeline Primer 2.0. 33 The Innovation Pipeline Primer 2.0 provides information about the evidence needed to safely move a good innovative idea into action, including sample outcomes. Thus, questions and indicators from EQUIP help inform the evidence for quality improvement and implementation effectiveness. Further use of the tool and having a place for feedback built into the tool will assist with future improvements. Strengths and Limitations The co-design process and usability testing were strengths that ensured the end product would be relevant to end-users. Although several frameworks were discussed, there may be others that were not considered but could have added value. However, the frameworks included were those that were familiar to end-users in Alberta. Depending on the scope of a given evaluation, certain perspectives may be missing and should be considered if relevant to that evaluation. Patients were not consulted when selecting the outcome definitions or indicators for the patient perspective. This appears to be a limitation of most frameworks. However, the Alberta Quality Matrix for Health is currently being refreshed and is involving patient family advisors [personal communication]. Future updates of the EQUIP tool will consider any important changes to outcomes or indicators for patients (by patients). The EQUIP tool was designed to help users incorporate implementation outcomes into quality evaluations. It can help with buy-in, build capacity, consider different perspectives, provide sample questions and indicators, and connect users to validated measures. The tool was not developed to identify stakeholders, assess readiness, build entire evaluation plans, or guide implementation planning. Conclusions The EQUIP tool was co-designed and refined with end users to create an accessible and useful online tool. This interactive tool helps people build robust evaluations that incorporate the six dimensions of health service quality, outlined in the Alberta Quality Matrix for Health, and standardized implementation outcomes. This work is an example of integrating quality and implementation science in ways that strengthen overall healthcare quality improvements. Abbreviations AbSPORU – Alberta Strategy for Patient-Oriented Research SUPPORT Unit EQUIP – Evaluating Quality and Implementation RE-AIM – Reach, Effectiveness, Adoption, Implementation, Maintenance Declarations Ethics approval and consent to participate Ethical approval was obtained from the University of Alberta Research Ethics Board 2 (Pro00130141). All participants provided informed consent. Consent for publication Not applicable. Availability of data and materials All data generated or analysed are not available due to participant privacy but are available from the corresponding author on reasonable request. Competing interests The authors declare that they have no competing interests. Funding This work was led by the Alberta SPOR SUPPORT Unit (AbSPORU), which is co-funded by the Strategy for Patient-Oriented Research program of the Canadian Institute for Health Research (CIHR), Alberta Innovates and the University Hospital Foundation. AbSPORU also acknowledges its implementation partners: the University of Alberta, the University of Calgary, the University of Lethbridge, Alberta Health Services, Athabasca University, the Women and Children's Health Research Institute, the Alberta Children's Hospital Research Institute and Alberta Health. Authors' contributions LM and GZ conceptualized the project. LM and CR led the collaboration with the tool development team: LM designed the development process, and CR led the design process. GZ supervised the overall project. LM, CR, EF, NP, EK, and GZ designed and developed the EQUIP Tool. LM led data collection and analysis. LM and GZ drafted the manuscript. CR, EF, NP, and EK reviewed and provided input on numerous versions of the manuscript. All authors read and approved the final manuscript. Acknowledgements Thank you, Anna Noga and Morgan Potter for contributing to early design sessions, Cody Alba for reviewing an early draft of the manuscript, and Stephanie Brooks and Denise Thomson for reviewing and revising manuscript drafts. This work was led by the AbSPORU. AbSPORU also acknowledges its implementation partners: the University of Alberta, the University of Calgary, the University of Lethbridge, Alberta Health Services, Athabasca University, the Women and Children's Health Research Institute, the Alberta Children's Hospital Research Institute and Alberta Health. References Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health . 2019;7:64. doi:10.3389/fpubh.2019.00064 Tabin M, Diacquenod C, Petitpierre G. Evaluating implementation outcomes of a measure of social vulnerability in adults with intellectual disabilities. Res Dev Disabil . 2021;119:104111. doi:10.1016/j.ridd.2021.104111 Yiu V, Belanger F, Todd K. Alberta’s Strategic Clinical Networks: Enabling health system innovation and improvement. Can Med Assoc J . 2019;191(Suppl):S1-S3. doi:10.1503/cmaj.191232 Rapport F, Smith J, Hutchinson K, et al. Too much theory and not enough practice? The challenge of implementation science application in healthcare practice. J Eval Clin Pract . Published online June 30, 2021. doi:10.1111/jep.13600 Moullin JC, Dickson KS, Stadnick NA, et al. Ten recommendations for using implementation frameworks in research and practice. Implement Sci Commun . 2020;1:42. doi:10.1186/s43058-020-00023-7 Proctor E, Silmere H, Raghavan R, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health . 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7 Quinn Patton M. Essentials of Utilization-Focused Evaluation . 1st ed. SAGE Publications, Inc; 2011. Accessed November 15, 2021. https://us.sagepub.com/en-us/nam/essentials-of-utilization-focused-evaluation/book233973 Beyea SC, Nicoll LH. Is it research or quality improvement? AORN J . 1998;68(1):117-119. doi:10.1016/S0001-2092(06)62732-4 Reinhardt AC, Ray LN. Differentiating quality improvement from research. Appl Nurs Res . 2003;16(1):2-8. doi:10.1053/apnr.2003.59000 Maritz R, Scheel-Sailer A, Schmitt K, Prodinger B. Overview of quality management models for inpatient healthcare settings. A scoping review. Int J Qual Health Care . 2019;31(6):404-410. doi:10.1093/intqhc/mzy180 Health Quality Council of Alberta. Alberta Quality Matrix for Health User Guide. Published online 2005. Accessed September 29, 2021. https://hqca.ca/about/how-we-work-health-quality-council-of-alberta/the-alberta-quality-matrix-for-health-1/ BC Patient Safety & Quality Council. BC Health Quality Matrix. Published online 2020. Accessed August 24, 2021. https://bcpsqc.ca/wp-content/uploads/2020/02/BC-Health-Quality-Matrix-March-2020.pdf Glasgow RE, Eckstein ET, ElZarrad MK. Implementation Science Perspectives and Opportunities for HIV/AIDS Research: Integrating Science, Practice, and Policy. JAIDS J Acquir Immune Defic Syndr . 2013;63:S26. doi:10.1097/QAI.0b013e3182920286 Moore JE. 5 ways to improve Implementation Science – lessons from quality improvement. The Center for Implementation. Published December 2019. Accessed November 15, 2021. https://thecenterforimplementation.com/implementation-in-action-bulletin/december-2019 Tyler A, Glasgow RE. Implementing Improvements: Opportunities to Integrate Quality Improvement and Implementation Science. Hosp Pediatr . 2021;11(5):536-545. doi:10.1542/hpeds.2020-002246 Services AH. About AHS. Alberta Health Services. Accessed May 19, 2022. https://www.albertahealthservices.ca/about/about.aspx Wensing M, Grol R. Knowledge translation in health: how implementation science could contribute more. BMC Med . 2019;17(1):88. doi:10.1186/s12916-019-1322-9 Wolfe A. Institute of Medicine Report: Crossing the Quality Chasm: A New Health Care System for the 21st Century. Policy Polit Nurs Pract . 2001;2(3):233-235. doi:10.1177/152715440100200312 Duncan E, O’Cathain A, Rousseau N, et al. Guidance for reporting intervention development studies in health research (GUIDED): an evidence-based consensus study. BMJ Open . 2020;10(4):e033516. doi:10.1136/bmjopen-2019-033516 Reiser RA, Dempsey JV. Trends and Issues in Instructional Design and Technology. Fourth edition. Pearson; 2018. https://login.ezproxy.library.ualberta.ca/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=cat03710a&AN=alb.7870354&site=eds-live&scope=site Doorley S, Holcomb S, Klebahn P, Segovia K, Utley J. Design Thinking Bootleg. Published online 2018. Accessed August 25, 2021. https://static1.squarespace.com/static/57c6b79629687fde090a0fdd/t/5b19b2f2aa4a99e99b26b6bb/1528410876119/dschool_bootleg_deck_2018_final_sm+%282%29.pdf Design Thinking Frequently Asked Questions (FAQ). IDEO | Design Thinking. Published November 12, 2021. Accessed November 12, 2021. https://designthinking.ideo.com/faq/whats-the-difference-between-human-centered-design-and-design-thinking Wandersman A, Duffy J, Flaspohler P, et al. Bridging the Gap Between Prevention Research and Practice: The Interactive Systems Framework for Dissemination and Implementation. Am J Community Psychol . 2008;41(3-4):171-181. doi:10.1007/s10464-008-9174-z Alberta Health Services. Enhancing Care in the Community. Alberta Health Services. Published August 25, 2021. Accessed August 25, 2021. https://www.albertahealthservices.ca/about/Page13457.aspx Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health . 2019;7:64. doi:10.3389/fpubh.2019.00064 Mettert K, Lewis C, Dorsey C, Halko H, Weiner B. Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties. Implement Res Pract . 2020;1:2633489520936644. doi:10.1177/2633489520936644 Khadjesari Z, Boufkhed S, Vitoratou S, et al. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implement Sci . 2020;15(1):66. doi:10.1186/s13012-020-01027-6 Willmeroth T, Wesselborg B, Kuske S. Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review. Inq J Health Care Organ Provis Financ . 2019;56:0046958019861257. doi:10.1177/0046958019861257 Langley J, Wolstenholme D, Cooke J. ‘Collective making’ as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res . 2018;18:585. doi:10.1186/s12913-018-3397-y Pereno A, Eriksson D. A multi-stakeholder perspective on sustainable healthcare: From 2030 onwards. Futures . 2020;122:102605. doi:10.1016/j.futures.2020.102605 Reilly KL, Kennedy S, Porter G, Estabrooks P. Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks. Front Public Health . 2020;8. Accessed May 19, 2022. https://www.frontiersin.org/article/10.3389/fpubh.2020.00430 Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci . 2017;12(1):108. doi:10.1186/s13012-017-0635-3 Waye A, Hughes B, Mrklas K, Fraser N. Innovation Pipeline: Intent to Scale for Impact. Published online 2020. Accessed August 24, 2021. https://www.albertahealthservices.ca/assets/about/scn/ahs-scn-so-innov-pipeline-primer.pdf Supplementary Files AdditionalFile1GUIDEDChecklist13Jun2024.pdf AdditionalFile2Questionnaires12Jun2024.pdf Cite Share Download PDF Status: Published Journal Publication published 31 Mar, 2025 Read the published version in Implementation Science Communications → Version 1 posted Editorial decision: Major revision 09 Sep, 2024 Reviewers agreed at journal 25 Jul, 2024 Reviewers invited by journal 19 Jun, 2024 Editor assigned by journal 14 Jun, 2024 First submitted to journal 13 Jun, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4572444","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":316515034,"identity":"d88f9e7a-415a-40f1-a506-d6fb580e82b9","order_by":0,"name":"Laura McAlpine","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFUlEQVRIiWNgGAWjYBACPjiLnYHhAAODDZDBg18LG5zFDNaSBmSAtCQQqQUIDhOhhf2M2YcfDNvk+Zt5Dx4uqDif2A9kPK78YcfA334AuxaeHOOZPQy3DWcc5ks4POPM7UQgI9nwTEIyg8QZ7FaxMeQYM/Aw3E5gOMxjcJi37XZiw2EeM8mGBGYGAxyuY+N/Y8z4B6hFHqzl37nE+Yd5zH82JNQzGPA/wK5FIseYGWSLAVhLw4HEDUBbGBsSDjMYSOCwReJZMbOMwW3DjSC/8BxLNgYykiUb0o7zSNzAbgs/f/JmxjcVt+Xljvce/sxTYyc773jvwY8NNtVy/P3YbYEAAxCBFukE0gDRakbBKBgFo2BEAgBp7FfTTk07OwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0009-0006-9476-2592","institution":"University of Alberta Department of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Laura","middleName":"","lastName":"McAlpine","suffix":""},{"id":316515035,"identity":"d482c6fc-4b4f-4546-a073-a13d2e43e8f1","order_by":1,"name":"Candace Ramjohn","email":"","orcid":"","institution":"University of Alberta Department of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Candace","middleName":"","lastName":"Ramjohn","suffix":""},{"id":316515036,"identity":"bcd3ada6-0dd5-4b28-8aaf-0ae9ab724817","order_by":2,"name":"Erin L. Faught","email":"","orcid":"","institution":"Edmonton Public Schools","correspondingAuthor":false,"prefix":"","firstName":"Erin","middleName":"L.","lastName":"Faught","suffix":""},{"id":316515037,"identity":"4957c77c-809a-42b8-b41f-9307b154b73e","order_by":3,"name":"Naomi Popeski","email":"","orcid":"","institution":"University of Calgary Cumming School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Naomi","middleName":"","lastName":"Popeski","suffix":""},{"id":316515038,"identity":"c10b8d4e-69c5-4cc9-8793-40958b2e1818","order_by":4,"name":"Eileen Keogh","email":"","orcid":"","institution":"AHS: Alberta Health Services","correspondingAuthor":false,"prefix":"","firstName":"Eileen","middleName":"","lastName":"Keogh","suffix":""},{"id":316515039,"identity":"b43ef3ce-c0be-4b98-9f89-03329fdce2af","order_by":5,"name":"Gabrielle L. Zimmermann","email":"","orcid":"","institution":"University of Alberta Department of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Gabrielle","middleName":"L.","lastName":"Zimmermann","suffix":""}],"badges":[],"createdAt":"2024-06-12 21:02:29","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4572444/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4572444/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s43058-025-00715-y","type":"published","date":"2025-03-31T15:57:23+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":59869382,"identity":"4dbceffc-877e-49b3-8204-2a1d083c48a9","added_by":"auto","created_at":"2024-07-08 16:44:48","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":97879,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparison of matrix and slide box layouts\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/fc2e497b6dd471b19d0637ff.jpg"},{"id":59869387,"identity":"30b14491-556e-4e6d-9e59-ed6dbeb38940","added_by":"auto","created_at":"2024-07-08 16:44:54","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":88927,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eUsers involved in usability testing (n=21)\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/eabb1fb7c2b7474498eb58de.jpg"},{"id":59869384,"identity":"5b10779a-af13-40b4-bd60-285e155d3569","added_by":"auto","created_at":"2024-07-08 16:44:50","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":75581,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eHighlighted feedback questionnaire responses (n=21)\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/d882dfb46355e302b68df7a8.jpg"},{"id":59869385,"identity":"87a0f28d-023f-4829-9fa9-aa23bef7c0e3","added_by":"auto","created_at":"2024-07-08 16:44:53","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":100046,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eHow the tool was streamlined after feedback from users (top = before, bottom = after)\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/8d4090eebb0ff88ac5db871f.jpg"},{"id":80082409,"identity":"db3003ab-85fc-47e6-be53-0a2886a4c7df","added_by":"auto","created_at":"2025-04-07 16:09:00","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1094371,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/765a761b-3797-4603-b95a-aecf13aca27f.pdf"},{"id":59869386,"identity":"cbe2050d-a07f-46a5-9368-64116d518894","added_by":"auto","created_at":"2024-07-08 16:44:53","extension":"pdf","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":827931,"visible":true,"origin":"","legend":"","description":"","filename":"AdditionalFile1GUIDEDChecklist13Jun2024.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/c550df78ee1744bf717515aa.pdf"},{"id":59869383,"identity":"40023f0b-0e48-4d8a-a71b-70244b44e80c","added_by":"auto","created_at":"2024-07-08 16:44:49","extension":"pdf","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":1334861,"visible":true,"origin":"","legend":"","description":"","filename":"AdditionalFile2Questionnaires12Jun2024.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4572444/v1/95bd9469f417a016323b0464.pdf"}],"financialInterests":"","formattedTitle":"Development and Testing of an Interactive Evaluation Tool: The Evaluating QUality and ImPlementation (EQUIP) Tool","fulltext":[{"header":"Background","content":"\u003cp\u003eThe Issue\u003c/p\u003e \u003cp\u003eEvaluating implementation outcomes is gaining momentum in health service delivery organizations.\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e Teams are recognizing the importance of capturing and learning from their implementation efforts.\u003csup\u003e\u003cspan additionalcitationids=\"CR4\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e–\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e This is especially critical if an innovation (i.e., a new way of doing things) is successful. Those responsible for implementing the innovation need to understand all the factors, formal and informal, seen and unseen, that influence the outcome of an implementation process, so that it can be successfully replicated elsewhere. As Proctor and others have described, when health-system innovations fail, and they often do, it is essential to know if the failure occurred because the innovation was ineffective (innovation failure) or if a good innovation was poorly implemented (implementation failure).\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eHealthcare has largely adopted quality improvement approaches as a way to improve processes and outcomes. As a result, quality improvement methods and tools aim to improve local healthcare performance,\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e,\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e and are widely accepted in health service delivery organizations.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e This is demonstrated by the extensive availability and uptake of quality improvement infrastructure, supports, and tools available across different care contexts.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e,\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e A local example of this is the Alberta Quality Matrix for Health, a single quality improvement health planning and evaluation framework that guides the design and development of most healthcare-related evaluations taking place in the province.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe emerging field of implementation science, dedicated to understanding methods and strategies to move research evidence into healthcare practice and policy,\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e is gaining popularity as a source of guidance that can strengthen quality improvement approaches. The growing movement to evaluate implementation in a robust way is an important example of this development. \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e Despite numerous requests to improve effective innovation uptake and sustainment by integrating implementation science and quality improvement concepts\u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e, there are few tools and training opportunities currently available to merge these areas.\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe lead authors of this paper (LM, GZ) identified a need for such a tool in their health innovation implementation consultation services with the Alberta Strategy for Patient-Oriented Research SUPPORT Unit (AbSPORU). In their consultations, they repeatedly noted how the lack of integration between quality improvement and implementation science limits improvement and implementation. This results in requests from across the health system to help build evaluations that incorporate both implementation outcomes and quality improvement outcomes outlined in the Alberta Quality Matrix for Health. Thus, the authors undertook a research project to co-design a product that helps users integrate quality and implementation outcomes and strengthen existing evaluation processes in the provincial health system.\u003c/p\u003e \u003cp\u003eAims\u003c/p\u003e \u003cp\u003eIn this paper, we describe the EQUIP (Evaluating QUality and ImPlementation) Tool, an online evaluation tool that integrates quality and implementation outcomes. This tool was co-designed with a team of content experts and health system partners and then tested and refined with a broader group of potential end-users (i.e., people who would use the tool) to create the final product.\u003c/p\u003e \u003cp\u003eThe questions used to guide the research project were:\u003c/p\u003e \u003cp\u003e1. How might we co-design an evaluation tool that brings together the Alberta Quality Matrix for Health with Proctor et al.’s taxonomy of implementation outcomes?\u003c/p\u003e \u003cp\u003e2. How might we test and refine the tool so that it meets the needs of intended users?\u003c/p\u003e \u003cp\u003e3. How might we ensure the tool is accessible and valuable to users and the work they do?\u003c/p\u003e \u003cp\u003eBy answering these questions, we aimed to support researchers, funders, and practitioners working in health service delivery organizations who want to, directly or indirectly, strengthen implementation evaluation capacity and establish routine evaluation of implementation outcomes in health research studies and practice change initiatives.\u003c/p\u003e \u003cp\u003eContext\u003c/p\u003e \u003cp\u003eThis project took place in Alberta, Canada, within the context of Alberta Health Services, the single health authority in the province that provides programs and services at more than 900 facilities throughout the province, including hospitals, clinics, continuing care facilities, cancer centres, mental health facilities, and community health sites.\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe Alberta Quality Matrix for Health is very well recognized and frequently used in evaluations taking place in Alberta Health Services (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). It is designed to assess patient outcomes and quality in a standardized way across the complex healthcare system.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e However, the matrix is missing measures of implementation, which are essential to produce robust evaluations following implementation of initiatives.\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDefinitions of quality dimensions and implementation outcomes\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDimension or Outcome\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDefinition\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eQuality Dimensions (from the Alberta Quality Matrix for Health)\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAcceptability\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHealth services are respectful and responsive to user needs, preferences and expectations\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccessibility\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHealth services are obtained in the most suitable setting in a reasonable time and distance\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAppropriateness\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHealth services are relevant to user needs and are based on accepted or evidence-based practice\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEffectiveness\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHealth services are based on scientific knowledge to achieve desired outcomes\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEfficiency\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResources are optimally used in achieving desired outcomes\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSafety\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMitigate risks to avoid unintended or harmful results\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eImplementation Outcomes (from Proctor et al.’s Taxonomy)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAcceptability\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerception among implementation stakeholders that a treatment, service, practice or innovation is agreeable, palatable, or satisfactory\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdoption\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIntention, initial decision, or action to try or employ an innovation or evidence-based practice\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAppropriateness\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePerceived fit, relevance, or compatibility of the innovation or evidence-based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation to address a particular issue or problem\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFeasibility\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExtent to which a new treatment, or an innovation, can be successfully used or carried out within a given agency or setting\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFidelity\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDegree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the program developers\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCost\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCost impact of an implementation effort\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePenetration\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIntegration of a practice within a service setting and its subsystems\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSustainability\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExtent to which a newly implemented treatment is maintained or institutionalized with a service setting’s ongoing, stable operations\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eIn the field of implementation science, a commonly used outcomes framework developed by Proctor and colleagues articulates eight implementation outcomes (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), which precede but are interrelated with service and client outcomes.\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e The service outcomes, which are based on the six quality improvement aims from the Institute of Medicine,\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e complement the quality dimensions of the Alberta Quality Matrix for Health. Thus, integrating quality measures from the Alberta Quality Matrix for Health and implementation measures from Proctor et al. is a natural fit for the development of an evaluation tool that integrates strengths of both quality improvement and implementation science.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eWe report the development of the EQUIP tool in accordance with the 'Guidance for reporting intervention development studies in health research' (GUIDED) checklist.\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e See Additional File 1 for the completed checklist. The design and usability stages of the project took place over eighteen months, from January 2020 to June 2021, with monitoring and further refinement throughout that time.\u003c/p\u003e\u003cp\u003eCreation of the Co-Design Team\u003c/p\u003e\u003cp\u003eDuring an initial brainstorming session, AbSPORU staff identified potential co-design team members based on pre-existing relationships with stakeholders from Alberta Health Services. The aim was to have at least three members from different areas of Alberta Health Services and three members from AbSPORU. Team members (n = 6) consisted of end-users, provincial health system partners from Alberta Health Services, AbSPORU and specialists in evaluation, implementation science, and design. Some team members were connected to more than one of these roles, and all team members were based in Alberta, Canada.\u003c/p\u003e\u003cp\u003eDesign\u003c/p\u003e\u003cp\u003eThe tool was developed using a co-design approach to incorporate end users’ perspectives in the product design. A combination of the Successive Approximation Model and Design Thinking was used to guide the process.\u003csup\u003e\u003cspan additionalcitationids=\"CR21\" citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e–\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e The Successive Approximation Model, from the field of instructional design, outlines an iterative, participatory design and development process that focuses on end users’ experiences, engagement, and motivation.\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e Design Thinking is a human-centered, solution-based approach to design, which encourages teams to focus on the users and their contexts. It accomplishes this by drawing on methods targeted at gaining a deep understanding of users’ needs, experiences, and desires from their perspective to develop solutions that are effective and accessible.\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e,\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eThe co-design team first developed a prototype through rapid, iterative, virtual design sessions held over a video conferencing platform, Zoom (Zoom Video Communications, Inc.). At the first design session, attendees discussed potential users and their contexts, formulated a “how might we” question to guide design and development, brainstormed possible solutions, and began prototyping. A voting exercise was held at the end of the session to select one prototype for further development. A second design session was held a few weeks later to revisit discussions held during the first session and confirm selection of the prototype. The team’s graphic designer (CR) then developed an online, interactive prototype that was further refined through usability testing with targeted user groups.\u003c/p\u003e\u003cp\u003eUsability\u003c/p\u003e\u003cp\u003eUsability testing was conducted using two different online questionnaires, administered using Google Forms: one on the tool’s functionality and another on the design and accessibility of the tool itself. See Additional File 2 for the complete questionnaires. The first questionnaire consisted of several tasks for users to work through to ensure they could access, navigate, and interact with the tool. The second questionnaire asked for feedback on the ease of use, relevance of the tool to their work, likelihood of recommending the tool, along with targeted questions on what to improve (and how), and what users liked about the tool. Data was collected from yes/no responses, multiple choice responses, Likert scales, and open-ended answers (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). A group of potential end-users were recruited through existing relationships with co-design team members to complete the usability testing.\u003c/p\u003e\u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eUsability testing questionnaire topics and example questions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQuestionnaire Topics\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExample Question\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQuestion Style\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUsability Questionnaire\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccessibility\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask 1 - Please go to the evaluation tool website by clicking on the link below. Are you able to access the website using your preferred web browser?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eYes/No\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNavigation\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask 3 - Please click on Use the Tool at the top of the page. Did the content on the Use the Tool page load properly?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eYes/No\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInteraction\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask 5 - Scroll down a little further to Healthcare Providers, find Feasibility (one of the blue boxes), and click on three buttons: Feasibility, Sample Implementation Question, and Sample Indicators. In the list of Sample Indicators, click on Feasibility of Intervention Measure and Technology Acceptance Survey. When you clicked on the Feasibility of Intervention Measure, did a new window open?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMultiple choice\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eFeedback Questionnaire\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEase of use\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow difficult was it to use and understand the tool?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5-point Likert\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRelevance\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDo you think this tool could be helpful for your evaluations?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMultiple choice\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGeneral feedback\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDo you have any suggestions on how we can improve the instructions?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOpen-ended\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLikelihood of recommending\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow likely are you to recommend the tool to others?\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5-point Likert\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eUsability feedback was collected and analyzed using descriptive statistics in Microsoft Excel. Adjustments were made to the prototype in an ongoing manner. Regular, live meetings with two lead team members (LM and CR) were held over Zoom every six to eight weeks to discuss results, identify improvements, and update the tool prototype (and accompanying website), increasing to every two weeks near the end of testing.\u003c/p\u003e\u003cp\u003eIn-depth consultations were conducted with a subset of usability testing respondents (n = 8) to discuss their usability testing experience, review resulting updates to the prototype, and better understand the tool’s relevance to their work. All consultations were carried out over Zoom over 6 months (December 2020 to June 2021) and ran from 30 to 45 minutes in length. All respondents were asked the same set of three questions during the consultations:\u003c/p\u003e\u003cp\u003e1. What do you think of the updates to the website, especially the overall design and function?\u003c/p\u003e\u003cp\u003e2. How might the tool help your work or the work of others on your team?\u003c/p\u003e\u003cp\u003e3. What improvements could we make to the tool and website to support this?\u003c/p\u003e\u003cp\u003eAs with the questionnaire responses, key points from the consultations were compiled by two lead team members (LM and CR) and used to refine the prototype and accompanying website.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eCo-Design and Initial Prototype\u003c/p\u003e \u003cp\u003eDuring the first design session, the team identified both researchers and practitioners in the health system as potential users of the tool, with practitioners encompassing community and acute care settings, clinical and support staff. It was felt that there was energy and interest among clinicians and managers around improvement, and that a tool like this could support improvement efforts, develop better evaluations, and demonstrate impact. It was also suggested that a tool like this could help groups applying for implementation science-focused funding opportunities. These often require applicants to incorporate implementation considerations in their studies, including implementation outcomes, which is often new to research teams. The tool needed to be user-friendly and easy to apply. It could not be too technical, theoretical or make peoples\u0026rsquo; work more difficult. To achieve this, team members agreed early on that an online, web-based format that would enable users to select what they need and create a customized form would be ideal.\u003c/p\u003e \u003cp\u003eThe initial prototype after the first design session was a matrix that mirrored the Alberta Quality Matrix for Health, with the Dimensions of Health Service Quality across the top, and the different types of implementation research outcomes (client, service and implementation) from Enola Proctor\u0026rsquo;s paper listed down the side. However, team members recognized that this initial prototype was at odds with users\u0026rsquo; needs. It took too much time to understand and navigate the sizeable table.\u003c/p\u003e \u003cp\u003eAfter the second design session, the table was refined to include the six Dimensions of Health Service Quality and Enola Proctor\u0026rsquo;s Taxonomy of Implementation Outcomes across the top, and various key perspectives listed down the side. Two of the six Alberta Quality Matrix for Health dimensions overlap with Proctor et al.\u0026rsquo;s recommended taxonomy of implementation outcomes (in name only): Acceptability and Appropriateness.\u003c/p\u003e \u003cp\u003eThis overlap between the two frameworks helped to connect them because the importance of assessing Acceptability and Appropriateness is already well established in practice. However, Proctor et al.\u0026rsquo;s outcomes focus mostly on the perspectives of healthcare providers and the healthcare setting while the Alberta Quality Matrix for Health has a major focus on the patient experience. The co-design team felt it was important to consider several perspectives in an evaluation. The Interactive Systems Framework and Alberta Health Services\u0026rsquo; use of the Quadruple Aim informed how to build the tool in light of this initial session and the question of which perspectives should be included in the final tool.\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e,\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e Team members also suggested that sample questions would be helpful to include. A frequent comment during the design sessions was that \u0026lsquo;people don\u0026rsquo;t know what they don\u0026rsquo;t know\u0026rsquo;. This started discussions around how this tool could prompt thinking and build capacity of the users.\u003c/p\u003e \u003cp\u003eWhile the matrix demonstrated the integration of the Alberta Quality Matrix for Health dimensions and the implementation outcomes, due to coding complications it would not generate the form as intended. Therefore, the team\u0026rsquo;s graphic designer developed this concept into an online slide-box format. A comparison between the two layouts (matrix vs slide box) was included in usability testing (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe iterative development process allowed the team to discuss and consider other tools and frameworks to adapt the prototype to better fit the local context. These included the British Columbia Health Quality Matrix and the RE-AIM framework (RE-AIM stands for Reach, Effectiveness, Adoption, Implementation, and Maintenance).\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e A member of the co-design team highlighted that the British Columbia Health Quality Matrix includes the dimension \u0026lsquo;Equity\u0026rsquo;, which had not yet made its way into the Alberta Quality Matrix for Health but is increasingly important to consider.\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e The British Columbia Health Quality Matrix also does a good job of defining the quality dimensions and indicators in a straightforward way, which the team felt was important. The RE-AIM Framework was identified by the co-design team as being widely used and well understood in the Alberta health system, therefore some of the language, definitions and indicators for the implementation outcomes of the EQUIP tool were adapted from the RE-AIM framework (e.g., Reach instead of Penetration).\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eUsability Testing\u003c/p\u003e \u003cp\u003eUsability testing of the prototype ran from November 2020 to May 2021.\u003c/p\u003e \u003cp\u003eImprovements were made to the first testable prototype iteratively based on ongoing usability testing and feedback results. For example, sample questions and indicators were added, others were removed, new pages were added to the accompanying website, and existing pages were overhauled. Although user responses varied during testing, most asked that the tool and website be kept as straightforward and applicable as possible.\u003c/p\u003e \u003cp\u003eTwenty-one people completed usability testing in three separate waves over the course of seven months. The majority of users came from either the healthcare system (57%) or academia (35%). Two users were evaluation specialists for community or health policy organizations (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). All users had experience in either quality improvement, evaluation or implementation science and the majority had knowledge of the Alberta Quality Matrix for Health.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAll users (n\u0026thinsp;=\u0026thinsp;21) completed all the tasks outlined in the usability questionnaire, and all users thought the tool could help with their work, while 81% (n\u0026thinsp;=\u0026thinsp;17) agreed that the included perspectives (patients, healthcare providers, support teams, and organizations) were thorough enough. Three respondents thought key perspectives were missing, including that of funders and care partners. The remaining respondent thought too many perspectives were already included. Overall, users\u0026rsquo; responses were positive and helpful for informing updates to the prototype (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Key suggestions for improvement included: tightening up the language, improving the layout of information on each page, and including an example evaluation. The overall structure and visuals as well as sample questions and indicators were features that users felt were done well and should not be changed. A majority of users (62%) said that working through the prototype prompted them to start thinking about including implementation outcomes in future evaluations.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eInitially the tool included five broad categories of perspectives and each perspective included all possible quality dimensions and implementation outcomes. Iterative feedback from the co-design team and usability testing respondents helped streamline which perspectives to include and which dimensions and outcomes were most relevant for each perspective. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e provides an example of how the tool changed after feedback from users and co-design team members was incorporated.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eCo-design and usability feedback helped to resolve inconsistencies in language in ways that help to integrate quality and implementation outcomes. For example, Appropriateness and Acceptability have different definitions in the Alberta Quality Matrix for Health and Proctor\u0026rsquo;s implementation outcomes, but the Quality definition was felt to suit the patient perspective, while the implementation definition suited providers. There was also some discussion about the difference between Appropriateness and Acceptability as they are conceptually similar which can cause confusion, but referring to the examples in Proctor\u0026rsquo;s paper provided clarity.\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eUsability testing significantly influenced the design and aesthetic of the tool. The initial prototype of the tool incorporated the colours and structure of the Alberta Quality Matrix for Health, which was originally included to help users by drawing on their familiarity with this matrix. Although some users resonated with the colours, others found it confusing and irrelevant. Content and design elements from the British Columbia Health Quality Matrix were then incorporated in response to this feedback. In further testing, it was found that these elements resonated better with users and made the tool feel simpler to use.\u003c/p\u003e \u003cp\u003eFinal Testing\u003c/p\u003e \u003cp\u003eEight consultations were completed with a subset of the respondents involved in usability testing to review updates and understand the relevance of the EQUIP Tool to their work. Of the eight, three were with implementation science networks within Alberta Health Services, three worked with other groups in Alberta Health Services, one was a health policy researcher, and one worked with the Health Quality Council of Alberta.\u003c/p\u003e \u003cp\u003eConsultations were conducted over the course of 6 months with improvements made iteratively such that the conversations that happened later had fewer suggestions for improvement and specifically mentioned things like:\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003eEasy to navigate\u003c/p\u003e\u003cp\u003eClicking on the boxes was easy to understand but makes site clean\u0026hellip; Text only when you want to see it\u003c/p\u003e\u003cp\u003eOverall, thought tool was great, usability navigation, visuals, examples, easy to use\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eImprovements included aesthetic updates to the tool prototype and accompanying website, simplifying the instructions for the tool, and adding links to validated measures directly within the tool.\u003c/p\u003e \u003cp\u003eThe overall response from users was very positive. Updates were well received, to both the tool and the accompanying website. Some users described how they were already using the tool to build evaluation capacity in their teams and inform evaluations of innovations in the health system. Users also shared how they already used the EQUIP tool in their work, including collaboratively designing evaluations, selecting standardized implementation measures, and informing an upcoming refresh of the Alberta Quality Matrix for Health.\u003c/p\u003e \u003cp\u003eThe Final Tool: The EQUIP (Evaluating QUality and ImPlementation) Tool (theequiptool.com)\u003c/p\u003e \u003cp\u003eThe final tool is housed on a website (theequiptool.com) that includes information about what the tool is, who the intended users are, and why it is important. There are interactive visuals that provide definitions of the dimensions of health service quality outlined in the Alberta Quality Matrix for Health, Proctor\u0026rsquo;s recommended list of implementation outcomes, and the different perspectives included. A resource page includes a link to a PDF version of the tool as well as an example.\u003c/p\u003e \u003cp\u003eThe tool itself is interactive, allowing users to explore the implementation outcomes and quality dimensions from the different perspectives and select those questions and indicators that are relevant to their project. After completing selections, users are able to save or print a hard copy record of them. Each outcome is defined and includes sample questions from either a quality perspective or implementation science perspective or both (for those domains represented by both Acceptability and Appropriateness). Examples of indicators and validated measures are included in the tool. Sample indicators and links to measures were pulled from several different evaluation frameworks used within Alberta Health Services as well as systematic reviews and scoping reviews that consider Proctor\u0026rsquo;s list of implementation outcomes.\u003csup\u003e\u003cspan additionalcitationids=\"CR27\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e The tool is meant to stimulate thinking and discussions. Therefore, any of the questions or indicators may need to be further developed by the user to suit their needs.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThrough rapid, iterative design sessions and usability testing, an interactive evaluation tool was developed that incorporates both quality and implementation outcomes. A co-design approach was used to ensure that the tool developed would be relevant and practical for end-users.\u003csup\u003e29(p)\u003c/sup\u003e The result is an interactive evaluation tool called EQUIP (Evaluating QUality and ImPlementation), available at theequiptool.com.\u003c/p\u003e \u003cp\u003eAn important feature of the tool is that it highlights the different perspectives that should be considered when evaluating implementation efforts. While improving patient outcomes is the ultimate goal of most quality improvement and implementation efforts, it is also important to consider the impact of a given implementation on the people and systems involved. In fact, all of these perspectives are needed to understand why implementation efforts succeed or fail and to determine what constitutes success in the first place.\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e,\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe perspectives included in the EQUIP Tool take into account the various roles involved in implementation and the broader context. They are based on the Interactive Systems Framework and Alberta Health Services use of the Quadruple Aim and refined by feedback from users.\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e,\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e The Interactive Systems Framework centers on the people, support and organizations that are needed to carry out the activities required for successful implementation.\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e The Quadruple Aim highlights not only patient experience but provider satisfaction in how they are supported in providing care.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eProviders are especially important as they hold a high level of influence in the system.\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e They are largely responsible for delivering products and services to patients while also being accountable to the systems within which they work. In fact, all of the implementation outcomes and several quality dimensions are considered from the provider perspective in the EQUIP tool. Without considering the whole context within which an intervention is being implemented, important evaluation questions could be missed. The EQUIP tool encourages users to consider important quality and implementation questions from each of these key perspectives. While the level of analysis for implementation outcomes has not been well studied,\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e the EQUIP tool provides an opportunity to explore if a given level or perspective is appropriate for a particular implementation outcome.\u003c/p\u003e \u003cp\u003eThe consultations provided an opportunity to delve deeper into different frameworks and consider other definitions and perspectives. As Reilly and colleagues point out, there is a great deal of consistency between RE-AIM and Proctor\u0026rsquo;s implementation outcomes.\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e However, until recently RE-AIM did not include Acceptability, Appropriateness, and Feasibility (which they include as antecedents of several implementation outcomes in their proposed expansion to RE-AIM indicators).\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e Different frameworks have different definitions for the various outcomes, therefore, ongoing discussions with users helped to find the right fit for the different perspectives in the tool. Similarly, Weiner et al\u003csup\u003e\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e noted that measures are rarely defined or distinguishable from one another, resulting in confusion around what is being assessed. Therefore, having sample questions and indicators in the tool helps with understanding the different dimensions and outcomes and having links to validated measures improves certainty.\u003c/p\u003e \u003cp\u003eThe engagement with the co-design team and potential end-users was an important part of ensuring the tool was relevant and responsive to users\u0026rsquo; needs. The result is a tool that some stakeholders are already using for training (e.g., to better understand the quality dimensions and implementation outcomes), to stimulate thinking (e.g., about the different perspectives involved), and develop evaluations. The tool has also since been included as a resource in the updated Alberta Health Services Innovation Pipeline Primer 2.0.\u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e The Innovation Pipeline Primer 2.0 provides information about the evidence needed to safely move a good innovative idea into action, including sample outcomes. Thus, questions and indicators from EQUIP help inform the evidence for quality improvement and implementation effectiveness. Further use of the tool and having a place for feedback built into the tool will assist with future improvements.\u003c/p\u003e \u003cp\u003eStrengths and Limitations\u003c/p\u003e \u003cp\u003eThe co-design process and usability testing were strengths that ensured the end product would be relevant to end-users. Although several frameworks were discussed, there may be others that were not considered but could have added value. However, the frameworks included were those that were familiar to end-users in Alberta. Depending on the scope of a given evaluation, certain perspectives may be missing and should be considered if relevant to that evaluation. Patients were not consulted when selecting the outcome definitions or indicators for the patient perspective. This appears to be a limitation of most frameworks. However, the Alberta Quality Matrix for Health is currently being refreshed and is involving patient family advisors [personal communication]. Future updates of the EQUIP tool will consider any important changes to outcomes or indicators for patients (by patients).\u003c/p\u003e \u003cp\u003eThe EQUIP tool was designed to help users incorporate implementation outcomes into quality evaluations. It can help with buy-in, build capacity, consider different perspectives, provide sample questions and indicators, and connect users to validated measures. The tool was not developed to identify stakeholders, assess readiness, build entire evaluation plans, or guide implementation planning.\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003eThe EQUIP tool was co-designed and refined with end users to create an accessible and useful online tool. This interactive tool helps people build robust evaluations that incorporate the six dimensions of health service quality, outlined in the Alberta Quality Matrix for Health, and standardized implementation outcomes. This work is an example of integrating quality and implementation science in ways that strengthen overall healthcare quality improvements.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e\u003cstrong\u003eAbSPORU \u0026ndash;\u0026nbsp;\u003c/strong\u003eAlberta Strategy for Patient-Oriented Research SUPPORT Unit\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEQUIP \u0026ndash;\u0026nbsp;\u003c/strong\u003eEvaluating Quality and Implementation\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRE-AIM \u0026ndash;\u0026nbsp;\u003c/strong\u003eReach, Effectiveness, Adoption, Implementation, Maintenance\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch3\u003eEthics approval and consent to participate\u003c/h3\u003e\n\u003cp\u003eEthical approval was obtained from the University of Alberta Research Ethics Board 2 (Pro00130141). All participants provided informed consent.\u003c/p\u003e\n\u003ch3\u003eConsent for publication\u003c/h3\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003ch3\u003eAvailability of data and materials\u003c/h3\u003e\n\u003cp\u003eAll data generated or analysed are not available due to participant privacy but are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003ch3\u003eCompeting interests\u003c/h3\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003ch3\u003eFunding\u003c/h3\u003e\n\u003cp\u003eThis work was led by the Alberta SPOR SUPPORT Unit (AbSPORU), which is co-funded by the Strategy for Patient-Oriented Research program of the Canadian Institute for Health Research (CIHR), Alberta Innovates\u003c/p\u003e\n\u003cp\u003eand the University Hospital Foundation. AbSPORU also acknowledges its implementation partners: the University of Alberta, the University of Calgary, the University of Lethbridge, Alberta Health Services, Athabasca University, the Women and Children's Health Research Institute, the Alberta Children's Hospital Research Institute and Alberta Health.\u003c/p\u003e\n\u003ch3\u003eAuthors' contributions\u003c/h3\u003e\n\u003cp\u003eLM and GZ conceptualized the project. LM and CR led the collaboration with the tool development team: LM designed the development process, and CR led the design process. GZ supervised the overall project. LM, CR, EF, NP, EK, and GZ designed and developed the EQUIP Tool. LM led data collection and analysis. LM and GZ drafted the manuscript. CR, EF, NP, and EK reviewed and provided input on numerous versions of the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003ch3\u003eAcknowledgements\u003c/h3\u003e\n\u003cp\u003eThank you, Anna Noga and Morgan Potter for contributing to early design sessions, Cody Alba for reviewing an early draft of the manuscript, and Stephanie Brooks and Denise Thomson for reviewing and revising manuscript drafts. This work was led by the AbSPORU. AbSPORU also acknowledges its implementation partners: the University of Alberta, the University of Calgary, the University of Lethbridge, Alberta Health Services, Athabasca University, the Women and Children's Health Research Institute, the Alberta Children's Hospital Research Institute and Alberta Health.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eGlasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. \u003cem\u003eFront Public Health\u003c/em\u003e. 2019;7:64. doi:10.3389/fpubh.2019.00064\u003c/li\u003e\n\u003cli\u003eTabin M, Diacquenod C, Petitpierre G. Evaluating implementation outcomes of a measure of social vulnerability in adults with intellectual disabilities. \u003cem\u003eRes Dev Disabil\u003c/em\u003e. 2021;119:104111. doi:10.1016/j.ridd.2021.104111\u003c/li\u003e\n\u003cli\u003eYiu V, Belanger F, Todd K. Alberta\u0026rsquo;s Strategic Clinical Networks: Enabling health system innovation and improvement. \u003cem\u003eCan Med Assoc J\u003c/em\u003e. 2019;191(Suppl):S1-S3. doi:10.1503/cmaj.191232\u003c/li\u003e\n\u003cli\u003eRapport F, Smith J, Hutchinson K, et al. Too much theory and not enough practice? The challenge of implementation science application in healthcare practice. \u003cem\u003eJ Eval Clin Pract\u003c/em\u003e. Published online June 30, 2021. doi:10.1111/jep.13600\u003c/li\u003e\n\u003cli\u003eMoullin JC, Dickson KS, Stadnick NA, et al. Ten recommendations for using implementation frameworks in research and practice. \u003cem\u003eImplement Sci Commun\u003c/em\u003e. 2020;1:42. doi:10.1186/s43058-020-00023-7\u003c/li\u003e\n\u003cli\u003eProctor E, Silmere H, Raghavan R, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. \u003cem\u003eAdm Policy Ment Health\u003c/em\u003e. 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7\u003c/li\u003e\n\u003cli\u003eQuinn Patton M. \u003cem\u003eEssentials of Utilization-Focused Evaluation\u003c/em\u003e. 1st ed. SAGE Publications, Inc; 2011. Accessed November 15, 2021. https://us.sagepub.com/en-us/nam/essentials-of-utilization-focused-evaluation/book233973\u003c/li\u003e\n\u003cli\u003eBeyea SC, Nicoll LH. Is it research or quality improvement? \u003cem\u003eAORN J\u003c/em\u003e. 1998;68(1):117-119. doi:10.1016/S0001-2092(06)62732-4\u003c/li\u003e\n\u003cli\u003eReinhardt AC, Ray LN. Differentiating quality improvement from research. \u003cem\u003eAppl Nurs Res\u003c/em\u003e. 2003;16(1):2-8. doi:10.1053/apnr.2003.59000\u003c/li\u003e\n\u003cli\u003eMaritz R, Scheel-Sailer A, Schmitt K, Prodinger B. Overview of quality management models for inpatient healthcare settings. A scoping review. \u003cem\u003eInt J Qual Health Care\u003c/em\u003e. 2019;31(6):404-410. doi:10.1093/intqhc/mzy180\u003c/li\u003e\n\u003cli\u003eHealth Quality Council of Alberta. Alberta Quality Matrix for Health User Guide. Published online 2005. Accessed September 29, 2021. https://hqca.ca/about/how-we-work-health-quality-council-of-alberta/the-alberta-quality-matrix-for-health-1/\u003c/li\u003e\n\u003cli\u003eBC Patient Safety \u0026amp; Quality Council. BC Health Quality Matrix. Published online 2020. Accessed August 24, 2021. https://bcpsqc.ca/wp-content/uploads/2020/02/BC-Health-Quality-Matrix-March-2020.pdf\u003c/li\u003e\n\u003cli\u003eGlasgow RE, Eckstein ET, ElZarrad MK. Implementation Science Perspectives and Opportunities for HIV/AIDS Research: Integrating Science, Practice, and Policy. \u003cem\u003eJAIDS J Acquir Immune Defic Syndr\u003c/em\u003e. 2013;63:S26. doi:10.1097/QAI.0b013e3182920286\u003c/li\u003e\n\u003cli\u003eMoore JE. 5 ways to improve Implementation Science \u0026ndash; lessons from quality improvement. The Center for Implementation. Published December 2019. Accessed November 15, 2021. https://thecenterforimplementation.com/implementation-in-action-bulletin/december-2019\u003c/li\u003e\n\u003cli\u003eTyler A, Glasgow RE. Implementing Improvements: Opportunities to Integrate Quality Improvement and Implementation Science. \u003cem\u003eHosp Pediatr\u003c/em\u003e. 2021;11(5):536-545. doi:10.1542/hpeds.2020-002246\u003c/li\u003e\n\u003cli\u003eServices AH. About AHS. Alberta Health Services. Accessed May 19, 2022. https://www.albertahealthservices.ca/about/about.aspx\u003c/li\u003e\n\u003cli\u003eWensing M, Grol R. Knowledge translation in health: how implementation science could contribute more. \u003cem\u003eBMC Med\u003c/em\u003e. 2019;17(1):88. doi:10.1186/s12916-019-1322-9\u003c/li\u003e\n\u003cli\u003eWolfe A. Institute of Medicine Report: Crossing the Quality Chasm: A New Health Care System for the 21st Century. \u003cem\u003ePolicy Polit Nurs Pract\u003c/em\u003e. 2001;2(3):233-235. doi:10.1177/152715440100200312\u003c/li\u003e\n\u003cli\u003eDuncan E, O\u0026rsquo;Cathain A, Rousseau N, et al. Guidance for reporting intervention development studies in health research (GUIDED): an evidence-based consensus study. \u003cem\u003eBMJ Open\u003c/em\u003e. 2020;10(4):e033516. doi:10.1136/bmjopen-2019-033516\u003c/li\u003e\n\u003cli\u003eReiser RA, Dempsey JV. \u003cem\u003eTrends and Issues in Instructional Design and Technology.\u003c/em\u003e Fourth edition. Pearson; 2018. https://login.ezproxy.library.ualberta.ca/login?url=https://search.ebscohost.com/login.aspx?direct=true\u0026amp;db=cat03710a\u0026amp;AN=alb.7870354\u0026amp;site=eds-live\u0026amp;scope=site\u003c/li\u003e\n\u003cli\u003eDoorley S, Holcomb S, Klebahn P, Segovia K, Utley J. Design Thinking Bootleg. Published online 2018. Accessed August 25, 2021. https://static1.squarespace.com/static/57c6b79629687fde090a0fdd/t/5b19b2f2aa4a99e99b26b6bb/1528410876119/dschool_bootleg_deck_2018_final_sm+%282%29.pdf\u003c/li\u003e\n\u003cli\u003eDesign Thinking Frequently Asked Questions (FAQ). IDEO | Design Thinking. Published November 12, 2021. Accessed November 12, 2021. https://designthinking.ideo.com/faq/whats-the-difference-between-human-centered-design-and-design-thinking\u003c/li\u003e\n\u003cli\u003eWandersman A, Duffy J, Flaspohler P, et al. Bridging the Gap Between Prevention Research and Practice: The Interactive Systems Framework for Dissemination and Implementation. \u003cem\u003eAm J Community Psychol\u003c/em\u003e. 2008;41(3-4):171-181. doi:10.1007/s10464-008-9174-z\u003c/li\u003e\n\u003cli\u003eAlberta Health Services. Enhancing Care in the Community. Alberta Health Services. Published August 25, 2021. Accessed August 25, 2021. https://www.albertahealthservices.ca/about/Page13457.aspx\u003c/li\u003e\n\u003cli\u003eGlasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. \u003cem\u003eFront Public Health\u003c/em\u003e. 2019;7:64. doi:10.3389/fpubh.2019.00064\u003c/li\u003e\n\u003cli\u003eMettert K, Lewis C, Dorsey C, Halko H, Weiner B. Measuring implementation outcomes: An updated systematic review of measures\u0026rsquo; psychometric properties. \u003cem\u003eImplement Res Pract\u003c/em\u003e. 2020;1:2633489520936644. doi:10.1177/2633489520936644\u003c/li\u003e\n\u003cli\u003eKhadjesari Z, Boufkhed S, Vitoratou S, et al. Implementation outcome instruments for use in physical healthcare settings: a systematic review. \u003cem\u003eImplement Sci\u003c/em\u003e. 2020;15(1):66. doi:10.1186/s13012-020-01027-6\u003c/li\u003e\n\u003cli\u003eWillmeroth T, Wesselborg B, Kuske S. Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review. \u003cem\u003eInq J Health Care Organ Provis Financ\u003c/em\u003e. 2019;56:0046958019861257. doi:10.1177/0046958019861257\u003c/li\u003e\n\u003cli\u003eLangley J, Wolstenholme D, Cooke J. \u0026lsquo;Collective making\u0026rsquo; as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. \u003cem\u003eBMC Health Serv Res\u003c/em\u003e. 2018;18:585. doi:10.1186/s12913-018-3397-y\u003c/li\u003e\n\u003cli\u003ePereno A, Eriksson D. A multi-stakeholder perspective on sustainable healthcare: From 2030 onwards. \u003cem\u003eFutures\u003c/em\u003e. 2020;122:102605. doi:10.1016/j.futures.2020.102605\u003c/li\u003e\n\u003cli\u003eReilly KL, Kennedy S, Porter G, Estabrooks P. Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks. \u003cem\u003eFront Public Health\u003c/em\u003e. 2020;8. Accessed May 19, 2022. https://www.frontiersin.org/article/10.3389/fpubh.2020.00430\u003c/li\u003e\n\u003cli\u003eWeiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. \u003cem\u003eImplement Sci\u003c/em\u003e. 2017;12(1):108. doi:10.1186/s13012-017-0635-3\u003c/li\u003e\n\u003cli\u003eWaye A, Hughes B, Mrklas K, Fraser N. Innovation Pipeline: Intent to Scale for Impact. Published online 2020. Accessed August 24, 2021. https://www.albertahealthservices.ca/assets/about/scn/ahs-scn-so-innov-pipeline-primer.pdf\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"implementation-science-communications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"iscm","sideBox":"Learn more about [Implementation Science Communications](https://implementationsciencecomms.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ISCM/default.aspx","title":"Implementation Science Communications","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Evaluation, quality improvement, implementation science, implementation outcomes, usability testing","lastPublishedDoi":"10.21203/rs.3.rs-4572444/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4572444/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground: \u003c/strong\u003eThis article describes the development of an interactive online tool that helps people build robust evaluations. Evaluating implementation outcomes is gaining momentum in health service delivery organizations. Teams are recognizing the importance of capturing and learning from their implementation efforts, and implementation scientists have published extensively on implementation outcomes. However, quality improvement approaches and tools are better known and routinely used in healthcare to improve processes and outcomes.\u003c/p\u003e\n\u003cp\u003eAn example of a widespread quality improvement tool is the Alberta Quality Matrix for Health, a planning and evaluation framework that guides most healthcare-related evaluations in Alberta, Canada. This matrix is missing measures of implementation, which are essential to produce robust evaluations of change initiatives. Despite requests to integrate implementation science and quality improvement concepts, there is a dearth of tools and training opportunities. This lack of integration limits improvement and implementation in the health system.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eThis work aimed to co-design a product that helps users integrate quality and implementation outcomes and strengthen existing evaluation processes. Rapid, iterative co-design sessions and usability testing were used to develop an interactive evaluation tool. Feedback collected and analyzed from usability testing in the seven-month testing period refined the digital prototype into a comprehensive online product.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eThe result is the EQUIP (Evaluating QUality and ImPlementation) Tool. This online, interactive tool integrates quality measures from the Alberta Quality Matrix for Health and implementation measures from a commonly used outcomes framework developed by Proctor and colleagues. It allows users to explore implementation outcomes and quality dimensions from different perspectives and select questions and indicators relevant to their project.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion: \u003c/strong\u003eThe EQUIP tool was co-designed and refined with end users to create an accessible and useful online tool. This work is an example of integrating quality and implementation science in ways that strengthen overall healthcare quality improvements.\u003c/p\u003e","manuscriptTitle":"Development and Testing of an Interactive Evaluation Tool: The Evaluating QUality and ImPlementation (EQUIP) Tool","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-08 16:44:41","doi":"10.21203/rs.3.rs-4572444/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2024-09-09T22:52:29+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"","date":"2024-07-25T16:46:25+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-19T16:31:42+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-06-14T08:44:05+00:00","index":"","fulltext":""},{"type":"submitted","content":"Implementation Science Communications","date":"2024-06-13T18:24:58+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"implementation-science-communications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"iscm","sideBox":"Learn more about [Implementation Science Communications](https://implementationsciencecomms.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ISCM/default.aspx","title":"Implementation Science Communications","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"51621b87-8701-4230-9843-adb991ead595","owner":[],"postedDate":"July 8th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-04-07T16:07:16+00:00","versionOfRecord":{"articleIdentity":"rs-4572444","link":"https://doi.org/10.1186/s43058-025-00715-y","journal":{"identity":"implementation-science-communications","isVorOnly":false,"title":"Implementation Science Communications"},"publishedOn":"2025-03-31 15:57:23","publishedOnDateReadable":"March 31st, 2025"},"versionCreatedAt":"2024-07-08 16:44:41","video":"","vorDoi":"10.1186/s43058-025-00715-y","vorDoiUrl":"https://doi.org/10.1186/s43058-025-00715-y","workflowStages":[]},"version":"v1","identity":"rs-4572444","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4572444","identity":"rs-4572444","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}