Implementing a rapid-learning health system in early intervention services for psychosis: Qualitative evaluation of its feasibility and acceptability

Implementing a rapid-learning health system in early intervention services for psychosis: Qualitative evaluation of its feasibility and acceptability | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Implementing a rapid-learning health system in early intervention services for psychosis: Qualitative evaluation of its feasibility and acceptability Manuela Ferrari, Marianne-Sarah Saulnier, Srividya N. Iyer, Marc-André Roy, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4216686/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 14 Mar, 2025 Read the published version in Health Research Policy and Systems → Version 1 posted 12 You are reading this latest preprint version Abstract Background: The early intervention service for psychosis (EIS) model is increasingly implemented. Yet, despite the development of guidelines and standards for EIS in several countries, heterogeneity persists in implementing the essential evidence-based practices. One remedy to these implementation challenges entails the use of a rapid-learning health system (RLHS), an approach that involves ongoing, systematic data collection, analysis and visualization using an electronic platform to design capacity-building activities and promote learning. In 2019, our team implemented SARPEP ( Système Apprenant Rapide pour les Programmes de Premiers Épisodes Psychotiques ), the first Canadian RLHS in EIS. This paper presents findings from the qualitative phase of the mixed-method study that evaluated the feasibility and acceptability of implementing a RLHS in EIS, which was the primary objective of the SARPEP research project. Methods: We conducted six focus groups, two with service users, and one each with family members, psychiatrists, managers, and team leaders to ensure multistakeholder perspectives. Participants (n=24) were asked to share their experiences with implementation of the RLHS over three years. Focus groups drew on the five dimensions of the RE-AIM framework, as did the quantitative study phase, while Lessard’s five dimensions for a learning health system guided development of the focus group questions. Results: Reach: invited members from all five stakeholder groups of the RLHS attended focus group meetings and shared their experiences: service users (n=5), family members (n=2), psychiatrists (n=4), managers (n=6), and team leaders (n=7). Effectiveness: all stakeholder groups confirmed the ability of the RLHS to improve the capacity of programs to collect data on core indicators and to promote evidence-based practices. Adoption: stakeholders acknowledged the value of selecting specific indicators for a RLHS and the need to improve technologies for data gathering. They also recognized challenges in deploying and integrating digital platforms into routine care. Implementation and maintenance: all stakeholders elaborated on how the RLHS enabled mutual learning, self-reflection, and shared practices among EIS in Québec. Conclusions: Stakeholders agreed that a RLHS can promote evidence-based care in EIS. The RLHS created a sense of belonging in a community aiming to learn and improve. Trial registration: N/A Figures Figure 1 Figure 2 Figure 3 Figure 4 INTRODUCTION More effective than routine care 1–7 , early intervention services (EIS) have become the treatment of choice for psychosis in many countries such as Australia, UK, Singapore, Hong Kong, US 8 , Denmark 9, 10 , Canada 11 and elsewhere. International guidelines 1–3, 12, 13 describe key components of EIS such as open referral, timely treatment access, service user and family engagement, appropriate patient-staff ratios, case management and continuous staff development. Team-based care combines pharmacological treatment with evidence-based psychosocial interventions (e.g. cognitive behavioral therapy). In Canada, EIS have been implemented over two decades under provincial jurisdiction, with British Columbia 14 , Nova Scotia, Ontario 15 and Québec 16: 56 leading policy development and creating EIS guidelines and standards. Vibrant EIS networks, the EPI APP 17 in British Columbia, EPION 18 in Ontario, AQPPEP 19 in Québec, and the Canadian Consortium for Early Intervention in Psychosis (CCEIP) 20 promote knowledge translation and evidence-based care. Yet, as in other countries 21, 22 , EIS implementation in real-life Canadian settings has proven challenging 23–25 , for example the uptake of certain essential components (e.g., open referral processes; appropriate patient/case manager ratios) based on evidence-based practices and provincial quality indicators 23–25 . Overall, implementing fidelity standards 26 remains a challenge 23–25 . The rapid-learning health system (RLHS) embeds data collection in clinical settings for real-time, problem-focused learning and continuous quality improvement 27 . The RLHS is “a continuously learning health system in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience” 27 . Core principles include commitment to a culture of continuous learning and improvement, stakeholder integration in planning and implementation; and care guided by the systematic use of real-time evidence. Using information technology methods, the RLHS captures and analyzes data on care experiences, continually evaluates outcomes and refines processes (e.g., training), integrating a feedback cycle for learning and improvement 28 . The RLHS promotes innovation and responsiveness 29 , bridges evidence and practice, and improves efficiency, effectiveness, and quality in healthcare delivery 30–35 . Successful implementation of a RLHS in EIS hinges on developing practices that ensure safety, accountability, and evidence-based, quality care, while fostering engagement, collaboration, appreciation of diverse knowledge, and innovation. In 2019 our research team created SARPEP ( Système Apprenant Rapide pour les Programmes de Premiers Épisodes Psychotiques) 36 , the first RLHS for first-episode psychosis programs in Canada, following substantial investment by the Québec Ministry of Health and Social Services. The 18 existing EIS programs in 2016 expanded to include a network of 33 EIS teams dispersed throughout Québec by 2020 36, 37 . Our multistakeholder team implemented SARPEP in 11 EIS in Quebec, where over 128 healthcare professionals, including 33 psychiatrists, 11 team leaders, and 10 managers were providing services to over 1,695 active patients, with 734 new cases annually. These very diverse EIS were selected to represent the Quebec reality. Stakeholders (service users, family members, psychiatrists, managers, team leaders) and researchers participated in all phases of the SARPEP project, starting with the selection of meaningful EIS quality indicators. A digital infrastructure with a user-friendly interface was designed and implemented, including three technologies for routine data collection, aggregation and sharing (quarterly) among EIS. Data were collected from: 1) team leaders on their program practices (REDCap platform); 2) service users and families, using: 2a) “happy or not” digital terminals in waiting rooms and a mobile app to collect data from a three-question evaluation of services received that day, and 2b) an in-depth electronic questionnaire on care received and nine recovery dimensions including health and quality of life (REDCap platform). The collected data were aggregated and anonymized. Data analysis and graphical data visualization performed by the digital infrastructure provided participating programs with real-time feedback on how their practices reached standards, compared with other SARPEP programs, and tailored recommendations for reaching targeted best practices. Families and service users did not receive graphical feedback on their data during the study, as development of this component was still ongoing. SARPEP also offered community of practice activities, leveraging insights from data to build program capacity while aligning program practices with provincial standards. The primary objective of the SARPEP project was to determine the feasibility and acceptability of implementing a RLHS in EIS 36 . We used mixed methods grounded in the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework 38 to achieve this aim. The present study reports the qualitative findings from focus groups conducted with SARPEP stakeholders, adding depth and insight to the quantitative results and enhancing rigor by integrating multiple perspectives including those of service users and families. METHODS Study Design Qualitative evaluation of the SARPEP project followed publication of the study protocol 36 . All stakeholders were invited to participate in focus groups: service users, family members, psychiatrists (medical directors of their respective EIS), managers (decision-makers overseeing mental health services in public health organizations located in catchment areas where EIS operated), and team leaders (team coordinators of clinical activities and staff, who also worked as case managers). Focus group methods were selected, as small group discussions generate rich data from individuals with a shared interest, who may support or disagree on issues 39 . To avoid power imbalances, homogenous stakeholder groups were created (e.g., service users only, team leaders only), creating safe spaces where participants could speak freely. The qualitative research incorporated the RE-AIM framework 38 , while development of the focus group questions drew upon the six dimensions of the Lessard et al 40 learning health system, namely: objectives pursued by a RLHS in promoting evidence-based, quality care; a social dimension focused on creating community; a technical dimension concerning integration of digital data into routine care; a scientific dimension enabling learning, innovation and discovery; and an ethical dimension to ensure that learning and innovation activities in the RLHS protected patient rights and privacy. The five RE-AIM dimensions 38 used to guide data collection were: Reach : Representativeness of the focus groups participants, shown by the number of participants from each stakeholder group (service users, family members, psychiatrists, managers, team leaders), reflecting stakeholder involvement in SARPEP. Effectiveness : Perceptions of RLHS capacity to promote evidence-based, quality care in EIS. Adoption : Perceptions regarding the feasibility for EIS to integrate data collected on indicators, and digital technology, into routine care, including the relative importance of indicators, their meanings, and the ease/difficulty of collecting data on them. Implementation : Perceptions on the deployment and uptake of three innovative digital technologies for collecting and aggregating data, and sharing feedback on data analyzed in routine care (e.g., service user’s recovery self-assessment and clinical service evaluations), and on measures to protect patient rights and privacy. Maintenance : Perceptions of how the RLHS maintains and fosters a learning community, innovation and discovery. Recruitment Eleven of the 33 Québec EIS were recruited based on environment (academic/non-academic), setting (urban, semi-urban, rural), years of operation (10), and age-related admission criteria (adolescents only, young adults only). EIS were also selected for their interest in improving services. Six focus groups (1.5 hours each) were conducted: two with service users, and one each with family members, psychiatrists, managers, and team leaders. The research coordinator sent email invitations to all stakeholders still actively involved in designing and implementing SARPEP (except researchers) to participate (Table 1 ). The focus group moderator contacted interested stakeholders to explain the study and complete the consent process. Table 1 Focus group recruitment and composition Service users* Family members Psychiatrists Managers Team leaders** Contacted 9 4 9 8 10 Responded 5 3 4 7 8 Participated 5 2 4 6 7 • * Divided into 2 focus groups (one with 3 participants, the other with 2) ** Team leaders included three social workers, two occupational therapists, one nurse and one psychologist. Data Collection The focus groups were conducted online (Zoom platform) and co-facilitated by a trained moderator (MSS) and senior research staff (KM), both experienced with EIS and focus groups. Engaging facilitators without direct involvement in the SARPEP community aimed to enhance sharing and minimize desirability bias among participants. The sessions were conducted in French, and audio-/video-recorded; the audio files were transcribed verbatim. The focus group questions guided exploration of the five RE-AIM dimensions, and Lessard’s learning health system elements 40 . The open-group discussion format allowed for information sharing on individual experiences, perceptions, and the impact of SARPEP on work and the EIS, including willingness to change, attitudes toward data collection (SARPEP indicator Table 2 ), and facilitators/barriers to RLHS implementation. Table 2 SARPEP INDICATORS PRESENTED DURING FOCUS GROUPS INDICATORS EXAMPLES 1. Service users’ engagement and satisfaction with services • Services adapted to the needs of service users • ‎Youth-friendly environment • Disengagement • Outreach practices • Youth satisfaction 2. Family engagement • Type of intervention offered • Percentage of families reached • Number of visits • Family member satisfaction 3. Access to care - pathways • Direct access • Referral sources, including self-referral and community referral • Inclusion and exclusion criteria • Number of contacts before access 4. Access to care - systemic delays Time between referral and: • First contact • ‎First assessment • Start of treatment 5. Continuous education • Number and type of continuing education events attended by workers • Supervision and mentoring 6. Provider-to-patient ratios • Patient: Mental health professional ratio • Patient: Psychiatrist ratio 7. Evidence-based and recovery-oriented practices • Cognitive behavioral therapy, family intervention, employment or study programs, integrated treatment for substance use disorders, and peer support • Types of specialists who offer the interventions • Percentage of patients receiving long-acting injectable antipsychotics • Percentage of patients receiving clozapine 8. Self-reported recovery outcomes by the patient • Patient evaluations of their health, recovery, and quality of life Data Analysis The analytical process described by Braun and Clark 41 was adopted. Team members first read the full transcripts, checking them for accuracy. Researchers (MF, MSS) developed a standardized code book sensitive to the RE-AIM framework 38 and Lessard 40 learning health system dimensions. Data coded independently by the researchers were verified by the project coordinator, then mapped onto tables for the five respondent groups, whose experiences were then compared/contrasted. Team analyses and feedback ensured methodological effectiveness 42 and enhanced rigor. The analyses, produced in French, included identification of themes and sub-themes, adding descriptions/links between them. At the writing stage, the quotations were translated into English (JS), with oversight by KM and AAB who are fluent in both languages. The principal investigators (AAB, MF, SI, MAR, AL) reviewed and refined the themes and subthemes after exposure to the various analytical phases, including meetings to discuss previously shared analyses. The analysis was further deepened as research team members elaborated drafts of the manuscript. Study participants provided member checking of the manuscript, which further enhanced trustworthiness of the data. Senior researchers experienced in qualitative methodology and implementation science trained and mentored research staff throughout the research process. The study findings presented below follow the consolidated criteria for reporting qualitative research (COREQ), a 32-item checklist recommended by the EQUATOR network. FINDINGS This section presents the findings according to the RE-AIM framework: Reach: In all, 24 individuals participated in six focus groups (Table 1 ), one for each of five stakeholder groups, and service users organized as two groups. Of the nine service users involved in the project and invited to participate, five responded positively (three in one focus group, and two in another). Of the four family members invited, three responded and two participated. Nine psychiatrists, one per program (excluding the 2 programs where the psychiatrists were co-PIs in the research), were also invited, four of whom responded and formed the focus group. Of eight managers invited, seven responded and six participated. Finally, ten team leaders were invited; eight responded and seven participated. The team leaders included three social workers, two occupational therapists, one nurse and one psychologist. Effectiveness: Before implementation of SARPEP, few EIS routinely collected clinical or program-level data, and only rarely asked service users or family members to complete forms or questionnaires. Moreover, data were not systematically analysed, unless for research purposes, yearly program evaluations or reports: We work with an Excel database that we created ourselves. For sure, (data) were missing compared with what you would find in SARPEP. It’s not the same thing; some information requested by SARPEP was unavailable to us, which forced our team leader to rework things to find certain data. (Psychiatrist, H3). SARPEP offered EIS an opportunity to improve their measurement-based practices. Importantly, the feedback provided by SARPEP comparing EIS performance on various indicators offered a comparative perspective on their services, allowing EIS to better evaluate their strengths and weaknesses, adherence to norms, and areas for improvement: In terms of the project, I would say that it provides a view of the services we offer compared with what is offered in other EIS: what our strengths are; what points we need to improve as a clinic. (Psychiatrist, F1) Psychiatrists and team leaders underlined the timeliness of the SARPEP project, coming in the early days of the pandemic, with more communication difficulties and a loss of resources, as one team leader described: I find that (SARPEP) has helped us keep a common thread in sustaining an EIS. Because, with all that has happened … the pandemic, and all that, I would say that we lost some benchmarks along the way. The SARPEP project, coming at the time it did, allowed us to keep track of what we were doing, to understand what an EIS is, and how to provide the service for the public and for our young people. (F7) One psychiatrist emphasized that the new RLHS put Québec EIS “on the map as a community that offers early intervention services for youth with psychosis …” (H1) For another psychiatrist, the RLHS helped them evaluate data collected (or not) in their program and identify areas of improvement. Participating in a RLHS also encouraged more disciplined data collection, considered a positive outcome as it prompted self-assessment in their program. (F1) Regular and automatic feedback provided to the EIS after each period of data collection (see Fig. 1 ) was well received and valued, as a team leader described: [As a new EIS], we are finishing our first 3 years [of operation]; so, I am thinking about transition [to other services post-EIS]. For me, it was very helpful to know how this was done elsewhere, and how we could apply this to our clinic. I find that it’s helpful for newer EIS to sit down with more established ones, and learn [from them] how to do things, share tools, and how to use them … it’s really great. (F2) Overall, participants across stakeholder groups emphasized that SARPEP had created a “common thread” or “sharing forum” for mutual assistance among EIS, keeping them vigilant about performance and opportunities to improve their services: It offers a sharing forum between EIS. (Family member, F1) Adoption : a) Feasibility of integrating indicators, or digital data, into routine care Focus group participants were given a list of eight indicators (Table 1 ) identified from a synthesis of the literature and EIS guidelines 36 . SARPEP stakeholders had selected these indicators by consensus, as impactful for service user outcomes and service quality but amenable to change within the initial, two-year project. Participants were asked to comment on the indicators in terms of personal recovery, quality of care for service users and family members, and the ease or difficulty of collecting these data. Participants regarded the indicators as precise, relevant, and helpful: I think the indicators are pretty specific. They give you a general idea of how things are going. (Service user, Gr 1, H1) Opinions also converged around the importance of certain indicators. Several participants viewed access to care as most urgent and highly consequential for the quality of care and service user recovery: I would say that the wait time for access to care is important, so the person’s condition doesn’t deteriorate rapidly. (Service user, Gr 2) Stakeholders endorsed the establishment of standardized practices throughout Québec EIS to support RLHS implementation, for instance, developing standards around referral sources, inclusion/exclusion criteria for access to care, and direct access: …there are several important aspects to consider in the process of accessing care. In terms of inclusion and exclusion criteria, to have a good understanding of who needs to enter EIS, or not; as well as referral sources and direct access, because I think that these measures are still not well established in a homogeneous manner everywhere in Québec. (Manager, F2) Participants viewed reinforcing staff capacity and competence as essential in actualizing the RLHS. Managers prioritized the need for proper service intensity, some emphasizing continuous staff training and mentorship in a fluid environment: Continuous staff training is also a must . (Team leader, F2) Appropriate service user/provider ratios also helped maintain service delivery standards: For sure, in the larger scale of things the patient/clinician ratio is important for good services… (Service user, Gr 1) All stakeholder groups viewed indicators aligned with service user and family satisfaction and engagement as crucial elements for EIS, reflecting acceptability of the RLHS by the primary beneficiaries: Engagement and satisfaction of youth (…) I would say that it’s the patient who is at the center of everything. (Family member, F2) Concerning the ease or difficulty of collecting data on indicators, quantifiable indicators were generally considered easiest, as noted below: delay in access to care is easy. It’s quantitative, so it goes well enough. (Mangers F1; H1) It’s about dates in a system. (Service user, Gr 1) the patient-provider ratio, that’s easy to count. (Service user, Gr 1) One team leader found family engagement more difficult to measure than other indicators such as staff training activities: It was easier to keep statistics on group interventions, presentations, and conferences than on family engagement. (F5) Asked to comment on the quality of services, participants offered different insights. They viewed data collection following clinical appointments using the "happy or not” devices and the short REDCap questionnaire (via QR code) as straightforward for service users (Service user, Gr 2; Psychiatrist, H3). For instance, service users responding to the question: “are you satisfied with the service you received today?” would select the appropriate smiley-face emoticon button on the terminal: very happy , happy , unhappy , or very unhappy (see quotes: next section). However, some clinicians mentioned difficulties around inviting patients to complete the REDCap questionnaires. For us that was a clear challenge, the whole question of having patients fill out (satisfaction) questionnaires. (Psychiatrist, H3) Stakeholders viewed data collection as challenging, more due to subject matter than technical aspects. Service user satisfaction was one area: Satisfaction is also difficult to grasp, I find, because we are often in relationships where patients are under involuntary care, or some kind of treatment order … (Manager, F1) Service users noted another challenge in collecting information about engagement in treatment related to the issue of disengagement: Information about engagement in treatment can be difficult to collect when the person disengages from treatment. It often becomes difficult to know why, exactly the situation … it can be difficult to obtain information directly from the person who left… (Service user, Gr 2) Service users also found it potentially difficult to capture data on family engagement, as family members bring different standpoints: A family has several family members. Should families respond as a unit? That would be biased. Should each family member fill out their own questionnaire, one day the brother and another day the sister? Like if one day it's the brother and sister who see the patient, or not? (Service user, Gr 1) Providers endorsed data collection around recovery-oriented practices, and patient-reported outcomes, but didn’t always know whether these data were being collected: I was not aware of patient self-assessment of clinical outcomes, and I'm very, very interested in this. A clinical measure for symptoms, but also for recovery" (H1, p.9). Various stakeholders reported needs for further data (Table 3 ). Psychiatrists also expressed the need for specific indicators to better track symptomatic and functional patient outcomes over time: We are talking about an illness that affects functioning, but we do not have the elements that indicate improvement or stagnation in the patient’s level of functioning. (H1) Figures 1 and 2 show how different stakeholder groups positioned themselves in relation to the relevance and simplicity of collecting data on indicators used in the SARPEP project. Overall, the data suggest that participants agreed on the meaningfulness of the selected indictors ( 1 ), yet some disagreement emerged on the ease (+) or difficulty (-) of collecting the relevant data (Fig. 3 ). The experience of participants in using different technologies to collect data are part of both Adoption and Implementation dimensions of the RE-AIM framework, as described in the following section. Implementation and Maintenance: The feasibility of implementing the RLHS in EIS a) Deploying and using technologies Stakeholders viewed the REDCap digital platform as an effective tool, but found data reporting time consuming. Yet, having an infrastructure to support data collection, visualization and sharing was appreciated (Fig. 1 ): I think it is pertinent to continue filling that out… However, this is not a small task. It’s time-consuming. (Psychiatrist, H2) Team leaders found the periodic evaluation of their work through graphical feedback especially useful, and important for lifting morale. The reports gave clear indications of where the EIS stood in meeting their objectives and documenting needs: I think if the parameters are very well and clearly defined, and very well collected, the information is reliable, and it will therefore always be useful to receive this information [graphical feedback] (Psychiatrist H1) This graphical feedback provides clinics with information on access delays, by indicating the number of days between referral and the psychiatric evaluation. Clinics can track their own progress over time, and compare their performance to that of other programs, provincial averages and standards. Many EIS managers and team leaders found it challenging to use REDCap for administering satisfaction questionnaires to service users and family members and observed that they were “not generating a lot of data” (e.g., Manager, F1): “you could count on your fingers those who responded to the long-form questionnaire” (Team Leader, F6). While clinicians and psychiatrists initially attempted to implement these questionnaires, many reported discomfort about asking patients to fill them out; uptake was therefore weak. Moreover, some noted concerns that the lack of systematic data collection risked biasing results in favor of individuals wholly satisfied or wholly dissatisfied, if not upset, after receiving services. Replacing subjective patient assessments based on in-person discussion required a more systematic process, particularly for psychiatrists: For sure, this is new to us, but we had difficulty getting clients to fill it out. There were reminders … but it wasn’t filled out regularly by our clients. So, we still have things to work on regarding this aspect. (Manager, F3) You end your meeting, it goes well; then you give [the evaluation form ] to the patient. But it’s a bit embarrassing; it’s like asking to give me positive feedback. (Psychiatrist, F1) A service user participant also underlined the importance of encouraging service users to express their opinions about their care; otherwise “people will only write when they are frustrated, and that can bias the data collection.” (Gr 1) Other stakeholders also viewed data collection on service user and family satisfaction as extremely important, suggesting that the timing of questionnaire administration could impact client uptake and the relevance of their feedback. They also viewed the QR code as an effective solution for accessing questionnaires: I find that there are pivotal moments in follow-up where, if we encouraged the young person in a well-defined way to fill it out, we would maybe get better feedback… (Team leader, F6) On using the “happy or not” terminal to measure client satisfaction, participants generally gave positive feedback (Fig. 4 ): I would say that it’s set up well. It was just in the corridor where we go for our psycho-ed activities or meetings with providers. The tablets are big enough, so they’re easy to read. (Service user, Gr1) Others described issues that made installation of the “happy or not” terminal somewhat complicated, including distancing measures during the pandemic, sharing space with other services, or lack of a fixed clinic location since they were mainly offering outreach services. b) Stakeholder perceptions of how the RLHS protects the rights and privacy of service users Digital tools and the shift to measurement-based care were viewed by professionals as inevitable. The key issues around use of digital tools concerned the need to develop competence around technical issues, ethical issues and a desire to put people at ease: Currently, there is a strong will at the ministerial level; there is a culture of measurement that comes with a new technology every six months or so … and documents to complete. Then, we mentioned earlier how important it was to convince our supervisors of various elements. (Manager, H2) For some service users and family members, confidentiality and data security were important issues: This is not a worry, but rather a comment. I would say (what is important is) that the psychological profile stays confidential. This is the only thing that is important for me… (Service user, Gr 1) Others expressed further concern about equity in the use of technology and about service users who may be left behind: Well, for sure I have already expressed a certain worry in terms of the technology that is being deployed. It’s a good technology, ‘user friendly’ for people who are still young, with telephones, QR codes, etc. But there are still many young people … who would have a lot of trouble using this, QR codes. (Service user, Gr 1) Service users described the importance of creating safe spaces where they could speak up on issues: Yes, it is important that the patient feels comfortable saying things. There can’t be ‘backlash’ in the interventions if someone complains, and afterward they don’t feel comfortable. (Service user, Gr 1) A family member reflected on ensuring full representation of service user voices, even when most feedback was positive and service users seemed satisfied, how to prevent biases and attend to the needs of services users less apt to express their opinions, or whose experiences diverged from majority opinion: I always see that it may be biased in the end, you know. Even if 90% of our clients are satisfied, this may be because people who weren’t satisfied didn’t answer… (Family member, F1) Overall, service users were strongly on board with digital tools, welcoming the “technological leap” (Gr 1, H1), another service user underlining that service users are “in permanent contact with technology, which is instinctive for us.” (Gr1, H2). Moreover, only service users addressed the conditions for successfully integrating digital tools, emphasizing measures that would make their peers more amenable to completing surveys. One noted that service users would have more ideas and be more “mentally prepared” when informed prior to meetings with providers or others that a questionnaire would be offered. They also requested better explanations of questionnaire concepts (Gr1, H2). Others noted the need for service users to better understand the importance of participation: It would be important to inform the service user about the purpose of the survey … how services are useful, and to demonstrate that their opinions can make a difference in improving care. It is also important to state this in advance… (Service user, Gr1) As another service user emphasized regarding meaningful participation of service users in RLHS data collection: …the participation of users needs to be meaningful enough that the findings represent something reliable, not just summary, or partial findings that lack credibility. Youth participation is central. (Service user, Gr 2) c) how the RHLS maintains and promotes learning, innovation, and discovery Stakeholders across the board were enthusiastic about participation in capacity-building activities. One psychiatrist characterized SARPEP as a “community of psychiatrists and carers meeting for the first time to discuss problems. Before, this did not exist.” (H2) Capacity-building activities addressed practical issues, reassuring stakeholders about the RLHS: I find (them) super pertinent because, again, I find that this is where the learning system really comes to life; It’s okay to see that I have a gap, but how do I improve on it? (Manager, H1) Sometimes I say to myself, ‘heck, are we the only ones functioning like this?’ … So, when we met, I said to myself, ‘Okay, my reality isn’t so different from that of other regions or teams.’ So, when we met maybe twice in the past year … I found that it normalized our challenges as well as our successes, and the organization of our services. It was pleasant. (Team leader, F6) All service users felt welcome at capacity-building activities and encouraged to share their ideas: I found it very respectful. I knew some people when I did the meetings … I wasn’t alone in my corner. I didn’t feel isolated. I felt really accepted by the others. I have only positive points to make … Listen, I gave my opinion many times … I said what I think. I asked some very frank questions to certain people. (Service user, Gr2) A family member added: “it’s a plus, plus, plus … It was a good experience”. (F2) Psychiatrists and managers commented on how SARPEP helped promote learning and innovation regarding quality care in EIS. As a psychiatrist suggested: SARPEP gave legitimacy to the entire implementation process for EIS and shed light on the Québec EIS guidelines written by the Ministry. It’s as if, at least symbolically, it signals a new tendency, a kind of accountability, I would say. It puts the managers in front of a structured project, for which there were objectives to be met … from my point of view, it contributed to the implementation of programs that are well structured, that work well and are useful. (H2) Finally, the findings suggest that SARPEP succeeded in creating a learning community, although for newer staff this was not always obvious: I think there is a nice feeling of community. There was a nice collaboration in meetings. All the EIS representatives spoke about their different realities. A lot of stuff was brought up by patients, psychiatrists, providers, and case managers. (Service user, Gr 1) Only a couple managers noticed that the early intervention for psychosis community in Québec included many “communities of practice”; and that SARPEP is “one element among many” (F2), showing the importance of effectively integrating SARPEP within the various Québec initiatives: With PEPP, we have so many communities of practice. There is AQPPEP, then there are meetings of team leaders with the Ministry’s National Centre of Excellence, there is SARPEP, there are lots of things. I am not sure whether SARPEP brings a community of practice, as such. It is one element among so many others. Nor am I sure whether people can separate very well what belongs to what… (F2) d) how to sustain and improve the RLHS overtime and the future of SARPEP Stakeholders expressed interest in developing training resources to support continued data collection and in disseminating information to make SARPEP better known in their networks: Basically, SARPEP should be better known to clinicians, managers, and higher-level managers, even to the general management. (Manager, F1) We must sensitize those responsible for the program in the hospital administration about the importance of this type of activity. So, we really must continue to engage these people in the process. And there, I think it is important to utilize the Québec mental health plan as a lever for SARPEP. (Psychiatrist, H1) Team leaders expressed a desire to involve various stakeholders (clinicians, service users, family members) in data collection: If clinicians were also more involved in data collection, that would perhaps make more sense, and would become a real team mobilization project. I think it would be easier to get the data, perhaps on a day-to-day or weekly basis, and the entire team could be involved, rather than just having the team leader enter the data. (Team leader, F6) Several stakeholders favored the extension of SARPEP: If we put SARPEP in all Québec EIS, I think that would be good, but, again, having the tools [Informatic technology tools and support] so that utilization will be easier (Manager, F2) Others, including service users, viewed extending SARPEP as well aligned with their interests: Yes, because it gives patients a certain usefulness when they come. It allows them to regain a clear understanding of the importance of treatment (Service user, Gr 1) Asked about key learning from the project on how to improve EIS services, service users, family members and team leaders identified integrating peer support: For sure, the development of peer support … will be very important for the future development of EIS. (Service user, Gr 2) One family member felt somewhat left out of SARPEP, because service users were the primary focus, i.e., “the accent wasn’t broad enough” (F2). Better engagement and inclusion of families was highlighted by managers and families, as “psychosis” impacts the entire family: Well, do families have a space to propose improvements? I think that this is information we would like to have, as managers anyway ... We have patient partners ... we have, or will have, family partners. (Manager, H2) A service user provided the last word on priorities for SARPEP in the EIS: Whatever the digital tools, the research, most important is that the person remains at the center of the project, that the opinion of the person who receives care continues to be considered, and that s/he be considered a whole person with full potential for recovery. (Service user, Gr 1) Table 3 RESULTS. EXTRA DATA RELATED TO ADOPTION, IMPLEMENTATION AND MAINTENANCE THEMES QUOTES Adoption: Feasibility of integrating indicators, or digital data, into routine care I also ... think they are all important … (Administrator, H1) …I think that all these indicators are important. But I also believe it would be important to know to what extent the programs are capable of collecting this information in a reliable way. (Psychiatrist, H1) Moi aussi trouve que y sont tous importants (Gestionnaire, H1) …Ben je pense que tous ces, tous ces items-là sont importants. Mais encore est-il je crois que c’est important de savoir dans quelle mesure les programmes sont capables de collecter cette information de manière … de manière fiable. (Psychiatre, H1) Indicators - Access to care For our young people, I find [important), to take them in at the very beginning… (Team leader, F2) In the first place, I would say the access to care process and the time it takes to access care are important… (Manager, F3) I really agree, we see that access delays make a difference in the speed of recovery; … there is also a difference in the interventions that are needed when we delay access compared to when we move very quickly. I also find that the process of accessing care is very important … that we ensure fluid and rapid communication and have the information needed to serve the young person – very, very important. (Team leader F4) I find that ... the delay in accessing care is super important when we talk about the quality of services. (Team leader, F6) Pour nos jeunes, j’trouve, d’les prendre au début, début. (Chef d’équipe, F2) Le processus d’accès aux soins, puis le délai d’accès aux soins dans un premier temps qui sont importants… (Gestionnaire, F3) Et j’suis vraiment d’accord, le délai d’accès on voit vraiment la différence dans la rapidité du rétablissement, puis les interventions nécessaires si jamais on tarde versus si on va très rapidement … Le processus d’accès aux soins, j’trouve que c’est aussi très important. De s’assurer d’une communication fluide, rapide, d’avoir l’info qui faut pour servir le jeune, très, très important » (Chef d’équipe, F4) …moi j’trouve que … le délai d’accès aux soins est super important quand on parle de qualité de service. (Chef d’équipe, F6) Indicators - staff capacity and competence …the other thing for me is continuous staff training. We know that there is enormous turnover in our teams, as all throughout the healthcare system; so, it isn’t easy. (Manager, F2) I see that with the new workers who arrive at the clinic, their need for supervision, for mentoring in terms of this approach. So, I also find this important. (Team leader, F5) L’autre chose pour moi c’est la formation continue du personnel. On sait qu’on a énormément de roulement dans nos équipes pour, ben comme dans l’ensemble du système de santé, puis c’est pas facile. (Gestionnaire, F2) …moi j’vois avec des nouveaux intervenants qui sont arrivés à la clinique tout le besoin de supervision, de mentorat par rapport à cette approche-là aussi. Donc ça aussi j’trouve ça important. (Chef d’équipe, F5) Important indicators - satisfaction and engagement The satisfaction of young people and their families is, in my view, critical among the indicators that were chosen. (Manager, H2) For me, the priority indicators are the delay in access to care, and engagement with the family. This is important because the providers can be working in one direction, while the family has diverging opinions; so, it is better that everyone works in the same direction. (Service user, Gr. 1) …to adapt to the young people, to reach them more easily, in a way that best accommodates them. (Team leader, F5) Les éléments de satisfaction des jeunes et d’leur famille sont, à mon avis, critiques dans les indicateurs qui ont été choisis. (Gestionnaire, H2) Ben moi c’est ça, mes indicateurs en priorité c’est délais d’accès aux soins, l’engagement avec la famille et l’entourage, pour moi, c’est important parce que les intervenants peuvent travailler dans une direction et la famille peut avoir des opinions divergentes donc on est mieux de travailler tout le monde dans la même direction. (Chef d’équipe, F5) Indicator - dis/satisfaction …but I think that recovery is a notion that is very personal ... Someone may think that a person has recovered well, whereas for someone else … the person has not recovered that much. (Service user, Gr 2) Je pense que le rétablissement, c’est difficile de dire que … c’est une notion qui est très personnelle, le rétablissement. Pour quelqu’un, la personne va être bien rétablie et pour un autre, la personne ne sera pas tant que ça. (Utilisateur de service, Gr2) Indicator – need for new data We do a systematic follow-up of the clients … Client evolution is evaluated rigorously, I’m going to know if there was any family intervention, but I won’t necessarily have the number of family interventions. (Team leader, F7) … they talk about the patient’s self-rated clinical outcomes, but how about the data on vocational status, financial autonomy, those things… (Team leader, F1) Motivation to pursue a career, to have a stable job, university studies. (Service user, Gr 1) Maybe their extra-EIS activities, those that don’t take place at the EIS (Service user, Gr 1) Substance use, because more than half of the youth who arrive here use substances. (Team leader, F1) L’évolution de la clientèle, ça on le fait de façon rigoureuse. Puis c’est un peu là-dedans où est-ce que j’vais savoir si justement y a eu de l’intervention familiale ou non, mais j’aurai pas le nombre nécessairement d’interventions familiales. (Chef d’équipe, F7) …y disent l’évolution clinique auto-rapportée par le patient, mais les données sur le statut vocationnel, l’autonomie financière, ces choses-là (Chef d’équipe, F1 p.10) La motivation de poursuivre une carrière, donc avoir un emploi stable, quelque chose à l’université. (Utilisateur de service, Gr1) Peut-être les activités extra PEP, donc pas dans le PEP. (Utilisateur de service, Gr 1) La consommation parce qu’y a plus que la moitié de nos jeunes qui arrivent qui consomment. (Chef d’équipe, F1) Indicators – symptomatic and functional outcomes In this list, there are no indicators of symptom severity and evolution We are talking about an illness that affects functioning, but we do not have the elements that indicate improvement or stagnation in the patient’s level of functioning. (Psychiatrist, H1) One important element is the treatment algorithms. How can they be implemented in an easy, practical, pragmatic way, and how can they be followed over time? (Psychiatrist, H1) It would be very, very important, I think, to evaluate program performance. (Psychiatrist, H2) On parle d’une maladie qui affecte le fonctionnement. On n’a pas d’éléments non plus qui reflètent un peu l’amélioration ou bien la stagnation de patient sur le plan du fonctionnement. (Psychiatre, H1) Donc y a un élément important, c’est les algorithmes de traitement. Comment est-ce qu’on peut le mettre en place de manière facile, pratique, pragmatique et comment est-ce qu’on peut le suivre au cours du temps? (Psychiatre, H1) Puis ça serait très, très important, je pense, pour voir … pour, c’est ça, évaluer la performance d’un programme. (Psychaitre, H2) Implementation and Maintenance Deploying and using technologies I’m going to be honest: given the time at my disposal, I wasn’t able to play around with this questionnaire … But, lets’ look at it in positive way, the reality is that, what it takes to implement this, is manpower. (Team leader, F4) J’vais être honnête, ce questionnaire-là avec le temps que j’avais, j’étais pas capable d’aller jouer dans ça … Mais la réalité c’est que si on regarde de façon positive quoi faire pour implanter ça, ça nous prend … ça prend, ça prend la main-d’œuvre. (Chef d’équipe, F4) Graphical feedback This [data and graphical feedback] brings concreteness in our everyday actions. (Team leader, F7) It’s also fun for them to see what went well in the previous 3–4 months, what we still need to work on, or to improve … it also allowed us to identify objectives… case managers also expressed satisfaction with this. (Team leader, F5) …when we see that it was a good month, that we had some positive comments (from the [visual feedback], we transmit that to the team; it’s fun for providers to know that they are appreciated by the clients and this helps morale among the troops. (Team leader, F1) I liked that this brought us in contact with the provincial government EIS counselor, that it was documenting for our chiefs and managers some of our needs based on the data we collected. (Team leader, F2) Ça ramène de la concrétude dans nos actions de tous les jours. Ça, c’est clair. (Chef d’équipe, F7) C’est le fun aussi pour eux aussi de voir qu’est-ce qui, qu’est-ce qui a bien été dans les 3–4 derniers mois, c’qui nous restait à travailler ou à améliorer aussi. Fait que ça permettait aussi d’identifier des objectifs ... Les intervenants ont signifié aussi leur satisfaction par rapport à ça. (Chef d’équipe, F5) ... quand mettons dans le mois on voit qu’on, que c’était un bon mois, puis on a ben eu des commentaires positifs, ben on le transmet à l’équipe, puis c’est comme le fun pour les intervenants de savoir que y sont appréciés par la clientèle, au niveau du moral des troupes. (Chef d’équipe, F1) J’aimais le fait que ça nous ramène quand même à être en contact soit avec notre conseiller au ministère, … de documenter auprès de nos chefs, auprès des administrateurs certains besoins qu’on a de façon un p’tit peu plus … c’est comme, on a documenté en tout cas à partir de données qu’on a colligées. (Chef d’équipe, F2) Satisfaction questionnaires You end your meeting, it goes well; then you give [the evaluation form ] to the patient. But it’s a bit embarrassing; it’s like asking to give me positive feedback. (Psychiatrist, F1) Fait que là tu finis ta rencontre, ça se passe bien, tu lui donnes, mais c’est un peu gênant. Ça va me faire des commentaires positifs. (Psychiatre, F1) Data collection on service user and family satisfaction There were some weeks where all the providers talked about it with all the patients at each meeting. We had one or two evaluations. Then, we tried to go in a strategic fashion, because sometimes … it’s the first meeting with patients; they can still be perplexed, symptomatic. So, we were trying to ‘time’ the moment that we talked about it. I’m not sure if this is a mistake, or if it’s really something we could work on. (Psychiatrist, H2) Y a eu des semaines où tous les intervenants en ont parlé avec tous les patients à chaque rencontre. On a eu une ou deux évaluations. Ensuite on a essayé d’y aller de façon stratégique parce qu’effectivement des fois on … c’est première rencontre avec des patients, puis y peuvent être encore perplexe, symptomatiques. Donc là on essayait de timer le moment où on en parlait. On sait pas trop si c’est une erreur ou si c’est vraiment quelque chose sur lequel on pourrait travailler. (Psychiatre, H2) Measure for client satisfaction using “happy or not” and REDCap satisfaction survey The ‘happy or not’ for me is playful, and it’s right at the door. So, it’s fun, ‘flashy’. (Manager, F1) I see that this gives me a very quick idea immediately after an appointment whether people are satisfied. (Manager, H1) Le Happy or not pour moi est ludique, puis y est vraiment à la porte de sortie. Fait que y est le fun, y flash. (Gestionnaire, F1) Moi j’vois d’avoir une idée très rapide après un rendez-vous si les gens sont satisfaits dans l’immédiat. (Gestionnaire, H1) “Happy or not” and location in the clinic We installed the famous machine … we couldn’t put it just anywhere because there are also places in the hospital that are multifunctional (…) maybe it isn’t installed in a good place. (Psychiatrist. H2) We don’t have the stable offices or waiting rooms. So, it’s complicated to move the big tablet around. We don’t have a fixed place to put it. (Team leader, F2) …well, I’m speaking for our place, I don’t know about elsewhere, but it is in the waiting room. Not very popular with infection control department. (Administrator, F1) Nous on l’a installé, la fameuse machine … on pouvait pas la mettre n’importe où parce que nous aussi y a des lieux dans l’hôpital qui sont multifonction ... Fait que on cette machine-là installée, mais peut-être qu’elle est pas installée au bon endroit. (Psychiatre, H2) Qu’on n’a pas de, nécessairement de locaux ou de salle d’attente. On est assez nomade. Fait que ça, c’est compliqué de trainer la grosse tablette, on n’a pas d’endroit fixe où la mettre. (Chef d’équipe, F2) J’parle pour chez nous, je sais pas ailleurs, mais y est dans la salle d’attente. Y est pas beaucoup aimé de la prévention des infections. (Gestionnaire, F1) RLHS protects the rights and privacy of service users and maintains equity …we have to pay a lot of attention, introducing things slowly, slowly … step by step, and see what the problems are, what difficulties we encounter in using these tools. We also need to consult with IT services, with the ethics committee, with patients. Everyone needs to be comfortable with this, and we must do it in stages and make sure that we don’t begin with a big project and then it flops and doesn’t work. A few clinicians who are engaged in this type of activities, need to be included, a few patients, and then improve the quality of the experience as we increase our experience… We have been saying this for 20 years, but I hope we will get there. These types of platforms will probably be part of our everyday life in future. (Psychiatrist, H1) I think that, yes, it’s how these data are utilized, where they are stored. This is confidential information, that’s it. (Service user, F1) But I’m thinking now, she (daughter) thinks that her cell phone is being hacked, you know, so if she goes to the doctor tomorrow morning, the last thing she wants is to respond on the tablet… (Family member, p. 16) I am trying to think of someone young, like me, who is in the streets; well, I don’t think they will use it. They have other things to do than to spend 10 minutes filling out a survey with him in the clinic. (Service user, Gr2) ... y va falloir faire très attention c’est-à-dire en introduisant les choses tranquillement, doucement, pas à pas et voir un peu quels sont les problèmes, les difficultés qu’on rencontre dans l’application de ces outils-là. Donc y faut également consulter avec nos services informatiques. Y faut consulter avec le comité d’éthique. Y faut consulter avec les patients. Y faut que tout le monde soit à l’aise avec ça et puis y faut que faire ça par étape et s’assurer que … s’assurer de ne pas commencer avec un grand projet et puis par la suite ça flop et ça ne fonctionne pas. Y faut inclure quelques cliniciens qui sont engagés dans ce genre d’activités, quelques patients et puis par la suite améliorer l’expérience et augmenter notre expérience …. Cela on le dit depuis maintenant 20 ans, mais j’espère qu’on va y arriver et ce genre de plateformes vont faire probablement partie de notre quotidien dans l’avenir. (Psychiatre, H1) Je pense que oui, c’est comment ces données-là sont utilisées, où est-ce qu’elles sont stockées. C’est de l’information confidentielle, c’est ça. (Utilisateur de service, F1) Mais je suis en train de penser en ce moment, elle pense que son téléphone cellulaire est hacké, la, tu sais … fait que la si elle s’en va au médecin demain matin, répondre sur la tablette c’est la dernière affaire qu’elle veut. (Membre de la famille, p.16) J’imagine que mettons, j’essaie de penser à quelqu’un de jeune comme moi, mais qui est dans la rue, bien je ne pense pas qu’ils vont le prendre, il y a d’autres besoins que de prendre 10 minutes pour remplir le sondage dans la clinique. (Utilisateur de service, Gr 2) RHLS maintains and promotes learning, innovation, and discovery …that helped the team somewhat to decide on what data to collect (Psychiatrist, H1) Sometimes I say to myself, ‘heck, are we the only ones functioning like this? …So, when we met, I said to myself, ‘Okay, my reality isn’t so different from that of other regions or teams. So, when we met maybe twice in the past year … I found that it normalized our challenges as well as our successes, and the organization of our services. It was pleasant. (Team leader, F6) I think that we started there (capacity-building activities). This is the first time that, as a community of psychiatrists, of carers, we are together to discuss these problems. Before that, nothing existed. (Psychiatrist, H1) I don’t think we have a sense of community for the managers … Or, maybe it’s ‘me’, because I just arrived… (Manager, H1) It allows us to keep a common thread. It allows exchanges between coordinators, team leaders … It’s like, yes, it keeps you aware of what’s going on elsewhere, and then it feels good to get out of your little belly button. (Team leader, F7) I think this has surely had an impact, because it offers a sharing forum between clinic … From what I’ve seen, there was more openness, more ideas. There was more communication between the different EIS, and that in addition to the fact that the 2015–2020 mental health plan spoke specifically about family and close ones (“entourage”), so this increased engagement with families. I think that this also helped. (Family member, F2) Ça a aidé l’équipe de décider un peu sur quoi, qu’est-ce qu’on collecte un peu dans nos données. (Psychiatrist, H1) …des fois j’me disais coudonc on est-tu dans le champ, on est-tu tout seul à fonctionner comme. Fait que quand on se réunissait, j’me disais OK ma réalité est pas si différente des autres régions ou des autres équipes. Fait que moi j’ai apprécié quand on se réunissait peut-être… peut-être deux fois dans la dernière année. Mais ça normalisait, j’trouve, autant nos défis que nos bons coups, que l’organisation des services. C’était agréable. (Chef d’équipe, F6) Je pense que là on commence. C’est la première fois que en tant que communauté de psychiatre, de soignants, de … on est ensemble pour discuter de ces problèmes-là. Avant ça n’existait pas. (Psychiatre, H1) J’pense pas qu’on a un sentiment de communauté pour les gestionnaires … Ou c’est peut-être moi parce que j’viens d’arriver... (Gestionnaire, H1) Ç’a permis de garder un fil conducteur. Ça permettait des échanges justement entre les coordonnateurs, chefs d’équipe … c’est comme oui à l’affût de qu’est-ce qui se passe ailleurs, puis ça fait du bien de sortir de son petit nombril. (Chef d’équipe, F7) Je pense que oui, moi. Ça a eu surement un impact parce que ça offre une tribune de partage entre les cliniques ... qu’est-ce que j’ai vu, c’est qu' il y avait plus d’ouvertures d'idées, il y avait plus de communication entre les PEPP, et ça avec le fait que le plan santé mentale de 2015–2020 a parlé spécifiquement de l’entourage, qu’on devrait engager l’entourage, et qu’on devrait élargir nos affaires avec eux, je pense que ça a aidé aussi. (Membre de la famille, F2) How SARPEP sustains and improves the RLHS over time Basically, SARPEP should be better known to clinicians, managers, and higher-level managers, even the general directors of our health organizations. I think it has to be recognized …. there has to be an awareness about SARPEP. (Manager, F1) I think that … what will really help, is when they rewrite the Québec EIS guidelines. That will be super useful, and we have already been told about it… (Team leader, F1) …there still remain challenges for our clinic. We don’t fulfill all the requirements of the Québec EIS guidelines.. So, what we would like is that this … helps us perhaps to have a little more weight [credibility] with management to be able to continue to improve … (Manager, F1) …it (SARPEP) allows the clinic to improve and put forward the concerns of young people, while adapting services to their needs and vision, to their image… To provide the best chances for success, recovery and to always aim for excellence in treatment… all this (Service user, Gr 1) We know that the data are going to be collated so changes can be made (Service user, Gr 1) À la base que ce soit mieux connu autant des intervenants, des gestionnaires, puis des gestionnaires de plus haut niveau voire même jusqu’aux directions générales. J’pense que faut que ça soit … faut que ce soit … pour être reconnu y faut que ce soit connu. (Gestionnaire, F1) Moi j’pense que tout ce projet-là, c’qui va vraiment aider, c’est quand y vont réécrire le cadre de référence des EIS au Québec. Ça va être hyper utile parce que nous on l’a dit. (Chef d’équipe, F1) …y reste encore des défis au sein de la clinique. On remplit pas toutes les exigences du cadre de référence. Donc ce qu’on aimerait c’est que ça, ça nous aide peut-être à avoir un peu plus de poids sur la gestion pour pouvoir continuer à améliorer. (Gestionnaire, F1) Ça permet toujours à la clinique de s’améliorer et de mettre de l’avant la préoccupation des jeunes … et de justement adapter leur service à leurs besoins, à leur image … Donner les meilleures chances de réussite, de rétablissement et de toujours viser l’excellence au niveau des soins, tout ça. (Utilisateur de service, Gr1) On le sait que les données vont être colligées pour qu’il y ait des changements. (Utilisateur de service, Gr1) How to improve SARPEP …often the peer support worker could, from his experience, help people to open up more about their symptoms, to express themselves, which was good. (Service user, Gr 1) In our program, we would like to include a patient partner, a peer support worker on the team … I think that could be very interesting for some clients who are a bit ambivalent about some of the proposed interventions. (Manager, F4) To have a family peer support worker in all the clinics! And to meet him/her at the beginning! Because I am working hard on that… (Family member, F2) …souvent, le pair aidant pouvait de son expérience et ça aidait les gens à s’ouvrir davantage sur leurs symptômes et à oser s’exprimer, donc ça, c’est bien. (Utilisateur de service, Gr1) Du côté des Laurentides on aimerait bien pouvoir inclure un patient partenaire, un pair aidant dans l’équipe … J’pense que ça pourrait être intéressant pour certains clients qui peuvent être un peu ambivalents face à certaines propositions d’interventions. (Gestionnaire, F4) Avoir un pair-aidant famille dans toutes les cliniques! … Et le rencontrer au début! Parce que là je suis en train de travailler fort… (Membre de la famille, F2) DISCUSSION In response to increasing public expectations of evidence-based and patient-centered health care, the RLHS represents a new paradigm for revamping health systems to achieve evidence-based, patient-centered care grounded in safe, timely, efficient, and equitable practices. Yet very few RLHS have been developed or evaluated in the mental health context. The SARPEP project created one of few RLHS in EIS for psychosis worldwide 43, 44 , and the first in Canada. This study presented qualitative findings from the mixed-method SARPEP project, describing experiences and insights of stakeholders involved in the design and 2-year implementation of the Québec RLHS in EIS for psychosis. Findings confirmed that the EIS accepted and highly valued the introduction of this innovative, measurement-based, digital system supporting the routine collection and sharing of program-level data. The RLHS, including capacity-building activities, also met with success in creating a learning community where all stakeholders, particularly service users and families, felt welcome and motivated to target areas of improvement in services. Study participants viewed the eight selected key indicators as highly relevant and useful. Access to care stood out as most consequential for recovery, followed by service user and family satisfaction. For some, maintaining quality standards depended upon ensuring service intensity (engaging/training staff; appropriate provider-patient ratios). Participants relied on the RLHS to boost their capacity to capture and monitor precise indicators aligned with evidence-based and recovery-oriented practices (Indicator 7) and patient-reported outcomes (e.g., self-evaluation of recovery dimensions) (Indicator 8). Participants found the selected indicators meaningful (Fig. 2 ) but disagreed somewhat on the ease or difficulty of collecting related data (Fig. 3 ). In this figure, + means indicators are considered easy to collect; - not as easy to collect) Regarding the three technologies for data collection, aggregation, analysis and visualization, the REDCap digital platform was considered effective overall, although challenges persisted for managers and team leaders around dedicated time and resources for data collection. The much-appreciated graphical feedback allowed them to evaluate clinical performance, celebrate good results with staff, and set new objectives. Uptake of satisfaction questionnaires by service users, whether using “happy or not” terminals or REDCap, was somewhat weak, provoking considerable stakeholder discussion around how to encourage systematic, patient self-assessment and service evaluation practices. Psychiatrists and clinicians expressed concerns about biased results favoring extreme satisfaction/dissatisfaction with services. Extended discussion by providers, service users and family members addressed issues around competence and comfort with technology, the full representation of service user voices, and equity regarding access to digital tools and skills-building, leaving no service user behind. Service users, families and psychiatrists advised careful consideration of ethics and data security in shifting to a culture of measurement and in using digital technologies. Most crucial was the common concern for service user and family satisfaction, particularly the urgent need for more peer support and family engagement in EIS. Our findings also reflected the five Lessard, et al 40 dimensions of learning health systems described above. Like other programs inspired by the learning health system vision, SARPEP prioritized safe, high-quality services 40, 45, 46 and better patient outcomes, goals that were achieved according to participants. For instance, all 11 EIS started to collect and share data on the indicators, improved their measurement-based practices, and committed to improve data quality and administrative/clinical practices. Moreover, the deployment of technologies by EIS demonstrated the capacity of the RLHS to support services and stakeholders, including service users and family members, as competent and equal partners in program design and evaluation activities 47 . Contrary to the view that successful RLHS implementation implies heavy data collection using sophisticated digital platforms and software, our findings support a growing body of research suggesting that access to an integrated data platform is a necessary, but insufficient, condition for a successful learning health system 30, 48 . Participants placed great importance on the emergence of an active learning community and sense of belonging, spearheaded by SARPEP, that enabled EIS to compare practices and learn from each other’s strategies, activities, and procedures. Lessard 40 also proposed a culture of continuous learning, citing transparency, collaboration, teamwork, and innovation as essential 49, 50 . The core elements needed to generate this culture include shared decision-making processes, alignment of stakeholder goals, requisite clinical and analytic expertise, and appropriate principles of governance and leadership 51 . SARPEP stakeholders appreciated this new community for sharing both problems and solutions, while adding credibility to their work and strengthening their representations to decision- and policy-makers. Empowering the social dimension of the RLHS involved harnessing the digital infrastructure, designed to support data gathering, sharing, visualization, and learning, to guide a shift in the healthcare ecosystem toward quality improvement and innovation. As findings described, the need for reliable and analyzable health data met with challenges: e.g., making the REDCap platform and “Happy or not” terminals more accessible and appealing to service users by introducing flexible digital solutions (QR code); technical issues involving data lifecycle management and verification, which ultimately resulted in a lengthening of the data collection-feedback cycles from three to four months to allow sufficient time for implementation; and challenges related to lack of time and resources for service providers, who recognized the need to involve multiple people in real-time data collection. Other studies have described similar challenges 52–55 . Overall, service users and family members accepted both short and long satisfaction questionnaires well, although their level of use (adoption) varied. Motivation among all stakeholders to participate fully in the RLHS also remained high, fostered by their sense of belonging to a movement nourished by a community of practice, and recognition that the RLHS created the capacity to measure important indicators, and bring improvement. The scientific dimension focused on innovations for improved health outcomes 30 . According to Lessard 40 , this dimension incorporates social and technical elements of the learning health system into a continuous learning circle that moves from data aggregation and analysis to interpretation and practice change . New data are generated for integration within the learning system. SARPEP promoted discovery, revealing practice gaps (e.g., lack of peer support workers in Québec EIS), while informing Québec EIS guidelines related to the anticipated incidence of new cases, by region, which prompted funding adjustments for human resources; also, the modification of clinic opening hours based on available resources while maintaining flexibility and easy patient access for regular appointments, or crises. Supported by the Québec Ministry, SARPEP will embark on a larger-scale implementation, adding 2–3 waves of EIS to the original 11 and aiming to include all 33 Québec EIS within approximately three years. SARPEP has also inspired new research projects and collaborations. Researchers have proposed a CBT implementation study, using the RLHS infrastructure for data collection on EIS practices and participant evaluations, the LHS media library for training, community of practice knowledge exchange sessions, and service user evaluations. Second is a cognitive remediation project that integrates user self-assessments into the REDCap platform and features a co-designed, stepped care intervention with service users, and capacity training for clinicians. PAIRPEP is a third project on peer support and family peer support implementation involving the entire RLHS system. PAIRPEP engages multiple RLHS stakeholders in partnership with local peer support organizations. The project also features a digital intervention (DIALOG+) for peer support workers and digital training using avatars. The ethical dimension in RLHS is often unaccounted for 40 , although preliminary work has led to a proposed ethical framework for RLHS activities 56 . Our study contributes to this area, especially in revealing the strong interest of service users and clinicians in preserving confidentiality and data security. Service users also expressed concerns about inequities in accessing digital technologies due to lack of skills or economic deprivation (e.g., youth homelessness). One peer support worker was assisting service user participation by helping them complete satisfaction and recovery self-assessment questionnaires. Findings strongly suggest that the Québec RLHS is advancing toward a truly participatory and inclusive EIS ecosystem. The strengths and limitations of this study should be noted. Stakeholder engagement and co-design with ongoing input throughout the implementation period was a definite strength, as was data collection using stakeholder-specific focus groups. Member-checking, a full review of study findings and contents by study participants, was also employed. One notable weakness in this qualitative evaluation was that the participants did not include family members currently receiving care in the EIS, although two service user participants were currently receiving services. As our focus in this study was on implementation of the SARPEP project, we considered the inclusion of service users and family members who were research partners as more pertinent to the composition of focus groups. Finally, given the richness of the data we dedicated a full manuscript to the qualitative results, even though mixed methods designs usually entail a simultaneous discussion of both qualitative and quantitative findings (convergent analysis). A future paper presenting a convergent analysis of qualitative and quantitative findings from the SARPEP project is planned, following the completion of further quantitative analyses. CONCLUSIONS This study described implementation of the first RLHS in early intervention services for psychosis in Canada, confirming the acceptability of the approach for promoting evidence-based care and measurement-based practices in EIS, despite initial challenges in deploying the technology and integrating data collection into routine care. The RLHS creates a welcoming community for learning and improvement. The findings support a range of provincial policies (system integration, healthcare monitoring, health informatics solutions, etc.). Most importantly, SARPEP has advanced our understanding of how the RLHS may be used and implemented in healthcare services and lays the foundations for extending this paradigm to other jurisdictions where existing EIS services are located, in Canada or elsewhere. Sharing common measures used in the RLHS operating in different provinces or countries, as a wider community of practice, may allow us to compare the impact of health policies and provincial health systems on the implementation of evidence-based practices. Declarations Human ethics and consent to participate: The research ethics board of the Centre de Recherche du Centre Hospitalier de l’Université de Montréal approved the study in December 2019 (reference: 19-282 and MP-02-2020-8627), followed by institutional approval from all 11 participating sites. Individuals provided written informed consent before participating in the focus groups. Only service users and family members received compensation, whereas other stakeholders participated during regular work hours. Consent for publication : All participants consent in their consent forms for anonymized and non-personal data publication Data availability declaration The datasets generated and/or analysed during the current study are not publicly available due privacy and confidentiality reasons but could be available from the corresponding author on reasonable request. Competing interests The authors declare that they have no competing interests as defined by BMC, or other interests that might be perceived to influence the results and/or discussion reported in this paper. Funding declaration This project is funded by Fonds de recherche du Québec-Santé (program name: Données de recherche en contexte réel - Partenariat Innovation-Québec-JANSSEN - Volet Schizophrénie - Cliniques intégrées en santé mentale dans un contexte réel , grant 284385).The aforementioned funding agency (which is the Quebec governmental health research funding agency) was not involved in the design of the study or data collection. Dr. Ferrari has a salary award from the Fonds de Recherche du Québec–Santé. Authors' contributions The researchers (AAB, SI, Annie Leblanc, MF, MAR - alphabetical order) designed the study. Researchers (MF, MSS) analyzed the data and developed the standardized code book. At the writing stage, the quotations were translated into English (Judith Sabetti), with oversight by KM and AAB who are fluent in both languages. The principal investigators (AAB, SI, Annie Leblanc, MF, MAR - alphabetical order) reviewed and refined the themes and subthemes after exposure to the various analytical phases, including meetings to discuss previously shared analyses. All authors elaborated drafts of the manuscript and reviewed the final manuscript. Acknowledgements The authors gratefully acknowledge the service users, family members, and clinical staff working in Quebec early intervention services for their involvement in the SARPEP project and focus groups. The authors also greatly appreciate the contribution of Camille Arbaud, Kevin MacDonald, and Judith Sabetti (alphabetical order) for their support in data collection, analysis, manuscript writing and formatting. References Breitborde NJ, Bell EK, Dawley D, et al. The Early Psychosis Intervention Center (EPICENTER): development and six-month outcomes of an American first-episode psychosis clinical service. BMC Psychiatry Oct 28 2015;15:266. Dixon LB, Goldman HH, Bennett ME, et al. Implementing Coordinated Specialty Care for Early Psychosis: The RAISE Connection Program. Psychiatr Serv Jul 2015;66(7):691–698. North CS, Simic Z, Burress J. Design, implementation, and assessment of a public comprehensive specialty care program for early psychosis. Journal of Psychiatric Practice 2019;25(2):91–102. Bertelsen M, Jeppesen P, Petersen L, et al. Five-year follow-up of a randomized multicenter trial of intensive early intervention vs standard treatment for patients with a first episode of psychotic illness: the OPUS trial. Archives of general psychiatry Jul 2008;65(7):762–771. Craig TK, Garety P, Power P, Rahaman N, Colbert S, Fornells-Ambrojo M, Dunn G. The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis. BMJ (Clinical research ed) Nov 6 2004;329(7474):1067. Anderson KK, Norman R, MacDougall A, Edwards J, Palaniyappan L, Lau C, Kurdyak P. Effectiveness of Early Psychosis Intervention: Comparison of Service Users and Nonusers in Population-Based Health Administrative Data. Am J Psychiatry May 1 2018;175(5):443–452. Harvey PO, Lepage M, Malla A. Benefits of enriched intervention compared with standard care for patients with recent-onset psychosis: a metaanalytic approach. Canadian journal of psychiatry Revue canadienne de psychiatrie Jul 2007;52(7):464–472. Humensky JL, Bello I, Malinovsky I, et al. OnTrackNY's learning healthcare system. J Clin Transl Sci Apr 6 2020;4(4):301–306. Malla A, McGorry P. Early Intervention in Psychosis in Young People: A Population and Public Health Perspective. Am J Public Health Jun 2019;109(S3):S181-S184. Malla AK, Norman RM, Joober R. First-episode psychosis, early intervention, and outcome: what have we learned? Canadian journal of psychiatry Revue canadienne de psychiatrie Dec 2005;50(14):881–891. Iyer S, Jordan G, MacDonald K, Joober R, Malla A. Early intervention for psychosis: a Canadian perspective. The Journal of nervous and mental disease May 2015;203(5):356–364. Chandra A, E. P, Hodge S, eds. Standards for early intervention in psychosis services – 1st Edition : Royal College of Psychiatrists, Early Intervention in Psychosis Network (EIPN); 2018. Early Psychosis Guidelines Writing Group and EPPIC National Support Program. Australian Clinical Guidelines for Early Psychosis, 2nd edition update , 2016, Orygen. Melbourne: The National Centre of Excellence in Youth Mental Health; 2016. Ehmann T. A Quiet Evolution: Early Psychosis Services in British Columbia. A Survey of Hospital and Community Resources 2003–2004.. 2004. Ontario Ministry of Health and Long-Term Care. Early Psychosis Intervention Program Standards 2011. Québec. Ministère de la Santé et des Services sociaux LDdc. 2017. Cadre de référence. Programmes d’interventions pour premiers épisodes psychotiques (PIPEP). Periodical 2017. The EPI Advanced Practice Project (EPI APP). Available at: https://www.earlypsychosis.ca/about-us/ . Early Psychosis Intervention Ontario Network (EPION). Available at: https://help4psychosis.ca/ . Association québécoise des programmes pour premiers épisodes psychotiques (AQPPEP). Available at: https://aqppep.ca/aqppep . Canadian Consortium for Early Intervention in Psychosis (CCEIP). Available at: https://www.earlypsychosisintervention.ca/schizophrenia-resources-websites#:~:text=CCEIP%20 WEBSITE,associated%20with%20early%20psychosis%20programs. Accessed July 27, 2023. Kotlicka-Antczak M, Podgorski M, Oliver D, Maric NP, Valmaggia L, Fusar-Poli P. Worldwide implementation of clinical services for the prevention of psychosis: The IEPA early intervention in mental health survey. Early Interv Psychiatry Dec 2020;14(6):741–750. Maric NP, Andric Petrovic S, Rojnic-Kuzman M, Riecher-Rossler A. Implementation of early detection and intervention services for psychosis in Central and Eastern Europe: Current status. Early Interv Psychiatry Oct 2019;13(5):1283–1288. Durbin J, Selick A, Hierlihy D, Moss S, Cheng C. A first step in system improvement: a survey of Early Psychosis Intervention Programmes in Ontario. Early Interv Psychiatry Dec 2016;10(6):485–493. Bertulies-Esposito B, Nolin M, Iyer S, Malla A, Tibbo P, Banks N, al. e. Implementation of Early Intervention Services for Psychosis in Quebec: A cross sectional study. Canadian Journal of Psychiatry Canadian Journal of Psychiatry. Nolin M, Malla A, Tibbo P, Norman R, Abdel-Baki A. Early Intervention for Psychosis in Canada: What Is the State of Affairs? Canadian journal of psychiatry Revue canadienne de psychiatrie Mar 2016;61(3):186–194. Addington DE, McKenzie E, Norman R, Wang J, Bond GR. Essential evidence-based components of first-episode psychosis services. Psychiatr Serv May 1 2013;64(5):452–457. Yong PL, Olsen L, McGinnis M. Value in Health Care. Accounting for cost, quality, safety, outcomes and innovation . Washington, D.C.: The National Academies Press; 2010. Agency for Healthcare Research and Quality. About Learning Health Systems. Content last reviewed May 2019. Available at: https://www.ahrq.gov/learning-health-systems/about.html . Greene SM, Reid RJ, Larson EB. Implementing the Learning Health System: From Concept to Action. Annals of Internal Medicine 2012;157(3):207–210. Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc Jan 2015;22(1):43–50. Menear M, Blanchette MA, Demers-Payette O, Roy D. A framework for value-creating learning health systems. Health Res Policy Syst Aug 9 2019;17(1):79. Abernethy AP, Etheredge LM, Ganz PA, Wallace P, German RR, Neti C, Bach PB, Murphy SB. Rapid-learning system for cancer care. J Clin Oncol Sep 20 2010;28(27):4268–4274. Delaney BC, Curcin V, Andreasson A, et al. Translational Medicine and Patient Safety in Europe: TRANSFoRm–Architecture for the Learning Health System in Europe. Biomed Res Int 2015;2015:961526. Platt R, Wilson MG, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network -- improving the evidence of medical-product safety. New England Journal of Medicine 2009;361(7):645–647. Wallace PJ. Reshaping cancer learning through the use of health information technology. Health Affairs 2007;26(2):w169-177. Ferrari M, Iyer S, LeBlanc A, Roy MA, Abdel-Baki A. A Rapid-Learning Health System to Support Implementation of Early Intervention Services for Psychosis in Quebec, Canada: Protocol. JMIR Res Protoc Jul 19 2022;11(7):e37346. Québec. Ministère de la Santé et des Services sociaux. Programme pour premiers épisodes paychotiques. Cadre de référence ; 2022. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG, Estabrooks PA. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health 2019;7:64. Kreuger RA, Casey MA. Focus groups: A practical guide for applied research . 5th ed. Thousand Oaks, CA: Sage Publications; 2014. Lessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz A. Architectural frameworks: defining the structures for implementing learning health systems. Implement Sci Jun 23 2017;12(1):78. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006/01/01 2006;3(2):77–101. Lincoln YS, Guba EG. Naturalistic inquiry . Newbury Park, CA: SAGE; 1985. Blanco C, Heinssen RK, Tenhula WN. Public Sector Learning Health Care Systems—Improving Patient Experience, Workforce Well-Being, and Recovery Outcomes. JAMA psychiatry 2024;81(1):9–10. Heinssen RK, Azrin ST. A national learning health experiment in early psychosis research and care. Psychiatric Services 2022;73(9):962–964. Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010;2(57):1–3. Yong PL, Olsen LA, McGinnis JM, eds. Roundtable on value & science-driven health care . Washington (DC): National Academies Press; 2010. Seid M, Margolis PA, Opipari-Arrigan L. Engagement, peer production, and the learning healthcare system. JAMA Pediatr Mar 2014;168(3):201–202. Dixon-Woods M, Campbell A, Chang T, et al. A qualitative study of design stakeholders' views of developing and implementing a registry-based learning health system. Implement Sci Mar 6 2020;15(1):16. Lowes LP, Noritz GH, Newmeyer A, Embi PJ, Yin H, Smoyer WE, Learn from Every Patient' Study G. 'Learn From Every Patient': implementation and early results of a learning health system. Dev Med Child Neurol Feb 2017;59(2):183–191. Olsen L, Saunders RS, McGinnis JM, eds. Patients charting the course: Citizen engagement and the learning health system . Washington (DC): National Academies Press; 2011. Abernethy AP. Demonstrating the learning health system through practical use cases. Pediatrics 2014;134(1):171–172. Grant RW, Uratsu CS, Estacio KR, et al. Pre-Visit Prioritization for complex patients with diabetes: Randomized trial design and implementation within an integrated health care system. Contemp Clin Trials Mar 2016;47:196–201. Nash DM, Brown JB, Thorpe C, Rayner J, Zwarenstein M. The Alliance for Healthier Communities as a Learning Health System for primary care: A qualitative analysis in Ontario, Canada. J Eval Clin Pract Dec 2022;28(6):1106–1112. Thuraisingam S, Chondros P, Dowsey MM, Spelman T, Garies S, Choong PF, Gunn J, Manski-Nankervis JA. Assessing the suitability of general practice electronic health records for clinical prediction model development: a data quality assessment. BMC Med Inform Decis Mak Oct 30 2021;21(1):297. Weir CR, Butler J, Thraen I, et al. Veterans Healthcare Administration providers' attitudes and perceptions regarding pragmatic trials embedded at the point of care. Clin Trials 2014;11(3):292–299. Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep Jan-Feb 2013;Spec No:S16-27. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 14 Mar, 2025 Read the published version in Health Research Policy and Systems → Version 1 posted Editorial decision: Revision requested 02 Nov, 2024 Reviews received at journal 30 Oct, 2024 Reviews received at journal 28 Oct, 2024 Reviews received at journal 24 Oct, 2024 Reviewers agreed at journal 11 Oct, 2024 Reviewers agreed at journal 08 Oct, 2024 Reviewers agreed at journal 07 Oct, 2024 Reviewers agreed at journal 20 May, 2024 Reviewers invited by journal 20 May, 2024 Submission checks completed at journal 21 Apr, 2024 Editor assigned by journal 21 Apr, 2024 First submitted to journal 04 Apr, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4216686","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":293823227,"identity":"aa84664d-f4ac-4852-a4aa-a5388ccbf0c0","order_by":0,"name":"Manuela Ferrari","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDUlEQVRIiWNgGAWjYNCCAgYG9gYwS0IORB54UEBIiwEDA88BiBZjsJYEA+K1MCSCbcOnRbf97DGJDwZ2DDzsvc8+/Nxjkb62/fhFkC3y/A3YtZidyUuTnGGQzMDDc9x4Zs8zidxtZ3IKQFoMZxzAoeVAjpk0jwEzg71EGjPQdUAtN3gSQFoSGHBpOf/GTPqPQT0Dj/wzZsY/ByTSzWBa5HFpuQG0hcHgMAOPBBszM9CWBLMb7AfAWgxwanljbNljcJyHhyeNmVnmgIQh0C+gQJYw3IjTYTmGN35UVMvxsB9jZnxzoE7e7Pjxxx8+VNjIy+HQAgM8yGxQpEjgV48G2B+QpHwUjIJRMAqGPQAA4SxXnv8JOLMAAAAASUVORK5CYII=","orcid":"","institution":"Douglas Mental Health University Institute","correspondingAuthor":true,"prefix":"","firstName":"Manuela","middleName":"","lastName":"Ferrari","suffix":""},{"id":293823228,"identity":"f670884a-d4c0-42e6-99bf-2dc5b25e4c5b","order_by":1,"name":"Marianne-Sarah Saulnier","email":"","orcid":"","institution":"University of Montreal","correspondingAuthor":false,"prefix":"","firstName":"Marianne-Sarah","middleName":"","lastName":"Saulnier","suffix":""},{"id":293823229,"identity":"194e6bc0-38af-485a-bef0-8733b80b96f3","order_by":2,"name":"Srividya N. Iyer","email":"","orcid":"","institution":"Douglas Mental Health University Institute","correspondingAuthor":false,"prefix":"","firstName":"Srividya","middleName":"N.","lastName":"Iyer","suffix":""},{"id":293823230,"identity":"7e5516e2-f3a6-4f0b-84f5-4fa9a2de30b3","order_by":3,"name":"Marc-André Roy","email":"","orcid":"","institution":"Université Laval","correspondingAuthor":false,"prefix":"","firstName":"Marc-André","middleName":"","lastName":"Roy","suffix":""},{"id":293823231,"identity":"ac029e11-57f9-4797-b91c-766a4c2adf57","order_by":4,"name":"Amal Abdel-Baki","email":"","orcid":"","institution":"Centre Hospitalier de l’Université de Montréal","correspondingAuthor":false,"prefix":"","firstName":"Amal","middleName":"","lastName":"Abdel-Baki","suffix":""}],"badges":[],"createdAt":"2024-04-04 08:59:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4216686/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4216686/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12961-024-01281-w","type":"published","date":"2025-03-14T15:58:45+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":55525430,"identity":"81d0f0b7-8a39-44aa-8639-67b30290d74c","added_by":"auto","created_at":"2024-04-29 14:42:58","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":76903,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eExample of graphical feedback provided to the clinics after each round of data collection\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-4216686/v1/1d8d7a56c14625d09006eba3.png"},{"id":55526117,"identity":"0986a0c2-56aa-4f98-b2e0-f5cb86676ecb","added_by":"auto","created_at":"2024-04-29 14:50:58","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":111533,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStakeholders’ acceptability of selected and used SARPEP indicators\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4216686/v1/5ee6466bd262c3f78fc08581.png"},{"id":55525429,"identity":"b6955428-79c8-4854-90a8-8641dc07a9f0","added_by":"auto","created_at":"2024-04-29 14:42:58","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":91547,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStakeholders’ assessment regarding the ease of collecting data on selected and used SARPEP indicators\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4216686/v1/dcd663862dc1b179810a62ce.png"},{"id":55525432,"identity":"7461b8dd-ed0d-425c-9e3f-a26f0c9740be","added_by":"auto","created_at":"2024-04-29 14:42:58","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":343342,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eThe “happy or not” terminal for service user evaluation\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4216686/v1/643b5254dbc7b62fcfd7464b.png"},{"id":78689737,"identity":"0a92d5be-5df4-485a-8f98-afb3a839764a","added_by":"auto","created_at":"2025-03-17 16:13:09","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2200131,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4216686/v1/fad251ef-8cce-45f4-ba90-91ab77298a2b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Implementing a rapid-learning health system in early intervention services for psychosis: Qualitative evaluation of its feasibility and acceptability","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eMore effective than routine care\u003csup\u003e1\u0026ndash;7\u003c/sup\u003e, early intervention services (EIS) have become the treatment of choice for psychosis in many countries such as Australia, UK, Singapore, Hong Kong, US\u003csup\u003e8\u003c/sup\u003e, Denmark\u003csup\u003e9, 10\u003c/sup\u003e, Canada\u003csup\u003e11\u003c/sup\u003e and elsewhere. International guidelines\u003csup\u003e1\u0026ndash;3, 12, 13\u003c/sup\u003e describe key components of EIS such as open referral, timely treatment access, service user and family engagement, appropriate patient-staff ratios, case management and continuous staff development. Team-based care combines pharmacological treatment with evidence-based psychosocial interventions (e.g. cognitive behavioral therapy).\u003c/p\u003e \u003cp\u003eIn Canada, EIS have been implemented over two decades under provincial jurisdiction, with British Columbia\u003csup\u003e14\u003c/sup\u003e, Nova Scotia, Ontario\u003csup\u003e15\u003c/sup\u003e and Qu\u0026eacute;bec\u003csup\u003e16: 56\u003c/sup\u003e leading policy development and creating EIS guidelines and standards. Vibrant EIS networks, the EPI APP\u003csup\u003e17\u003c/sup\u003e in British Columbia, EPION\u003csup\u003e18\u003c/sup\u003e in Ontario, AQPPEP\u003csup\u003e19\u003c/sup\u003e in Qu\u0026eacute;bec, and the Canadian Consortium for Early Intervention in Psychosis (CCEIP)\u003csup\u003e20\u003c/sup\u003e promote knowledge translation and evidence-based care. Yet, as in other countries\u003csup\u003e21, 22\u003c/sup\u003e, EIS implementation in real-life Canadian settings has proven challenging\u003csup\u003e23\u0026ndash;25\u003c/sup\u003e, for example the uptake of certain essential components (e.g., open referral processes; appropriate patient/case manager ratios) based on evidence-based practices and provincial quality indicators\u003csup\u003e23\u0026ndash;25\u003c/sup\u003e. Overall, implementing fidelity standards\u003csup\u003e26\u003c/sup\u003e remains a challenge\u003csup\u003e23\u0026ndash;25\u003c/sup\u003e .\u003c/p\u003e \u003cp\u003eThe rapid-learning health system (RLHS) embeds data collection in clinical settings for real-time, problem-focused learning and continuous quality improvement\u003csup\u003e27\u003c/sup\u003e. The RLHS is \u0026ldquo;a continuously learning health system in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience\u0026rdquo;\u003csup\u003e27\u003c/sup\u003e. Core principles include commitment to a culture of continuous learning and improvement, stakeholder integration in planning and implementation; and care guided by the systematic use of real-time evidence. Using information technology methods, the RLHS captures and analyzes data on care experiences, continually evaluates outcomes and refines processes (e.g., training), integrating a feedback cycle for learning and improvement\u003csup\u003e28\u003c/sup\u003e. The RLHS promotes innovation and responsiveness\u003csup\u003e29\u003c/sup\u003e, bridges evidence and practice, and improves efficiency, effectiveness, and quality in healthcare delivery\u003csup\u003e30\u0026ndash;35\u003c/sup\u003e. Successful implementation of a RLHS in EIS hinges on developing practices that ensure safety, accountability, and evidence-based, quality care, while fostering engagement, collaboration, appreciation of diverse knowledge, and innovation.\u003c/p\u003e \u003cp\u003eIn 2019 our research team created SARPEP (\u003cem\u003eSyst\u0026egrave;me Apprenant Rapide pour les Programmes de Premiers \u0026Eacute;pisodes Psychotiques)\u003c/em\u003e\u003csup\u003e36\u003c/sup\u003e, the first RLHS for first-episode psychosis programs in Canada, following substantial investment by the Qu\u0026eacute;bec Ministry of Health and Social Services. The 18 existing EIS programs in 2016 expanded to include a network of 33 EIS teams dispersed throughout Qu\u0026eacute;bec by 2020\u003csup\u003e36, 37\u003c/sup\u003e. Our multistakeholder team implemented SARPEP in 11 EIS in Quebec, where over 128 healthcare professionals, including 33 psychiatrists, 11 team leaders, and 10 managers were providing services to over 1,695 active patients, with 734 new cases annually. These very diverse EIS were selected to represent the Quebec reality.\u003c/p\u003e \u003cp\u003eStakeholders (service users, family members, psychiatrists, managers, team leaders) and researchers participated in all phases of the SARPEP project, starting with the selection of meaningful EIS quality indicators. A digital infrastructure with a user-friendly interface was designed and implemented, including three technologies for routine data collection, aggregation and sharing (quarterly) among EIS. Data were collected from: 1) team leaders on their program practices (REDCap platform); 2) service users and families, using: 2a) \u0026ldquo;happy or not\u0026rdquo; digital terminals in waiting rooms and a mobile app to collect data from a three-question evaluation of services received that day, and 2b) an in-depth electronic questionnaire on care received and nine recovery dimensions including health and quality of life (REDCap platform). The collected data were aggregated and anonymized. Data analysis and graphical data visualization performed by the digital infrastructure provided participating programs with real-time feedback on how their practices reached standards, compared with other SARPEP programs, and tailored recommendations for reaching targeted best practices. Families and service users did not receive graphical feedback on their data during the study, as development of this component was still ongoing. SARPEP also offered community of practice activities, leveraging insights from data to build program capacity while aligning program practices with provincial standards.\u003c/p\u003e \u003cp\u003eThe primary objective of the SARPEP project was to determine the feasibility and acceptability of implementing a RLHS in EIS\u003csup\u003e36\u003c/sup\u003e. We used mixed methods grounded in the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework\u003csup\u003e38\u003c/sup\u003e to achieve this aim. The present study reports the qualitative findings from focus groups conducted with SARPEP stakeholders, adding depth and insight to the quantitative results and enhancing rigor by integrating multiple perspectives including those of service users and families.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy Design\u003c/h2\u003e \u003cp\u003eQualitative evaluation of the SARPEP project followed publication of the study protocol\u003csup\u003e36\u003c/sup\u003e. All stakeholders were invited to participate in focus groups: service users, family members, psychiatrists (medical directors of their respective EIS), managers (decision-makers overseeing mental health services in public health organizations located in catchment areas where EIS operated), and team leaders (team coordinators of clinical activities and staff, who also worked as case managers). Focus group methods were selected, as small group discussions generate rich data from individuals with a shared interest, who may support or disagree on issues\u003csup\u003e39\u003c/sup\u003e. To avoid power imbalances, homogenous stakeholder groups were created (e.g., service users only, team leaders only), creating safe spaces where participants could speak freely.\u003c/p\u003e \u003cp\u003eThe qualitative research incorporated the RE-AIM framework\u003csup\u003e38\u003c/sup\u003e, while development of the focus group questions drew upon the six dimensions of the Lessard et al\u003csup\u003e40\u003c/sup\u003e learning health system, namely: \u003cem\u003eobjectives\u003c/em\u003e pursued by a RLHS in promoting evidence-based, quality care; a \u003cem\u003esocial dimension\u003c/em\u003e focused on creating community; a \u003cem\u003etechnical dimension\u003c/em\u003e concerning integration of digital data into routine care; a \u003cem\u003escientific dimension\u003c/em\u003e enabling learning, innovation and discovery; and an \u003cem\u003eethical dimension\u003c/em\u003e to ensure that learning and innovation activities in the RLHS protected patient rights and privacy.\u003c/p\u003e \u003cp\u003eThe five RE-AIM dimensions\u003csup\u003e38\u003c/sup\u003e used to guide data collection were:\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eReach\u003c/b\u003e: Representativeness of the focus groups participants, shown by the number of participants from each stakeholder group (service users, family members, psychiatrists, managers, team leaders), reflecting stakeholder involvement in SARPEP.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eEffectiveness\u003c/b\u003e: Perceptions of RLHS capacity to promote evidence-based, quality care in EIS.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eAdoption\u003c/b\u003e: Perceptions regarding the feasibility for EIS to integrate data collected on indicators, and digital technology, into routine care, including the relative importance of indicators, their meanings, and the ease/difficulty of collecting data on them.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eImplementation\u003c/b\u003e: Perceptions on the deployment and uptake of three innovative digital technologies for collecting and aggregating data, and sharing feedback on data analyzed in routine care (e.g., service user’s recovery self-assessment and clinical service evaluations), and on measures to protect patient rights and privacy.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMaintenance\u003c/b\u003e: Perceptions of how the RLHS maintains and fosters a learning community, innovation and discovery.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003cp\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eRecruitment\u003c/h2\u003e \u003cp\u003eEleven of the 33 Québec EIS were recruited based on environment (academic/non-academic), setting (urban, semi-urban, rural), years of operation (\u0026lt; 5 vs. \u0026gt;10), and age-related admission criteria (adolescents only, young adults only). EIS were also selected for their interest in improving services.\u003c/p\u003e \u003cp\u003eSix focus groups (1.5 hours each) were conducted: two with service users, and one each with family members, psychiatrists, managers, and team leaders. The research coordinator sent email invitations to all stakeholders still actively involved in designing and implementing SARPEP (except researchers) to participate (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The focus group moderator contacted interested stakeholders to explain the study and complete the consent process.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFocus group recruitment and composition\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eService users*\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFamily members\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePsychiatrists\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eManagers\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTeam leaders**\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eContacted\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eResponded\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eParticipated\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e• \u003cb\u003e*\u003c/b\u003eDivided into 2 focus groups (one with 3 participants, the other with 2)\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003e**\u003c/b\u003eTeam leaders included three social workers, two occupational therapists, one nurse and one psychologist.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003cp\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eData Collection\u003c/h2\u003e \u003cp\u003eThe focus groups were conducted online (Zoom platform) and co-facilitated by a trained moderator (MSS) and senior research staff (KM), both experienced with EIS and focus groups. Engaging facilitators without direct involvement in the SARPEP community aimed to enhance sharing and minimize desirability bias among participants. The sessions were conducted in French, and audio-/video-recorded; the audio files were transcribed verbatim. The focus group questions guided exploration of the five RE-AIM dimensions, and Lessard’s learning health system elements\u003csup\u003e40\u003c/sup\u003e. The open-group discussion format allowed for information sharing on individual experiences, perceptions, and the impact of SARPEP on work and the EIS, including willingness to change, attitudes toward data collection (SARPEP indicator Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e), and facilitators/barriers to RLHS implementation.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSARPEP INDICATORS PRESENTED DURING FOCUS GROUPS\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eINDICATORS\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEXAMPLES\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e1. Service users’ engagement and satisfaction with services\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Services adapted to the needs of service users\u003c/p\u003e \u003cp\u003e• ‎Youth-friendly environment\u003c/p\u003e \u003cp\u003e• Disengagement\u003c/p\u003e \u003cp\u003e• Outreach practices\u003c/p\u003e \u003cp\u003e• Youth satisfaction\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e2. Family engagement\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Type of intervention offered\u003c/p\u003e \u003cp\u003e• Percentage of families reached\u003c/p\u003e \u003cp\u003e• Number of visits\u003c/p\u003e \u003cp\u003e• Family member satisfaction\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e3. Access to care - pathways\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Direct access\u003c/p\u003e \u003cp\u003e• Referral sources, including self-referral and community referral\u003c/p\u003e \u003cp\u003e• Inclusion and exclusion criteria\u003c/p\u003e \u003cp\u003e• Number of contacts before access\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e4. Access to care - systemic delays\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTime between referral and:\u003c/p\u003e \u003cp\u003e• First contact\u003c/p\u003e \u003cp\u003e• ‎First assessment\u003c/p\u003e \u003cp\u003e• Start of treatment\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e5. Continuous education\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Number and type of continuing education events attended by workers\u003c/p\u003e \u003cp\u003e• Supervision and mentoring\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e6. Provider-to-patient ratios\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Patient: Mental health professional ratio\u003c/p\u003e \u003cp\u003e• Patient: Psychiatrist ratio\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e7. Evidence-based and recovery-oriented practices\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Cognitive behavioral therapy, family intervention, employment or study programs, integrated treatment for substance use disorders, and peer support\u003c/p\u003e \u003cp\u003e• Types of specialists who offer the interventions\u003c/p\u003e \u003cp\u003e• Percentage of patients receiving long-acting injectable antipsychotics\u003c/p\u003e \u003cp\u003e• Percentage of patients receiving clozapine\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e8. Self-reported recovery outcomes by the patient\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e• Patient evaluations of their health, recovery, and quality of life\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eData Analysis\u003c/h2\u003e \u003cp\u003eThe analytical process described by Braun and Clark\u003csup\u003e41\u003c/sup\u003e was adopted. Team members first read the full transcripts, checking them for accuracy. Researchers (MF, MSS) developed a standardized code book sensitive to the RE-AIM framework\u003csup\u003e38\u003c/sup\u003e and Lessard\u003csup\u003e40\u003c/sup\u003e learning health system dimensions. Data coded independently by the researchers were verified by the project coordinator, then mapped onto tables for the five respondent groups, whose experiences were then compared/contrasted. Team analyses and feedback ensured methodological effectiveness\u003csup\u003e42\u003c/sup\u003e and enhanced rigor. The analyses, produced in French, included identification of themes and sub-themes, adding descriptions/links between them. At the writing stage, the quotations were translated into English (JS), with oversight by KM and AAB who are fluent in both languages. The principal investigators (AAB, MF, SI, MAR, AL) reviewed and refined the themes and subthemes after exposure to the various analytical phases, including meetings to discuss previously shared analyses. The analysis was further deepened as research team members elaborated drafts of the manuscript.\u003c/p\u003e \u003cp\u003eStudy participants provided member checking of the manuscript, which further enhanced trustworthiness of the data. Senior researchers experienced in qualitative methodology and implementation science trained and mentored research staff throughout the research process. The study findings presented below follow the consolidated criteria for reporting qualitative research (COREQ), a 32-item checklist recommended by the EQUATOR network.\u003c/p\u003e \u003c/div\u003e "},{"header":"FINDINGS","content":"\u003cp\u003eThis section presents the findings according to the RE-AIM framework:\u003c/p\u003e\u003ch2\u003eReach:\u003c/h2\u003e\u003cp\u003eIn all, 24 individuals participated in six focus groups (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), one for each of five stakeholder groups, and service users organized as two groups. Of the nine service users involved in the project and invited to participate, five responded positively (three in one focus group, and two in another). Of the four family members invited, three responded and two participated. Nine psychiatrists, one per program (excluding the 2 programs where the psychiatrists were co-PIs in the research), were also invited, four of whom responded and formed the focus group. Of eight managers invited, seven responded and six participated. Finally, ten team leaders were invited; eight responded and seven participated. The team leaders included three social workers, two occupational therapists, one nurse and one psychologist.\u003c/p\u003e\u003ch2\u003eEffectiveness:\u003c/h2\u003e\u003cp\u003eBefore implementation of SARPEP, few EIS routinely collected clinical or program-level data, and only rarely asked service users or family members to complete forms or questionnaires. Moreover, data were not systematically analysed, unless for research purposes, yearly program evaluations or reports:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWe work with an Excel database that we created ourselves. For sure, (data) were missing compared with what you would find in SARPEP. It’s not the same thing; some information requested by SARPEP was unavailable to us, which forced our team leader to rework things to find certain data. (Psychiatrist, H3).\u003c/em\u003e \u003c/p\u003e\u003cp\u003eSARPEP offered EIS an opportunity to improve their measurement-based practices. Importantly, the feedback provided by SARPEP comparing EIS performance on various indicators offered a comparative perspective on their services, allowing EIS to better evaluate their strengths and weaknesses, adherence to norms, and areas for improvement:\u003c/p\u003e\u003cp\u003e \u003cem\u003eIn terms of the project, I would say that it provides a view of the services we offer compared with what is offered in other EIS: what our strengths are; what points we need to improve as a clinic. (Psychiatrist, F1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003ePsychiatrists and team leaders underlined the timeliness of the SARPEP project, coming in the early days of the pandemic, with more communication difficulties and a loss of resources, as one team leader described:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI find that (SARPEP) has helped us keep a common thread in sustaining an EIS. Because, with all that has happened … the pandemic, and all that, I would say that we lost some benchmarks along the way. The SARPEP project, coming at the time it did, allowed us to keep track of what we were doing, to understand what an EIS is, and how to provide the service for the public and for our young people. (F7)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOne psychiatrist emphasized that the new RLHS put Québec EIS “on the map as a community that offers early intervention services for youth with psychosis …” (H1) For another psychiatrist, the RLHS helped them evaluate data collected (or not) in their program and identify areas of improvement. Participating in a RLHS also encouraged more disciplined data collection, considered a positive outcome as it prompted self-assessment in their program. (F1)\u003c/p\u003e\u003cp\u003eRegular and automatic feedback provided to the EIS after each period of data collection (see Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) was well received and valued, as a team leader described:\u003c/p\u003e\u003cp\u003e \u003cem\u003e[As a new EIS], we are finishing our first 3 years [of operation]; so, I am thinking about transition [to other services post-EIS]. For me, it was very helpful to know how this was done elsewhere, and how we could apply this to our clinic. I find that it’s helpful for newer EIS to sit down with more established ones, and learn [from them] how to do things, share tools, and how to use them … it’s really great. (F2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOverall, participants across stakeholder groups emphasized that SARPEP had created a “common thread” or “sharing forum” for mutual assistance among EIS, keeping them vigilant about performance and opportunities to improve their services:\u003c/p\u003e\u003ch2\u003eIt offers a sharing forum between EIS. (Family member, F1)\u003c/h2\u003e\u003cp\u003e \u003cb\u003eAdoption\u003c/b\u003e:\u003c/p\u003e\u003cb\u003ea) Feasibility of integrating indicators, or digital data, into routine care\u003c/b\u003e\u003cp\u003eFocus group participants were given a list of eight indicators (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) identified from a synthesis of the literature and EIS guidelines\u003csup\u003e36\u003c/sup\u003e. SARPEP stakeholders had selected these indicators by consensus, as impactful for service user outcomes and service quality but amenable to change within the initial, two-year project.\u003c/p\u003e\u003cp\u003eParticipants were asked to comment on the indicators in terms of personal recovery, quality of care for service users and family members, and the ease or difficulty of collecting these data. Participants regarded the indicators as precise, relevant, and helpful:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI think the indicators are pretty specific. They give you a general idea of how things are going. (Service user, Gr 1, H1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eOpinions also converged around the importance of certain indicators. Several participants viewed access to care as most urgent and highly consequential for the quality of care and service user recovery:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI would say that the wait time for access to care is important, so the person’s condition doesn’t deteriorate rapidly. (Service user, Gr 2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eStakeholders endorsed the establishment of standardized practices throughout Québec EIS to support RLHS implementation, for instance, developing standards around referral sources, inclusion/exclusion criteria for access to care, and direct access:\u003c/p\u003e\u003cp\u003e \u003cem\u003e…there are several important aspects to consider in the process of accessing care. In terms of inclusion and exclusion criteria, to have a good understanding of who needs to enter EIS, or not; as well as referral sources and direct access, because I think that these measures are still not well established in a homogeneous manner everywhere in Québec. (Manager, F2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eParticipants viewed reinforcing staff capacity and competence as essential in actualizing the RLHS. Managers prioritized the need for proper service intensity, some emphasizing continuous staff training and mentorship in a fluid environment:\u003c/p\u003e\u003cp\u003e \u003cem\u003eContinuous staff training is also a\u003c/em\u003e \u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003emust\u003c/span\u003e. \u003cem\u003e(Team leader, F2)\u003c/em\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAppropriate service user/provider ratios also helped maintain service delivery standards:\u003c/p\u003e\u003cp\u003e \u003cem\u003eFor sure, in the larger scale of things the patient/clinician ratio is important for good services… (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAll stakeholder groups viewed indicators aligned with service user and family satisfaction and engagement as crucial elements for EIS, reflecting acceptability of the RLHS by the primary beneficiaries:\u003c/p\u003e\u003cp\u003e \u003cem\u003eEngagement and satisfaction of youth (…) I would say that it’s the patient who is at the center of everything. (Family member, F2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eConcerning the ease or difficulty of collecting data on indicators, quantifiable indicators were generally considered easiest, as noted below:\u003c/p\u003e\u003cp\u003e \u003cem\u003edelay in access to care is easy. It’s quantitative, so it goes well enough. (Mangers F1; H1)\u003c/em\u003e \u003c/p\u003e\u003ch2\u003eIt’s about dates in a system. (Service user, Gr 1)\u003c/h2\u003e\u003ch2\u003ethe patient-provider ratio, that’s easy to count. (Service user, Gr 1)\u003c/h2\u003e\u003cp\u003eOne team leader found family engagement more difficult to measure than other indicators such as staff training activities:\u003c/p\u003e\u003cp\u003e \u003cem\u003eIt was easier to keep statistics on group interventions, presentations, and conferences than on family engagement. (F5)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eAsked to comment on the quality of services, participants offered different insights. They viewed data collection following clinical appointments using the \"happy or not” devices and the short REDCap questionnaire (via QR code) as straightforward for service users (Service user, Gr 2; Psychiatrist, H3). For instance, service users responding to the question: “are you satisfied with the service you received today?” would select the appropriate smiley-face emoticon button on the terminal: \u003cem\u003every happy\u003c/em\u003e, \u003cem\u003ehappy\u003c/em\u003e, \u003cem\u003eunhappy\u003c/em\u003e, or \u003cem\u003every unhappy\u003c/em\u003e (see quotes: next section). However, some clinicians mentioned difficulties around inviting patients to complete the REDCap questionnaires.\u003c/p\u003e\u003cp\u003e \u003cem\u003eFor us that was a clear challenge, the whole question of having patients fill out (satisfaction) questionnaires. (Psychiatrist, H3)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eStakeholders viewed data collection as challenging, more due to subject matter than technical aspects. Service user satisfaction was one area:\u003c/p\u003e\u003cp\u003e\u003cem\u003eSatisfaction is also difficult to grasp, I find, because we are often in relationships where patients are under involuntary care, or some kind of treatment order … (Manager, F1)\u003c/em\u003e\u003c/p\u003e\u003cp\u003eService users noted another challenge in collecting information about engagement in treatment related to the issue of disengagement:\u003c/p\u003e\u003cp\u003e \u003cem\u003eInformation about engagement in treatment can be difficult to collect when the person disengages from treatment. It often becomes difficult to know why, exactly the situation … it can be difficult to obtain information directly from the person who left… (Service user, Gr 2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eService users also found it potentially difficult to capture data on family engagement, as family members bring different standpoints:\u003c/p\u003e\u003cp\u003e \u003cem\u003eA family has several family members. Should families respond as a unit? That would be biased. Should each family member fill out their own questionnaire, one day the brother and another day the sister? Like if one day it's the brother and sister who see the patient, or not? (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eProviders endorsed data collection around recovery-oriented practices, and patient-reported outcomes, but didn’t always know whether these data were being collected:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI was not aware of patient self-assessment of clinical outcomes, and I'm very, very interested in this. A clinical measure for symptoms, but also for recovery\" (H1, p.9).\u003c/em\u003e \u003c/p\u003e\u003cp\u003eVarious stakeholders reported needs for further data (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e\u003cp\u003ePsychiatrists also expressed the need for specific indicators to better track symptomatic and functional patient outcomes over time:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWe are talking about an illness that affects functioning, but we do not have the elements that indicate improvement or stagnation in the patient’s level of functioning. (H1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eFigures \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e2\u003c/span\u003e show how different stakeholder groups positioned themselves in relation to the relevance and simplicity of collecting data on indicators used in the SARPEP project. Overall, the data suggest that participants agreed on the meaningfulness of the selected indictors (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e), yet some disagreement emerged on the ease (+) or difficulty (-) of collecting the relevant data (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eThe experience of participants in using different technologies to collect data are part of both Adoption and Implementation dimensions of the RE-AIM framework, as described in the following section.\u003c/p\u003e\u003cp\u003e \u003cb\u003eImplementation and Maintenance: The feasibility of implementing the RLHS in EIS\u003c/b\u003e \u003c/p\u003e\u003ch2\u003ea) Deploying and using technologies\u003c/h2\u003e\u003cp\u003eStakeholders viewed the REDCap digital platform as an effective tool, but found data reporting time consuming. Yet, having an infrastructure to support data collection, visualization and sharing was appreciated (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e):\u003c/p\u003e\u003cp\u003e \u003cem\u003eI think it is pertinent to continue filling that out… However, this is not a small task. It’s time-consuming. (Psychiatrist, H2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eTeam leaders found the periodic evaluation of their work through graphical feedback especially useful, and important for lifting morale. The reports gave clear indications of where the EIS stood in meeting their objectives and documenting needs:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI think if the parameters are very well and clearly defined, and very well collected, the information is reliable, and it will therefore always be useful to receive this information [graphical feedback] (Psychiatrist H1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eThis graphical feedback provides clinics with information on access delays, by indicating the number of days between referral and the psychiatric evaluation. Clinics can track their own progress over time, and compare their performance to that of other programs, provincial averages and standards.\u003c/p\u003e\u003cp\u003eMany EIS managers and team leaders found it challenging to use REDCap for administering satisfaction questionnaires to service users and family members and observed that they were “not generating a lot of data” (e.g., Manager, F1): “you could count on your fingers those who responded to the long-form questionnaire” (Team Leader, F6). While clinicians and psychiatrists initially attempted to implement these questionnaires, many reported discomfort about asking patients to fill them out; uptake was therefore weak. Moreover, some noted concerns that the lack of systematic data collection risked biasing results in favor of individuals wholly satisfied or wholly dissatisfied, if not upset, after receiving services. Replacing subjective patient assessments based on in-person discussion required a more systematic process, particularly for psychiatrists:\u003c/p\u003e\u003cp\u003e \u003cem\u003eFor sure, this is new to us, but we had difficulty getting clients to fill it out. There were reminders … but it wasn’t filled out regularly by our clients. So, we still have things to work on regarding this aspect. (Manager, F3)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e \u003cem\u003eYou end your meeting, it goes well; then you give [the evaluation form\u003c/em\u003e] \u003cem\u003eto the patient. But it’s a bit embarrassing; it’s like asking to give me positive feedback. (Psychiatrist, F1)\u003c/em\u003e\u003c/p\u003e\u003cp\u003eA service user participant also underlined the importance of encouraging service users to express their opinions about their care; otherwise “people will only write when they are frustrated, and that can bias the data collection.” (Gr 1)\u003c/p\u003e\u003cp\u003eOther stakeholders also viewed data collection on service user and family satisfaction as extremely important, suggesting that the timing of questionnaire administration could impact client uptake and the relevance of their feedback. They also viewed the QR code as an effective solution for accessing questionnaires:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI find that there are pivotal moments in follow-up where, if we encouraged the young person in a well-defined way to fill it out, we would maybe get better feedback… (Team leader, F6)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e On using the “happy or not” terminal to measure client satisfaction, participants generally gave positive feedback (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e4\u003c/span\u003e):\u003c/p\u003e\u003cp\u003e\u003cem\u003eI would say that it’s set up well. It was just in the corridor where we go for our psycho-ed activities or meetings with providers. The tablets are big enough, so they’re easy to read. (Service user, Gr1)\u003c/em\u003e\u003c/p\u003e\u003cp\u003eOthers described issues that made installation of the “happy or not” terminal somewhat complicated, including distancing measures during the pandemic, sharing space with other services, or lack of a fixed clinic location since they were mainly offering outreach services.\u003c/p\u003e\u003ch2\u003eb) Stakeholder perceptions of how the RLHS protects the rights and privacy of service users\u003c/h2\u003e\u003cp\u003eDigital tools and the shift to measurement-based care were viewed by professionals as inevitable. The key issues around use of digital tools concerned the need to develop competence around technical issues, ethical issues and a desire to put people at ease:\u003c/p\u003e\u003cp\u003e \u003cem\u003eCurrently, there is a strong will at the ministerial level; there is a culture of measurement that comes with a new technology every six months or so … and documents to complete. Then, we mentioned earlier how important it was to convince our supervisors of various elements. (Manager, H2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eFor some service users and family members, confidentiality and data security were important issues:\u003c/p\u003e\u003cp\u003e \u003cem\u003eThis is not a worry, but rather a comment. I would say (what is important is) that the psychological profile stays confidential. This is the only thing that is important for me… (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOthers expressed further concern about equity in the use of technology and about service users who may be left behind:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWell, for sure I have already expressed a certain worry in terms of the technology that is being deployed. It’s a good technology, ‘user friendly’ for people who are still young, with telephones, QR codes, etc. But there are still many young people … who would have a lot of trouble using this, QR codes. (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eService users described the importance of creating safe spaces where they could speak up on issues:\u003c/p\u003e\u003cp\u003e \u003cem\u003eYes, it is important that the patient feels comfortable saying things. There can’t be ‘backlash’ in the interventions if someone complains, and afterward they don’t feel comfortable. (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eA family member reflected on ensuring full representation of service user voices, even when most feedback was positive and service users seemed satisfied, how to prevent biases and attend to the needs of services users less apt to express their opinions, or whose experiences diverged from majority opinion:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI always see that it may be biased in the end, you know. Even if 90% of our clients are satisfied, this may be because people who weren’t satisfied didn’t answer… (Family member, F1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOverall, service users were strongly on board with digital tools, welcoming the “technological leap” (Gr 1, H1), another service user underlining that service users are “in permanent contact with technology, which is instinctive for us.” (Gr1, H2). Moreover, only service users addressed the conditions for successfully integrating digital tools, emphasizing measures that would make their peers more amenable to completing surveys. One noted that service users would have more ideas and be more “mentally prepared” when informed prior to meetings with providers or others that a questionnaire would be offered. They also requested better explanations of questionnaire concepts (Gr1, H2). Others noted the need for service users to better understand the importance of participation:\u003c/p\u003e\u003cp\u003e \u003cem\u003eIt would be important to inform the service user about the purpose of the survey … how services are useful, and to demonstrate that their opinions can make a difference in improving care. It is also important to state this in advance… (Service user, Gr1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eAs another service user emphasized regarding meaningful participation of service users in RLHS data collection:\u003c/p\u003e\u003cp\u003e \u003cem\u003e…the participation of users needs to be meaningful enough that the findings represent something reliable, not just summary, or partial findings that lack credibility. Youth participation is central. (Service user, Gr 2)\u003c/em\u003e \u003c/p\u003e\u003ch2\u003ec) how the RHLS maintains and promotes learning, innovation, and discovery\u003c/h2\u003e\u003cp\u003e Stakeholders across the board were enthusiastic about participation in capacity-building activities. One psychiatrist characterized SARPEP as a “community of psychiatrists and carers meeting for the first time to discuss problems. Before, this did not exist.” (H2)\u003c/p\u003e\u003cp\u003eCapacity-building activities addressed practical issues, reassuring stakeholders about the RLHS:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI find (them) super pertinent because, again, I find that this is where the learning system really comes to life; It’s okay to see that I have a gap, but how do I improve on it? (Manager, H1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e \u003cem\u003eSometimes I say to myself, ‘heck, are we the only ones functioning like this?’ … So, when we met, I said to myself, ‘Okay, my reality isn’t so different from that of other regions or teams.’ So, when we met maybe twice in the past year … I found that it normalized our challenges as well as our successes, and the organization of our services. It was pleasant. (Team leader, F6)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eAll service users felt welcome at capacity-building activities and encouraged to share their ideas:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI found it very respectful. I knew some people when I did the meetings … I wasn’t alone in my corner. I didn’t feel isolated. I felt really accepted by the others. I have only positive points to make … Listen, I gave my opinion many times … I said what I think. I asked some very frank questions to certain people. (Service user, Gr2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eA family member added: “it’s a plus, plus, plus … It was a good experience”. (F2)\u003c/p\u003e\u003cp\u003ePsychiatrists and managers commented on how SARPEP helped promote learning and innovation regarding quality care in EIS. As a psychiatrist suggested:\u003c/p\u003e\u003cp\u003e\u003cem\u003eSARPEP gave legitimacy to the entire implementation process for EIS and shed light on the Québec EIS guidelines written by the Ministry. It’s as if, at least symbolically, it signals a new tendency, a kind of accountability, I would say. It puts the managers in front of a structured project, for which there were objectives to be met … from my point of view, it contributed to the implementation of programs that are well structured, that work well and are useful. (H2)\u003c/em\u003e\u003c/p\u003e\u003cp\u003eFinally, the findings suggest that SARPEP succeeded in creating a learning community, although for newer staff this was not always obvious:\u003c/p\u003e\u003cp\u003e \u003cem\u003eI think there is a nice feeling of community. There was a nice collaboration in meetings. All the EIS representatives spoke about their different realities. A lot of stuff was brought up by patients, psychiatrists, providers, and case managers. (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOnly a couple managers noticed that the early intervention for psychosis community in Québec included many “communities of practice”; and that SARPEP is “one element among many” (F2), showing the importance of effectively integrating SARPEP within the various Québec initiatives:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWith PEPP, we have so many communities of practice. There is AQPPEP, then there are meetings of team leaders with the Ministry’s National Centre of Excellence, there is SARPEP, there are lots of things. I am not sure whether SARPEP brings a community of practice, as such. It is one element among so many others. Nor am I sure whether people can separate very well what belongs to what… (F2)\u003c/em\u003e \u003c/p\u003e\u003ch2\u003ed) how to sustain and improve the RLHS overtime and the future of SARPEP\u003c/h2\u003e\u003cp\u003eStakeholders expressed interest in developing training resources to support continued data collection and in disseminating information to make SARPEP better known in their networks:\u003c/p\u003e\u003cp\u003e \u003cem\u003eBasically, SARPEP should be better known to clinicians, managers, and higher-level managers, even to the general management. (Manager, F1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003e \u003cem\u003eWe must sensitize those responsible for the program in the hospital administration about the importance of this type of activity. So, we really must continue to engage these people in the process. And there, I think it is important to utilize the Québec mental health plan as a lever for SARPEP. (Psychiatrist, H1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eTeam leaders expressed a desire to involve various stakeholders (clinicians, service users, family members) in data collection:\u003c/p\u003e\u003cp\u003e \u003cem\u003eIf clinicians were also more involved in data collection, that would perhaps make more sense, and would become a real team mobilization project. I think it would be easier to get the data, perhaps on a day-to-day or weekly basis, and the entire team could be involved, rather than just having the team leader enter the data. (Team leader, F6)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eSeveral stakeholders favored the extension of SARPEP:\u003c/p\u003e\u003cp\u003e \u003cem\u003eIf we put SARPEP in all Québec EIS, I think that would be good, but, again, having the tools [Informatic technology tools and support] so that utilization will be easier (Manager, F2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOthers, including service users, viewed extending SARPEP as well aligned with their interests:\u003c/p\u003e\u003cp\u003e \u003cem\u003eYes, because it gives patients a certain usefulness when they come. It allows them to regain a clear understanding of the importance of treatment (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eAsked about key learning from the project on how to improve EIS services, service users, family members and team leaders identified integrating peer support:\u003c/p\u003e\u003cp\u003e \u003cem\u003eFor sure, the development of peer support … will be very important for the future development of EIS. (Service user, Gr 2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eOne family member felt somewhat left out of SARPEP, because service users were the primary focus, i.e., “the accent wasn’t broad enough” (F2). Better engagement and inclusion of families was highlighted by managers and families, as “psychosis” impacts the entire family:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWell, do families have a space to propose improvements? I think that this is information we would like to have, as managers anyway ... We have patient partners ... we have, or will have, family partners. (Manager, H2)\u003c/em\u003e \u003c/p\u003e\u003cp\u003eA service user provided the last word on priorities for SARPEP in the EIS:\u003c/p\u003e\u003cp\u003e \u003cem\u003eWhatever the digital tools, the research, most important is that the person remains at the center of the project, that the opinion of the person who receives care continues to be considered, and that s/he be considered a whole person with full potential for recovery. (Service user, Gr 1)\u003c/em\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eRESULTS. EXTRA DATA RELATED TO ADOPTION, IMPLEMENTATION AND MAINTENANCE\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTHEMES\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQUOTES\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdoption:\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFeasibility of integrating indicators, or digital data, into routine care\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eI also ... think they are all important … (Administrator, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…I think that all these indicators are important. But I also believe it would be important to know to what extent the programs are capable of collecting this information in a reliable way. (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eMoi aussi trouve que y sont tous importants (Gestionnaire, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…Ben je pense que tous ces, tous ces items-là sont importants. Mais encore est-il je crois que c’est important de savoir dans quelle mesure les programmes sont capables de collecter cette information de manière … de manière fiable. (Psychiatre, H1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndicators - Access to care\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eFor our young people, I find [important), to take them in at the very beginning… (Team leader, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eIn the first place, I would say the access to care process and the time it takes to access care are important… (Manager, F3)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI really agree, we see that access delays make a difference in the speed of recovery; … there is also a difference in the interventions that are needed when we delay access compared to when we move very quickly. I also find that the process of accessing care is very important … that we ensure fluid and rapid communication and have the information needed to serve the young person – very, very important. (Team leader F4)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI find that ... the delay in accessing care is super important when we talk about the quality of services. (Team leader, F6)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003ePour nos jeunes, j’trouve, d’les prendre au début, début. (Chef d’équipe, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eLe processus d’accès aux soins, puis le délai d’accès aux soins dans un premier temps qui sont importants…\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e(Gestionnaire, F3)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eEt j’suis vraiment d’accord, le délai d’accès on voit vraiment la différence dans la rapidité du rétablissement, puis les interventions nécessaires si jamais on tarde versus si on va très rapidement … Le processus d’accès aux soins, j’trouve que c’est aussi très important. De s’assurer d’une communication fluide, rapide, d’avoir l’info qui faut pour servir le jeune, très, très important\u0026nbsp;» (Chef d’équipe, F4)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…moi j’trouve que … le délai d’accès aux soins est super important quand on parle de qualité de service. (Chef d’équipe, F6)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndicators - staff capacity and competence\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003e…the other thing for me is continuous staff training. We know that there is enormous turnover in our teams, as all throughout the healthcare system; so, it isn’t easy. (Manager, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI see that with the new workers who arrive at the clinic, their need for supervision, for mentoring in terms of this approach. So, I also find this important. (Team leader, F5)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eL’autre chose pour moi c’est la formation continue du personnel. On sait qu’on a énormément de roulement dans nos équipes pour, ben comme dans l’ensemble du système de santé, puis c’est pas facile. (Gestionnaire, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…moi j’vois avec des nouveaux intervenants qui sont arrivés à la clinique tout le besoin de supervision, de mentorat par rapport à cette approche-là aussi. Donc ça aussi j’trouve ça important. (Chef d’équipe, F5)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eImportant indicators - satisfaction and engagement\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eThe satisfaction of young people and their families is, in my view, critical among the indicators that were chosen. (Manager, H2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eFor me, the priority indicators are the delay in access to care, and engagement with the family. This is important because the providers can be working in one direction, while the family has diverging opinions; so, it is better that everyone works in the same direction. (Service user, Gr. 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…to adapt to the young people, to reach them more easily, in a way that best accommodates them. (Team leader, F5)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eLes éléments de satisfaction des jeunes et d’leur famille sont, à mon avis, critiques dans les indicateurs qui ont été choisis. (Gestionnaire, H2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eBen moi c’est ça, mes indicateurs en priorité c’est délais d’accès aux soins, l’engagement avec la famille et l’entourage, pour moi, c’est important parce que les intervenants peuvent travailler dans une direction et la famille peut avoir des opinions divergentes donc on est mieux de travailler tout le monde dans la même direction. (Chef d’équipe, F5)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndicator - dis/satisfaction\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003e…but I think that recovery is a notion that is very personal ... Someone may think that a person has recovered well, whereas for someone else … the person has not recovered that much. (Service user, Gr 2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eJe pense que le rétablissement, c’est difficile de dire que … c’est une notion qui est très personnelle, le rétablissement. Pour quelqu’un, la personne va être bien rétablie et pour un autre, la personne ne sera pas tant que ça. (Utilisateur de service, Gr2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndicator – need for new data\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eWe do a systematic follow-up of the clients … Client evolution is evaluated rigorously, I’m going to know if there was any family intervention, but I won’t necessarily have the number of family interventions. (Team leader, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e… they talk about the patient’s self-rated clinical outcomes, but how about the data on vocational status, financial autonomy, those things… (Team leader, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eMotivation to pursue a career, to have a stable job, university studies. (Service user, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eMaybe their extra-EIS activities, those that don’t take place at the EIS (Service user, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eSubstance use, because more than half of the youth who arrive here use substances. (Team leader, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eL’évolution de la clientèle, ça on le fait de façon rigoureuse. Puis c’est un peu là-dedans où est-ce que j’vais savoir si justement y a eu de l’intervention familiale ou non, mais j’aurai pas le nombre nécessairement d’interventions familiales. (Chef d’équipe, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…y disent l’évolution clinique auto-rapportée par le patient, mais les données sur le statut vocationnel, l’autonomie financière, ces choses-là\u0026nbsp; (Chef d’équipe, F1 p.10)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eLa motivation de poursuivre une carrière, donc avoir un emploi stable, quelque chose à l’université. (Utilisateur de service, Gr1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003ePeut-être les activités extra PEP, donc pas dans le PEP. (Utilisateur de service, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eLa consommation parce qu’y a plus que la moitié de nos jeunes qui arrivent qui consomment. (Chef d’équipe, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndicators – symptomatic and functional outcomes\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eIn this list, there are no indicators of symptom severity and evolution We are talking about an illness that affects functioning, but we do not have the elements that indicate improvement or stagnation in the patient’s level of functioning. (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eOne important element is the treatment algorithms. How can they be implemented in an easy, practical, pragmatic way, and how can they be followed over time? (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eIt would be very, very important, I think, to evaluate program performance. (Psychiatrist, H2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eOn parle d’une maladie qui affecte le fonctionnement. On n’a pas d’éléments non plus qui reflètent un peu l’amélioration ou bien la stagnation de patient sur le plan du fonctionnement. (Psychiatre, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eDonc y a un élément important, c’est les algorithmes de traitement. Comment est-ce qu’on peut le mettre en place de manière facile, pratique, pragmatique et comment est-ce qu’on peut le suivre au cours du temps? (Psychiatre, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003ePuis ça serait très, très important, je pense, pour voir … pour, c’est ça, évaluer la performance d’un programme. (Psychaitre, H2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eImplementation and Maintenance\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDeploying and using technologies\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eI’m going to be honest: given the time at my disposal, I wasn’t able to play around with this questionnaire … But, lets’ look at it in positive way, the reality is that, what it takes to implement this, is manpower. (Team leader, F4)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eJ’vais être honnête, ce questionnaire-là avec le temps que j’avais, j’étais pas capable d’aller jouer dans ça … Mais la réalité c’est que si on regarde de façon positive quoi faire pour implanter ça, ça nous prend … ça prend, ça prend la main-d’œuvre. (Chef d’équipe, F4)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGraphical feedback\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eThis [data and\u003c/em\u003e graphical feedback] \u003cem\u003ebrings concreteness in our everyday actions. (Team leader, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eIt’s also fun for them to see what went well in the previous 3–4 months, what we still need to work on, or to improve … it also allowed us to identify objectives… case managers also expressed satisfaction with this. (Team leader, F5)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…when we see that it was a good month, that we had some positive comments (from the [visual feedback], we transmit that to the team; it’s fun for providers to know that they are appreciated by the clients and this helps morale among the troops. (Team leader, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI liked that this brought us in contact with the provincial government EIS counselor, that it was documenting for our chiefs and managers some of our needs based on the data we collected. (Team leader, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eÇa ramène de la concrétude dans nos actions de tous les jours. Ça, c’est clair. (Chef d’équipe, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eC’est le fun aussi pour eux aussi de voir qu’est-ce qui, qu’est-ce qui a bien été dans les 3–4\u0026nbsp;derniers mois, c’qui nous restait à travailler ou à améliorer aussi. Fait que ça permettait aussi d’identifier des objectifs ... Les intervenants ont signifié aussi leur satisfaction par rapport à ça. (Chef d’équipe, F5)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e...\u003cem\u003equand mettons dans le mois on voit qu’on, que c’était un bon mois, puis on a ben eu des commentaires positifs, ben on le transmet à l’équipe, puis c’est comme le fun pour les intervenants de savoir que y sont appréciés par la clientèle, au niveau du moral des troupes. (Chef d’équipe, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJ’aimais le fait que ça nous ramène quand même à être en contact soit avec notre conseiller au ministère, … de documenter auprès de nos chefs, auprès des administrateurs certains besoins qu’on a de façon un p’tit peu plus … c’est comme, on a documenté en tout cas à partir de données qu’on a colligées. (Chef d’équipe, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSatisfaction questionnaires\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eYou end your meeting, it goes well; then you give [the evaluation form\u003c/em\u003e] \u003cem\u003eto the patient. But it’s a bit embarrassing; it’s like asking to give me positive feedback. (Psychiatrist, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eFait que là tu finis ta rencontre, ça se passe bien, tu lui donnes, mais c’est un peu gênant. Ça va me faire des commentaires positifs. (Psychiatre, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eData collection on service user and family satisfaction\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eThere were some weeks where all the providers talked about it with all the patients at each meeting. We had one or two evaluations. Then, we tried to go in a strategic fashion, because sometimes … it’s the first meeting with patients; they can still be perplexed, symptomatic. So, we were trying to ‘time’ the moment that we talked about it. I’m not sure if this is a mistake, or if it’s really something we could work on. (Psychiatrist, H2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eY a eu des semaines où tous les intervenants en ont parlé avec tous les patients à chaque rencontre. On a eu une ou deux évaluations. Ensuite on a essayé d’y aller de façon stratégique parce qu’effectivement des fois on … c’est première rencontre avec des patients, puis y peuvent être encore perplexe, symptomatiques. Donc là on essayait de timer le moment où on en parlait. On sait pas trop si c’est une erreur ou si c’est vraiment quelque chose sur lequel on pourrait travailler. (Psychiatre, H2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMeasure for client satisfaction using “happy or not” and REDCap satisfaction survey\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eThe ‘happy or not’ for me is playful, and it’s right at the door. So, it’s fun, ‘flashy’. (Manager, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI see that this gives me a very quick idea immediately after an appointment whether people are satisfied. (Manager, H1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eLe Happy or not pour moi est ludique, puis y est vraiment à la porte de sortie. Fait que y est le fun, y flash. (Gestionnaire, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eMoi j’vois d’avoir une idée très rapide après un rendez-vous si les gens sont satisfaits dans l’immédiat. (Gestionnaire, H1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e“Happy or not” and location in the clinic\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eWe installed the famous machine … we couldn’t put it just anywhere because there are also places in the hospital that are multifunctional (…) maybe it isn’t installed in a good place. (Psychiatrist. H2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eWe don’t have the stable offices or waiting rooms. So, it’s complicated to move the big tablet around. We don’t have a fixed place to put it. (Team leader, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…well, I’m speaking for our place, I don’t know about elsewhere, but it is in the waiting room. Not very popular with infection control department. (Administrator, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eNous on l’a installé, la fameuse machine … on pouvait pas la mettre n’importe où parce que nous aussi y a des lieux dans l’hôpital qui sont multifonction ... Fait que on cette machine-là installée, mais peut-être qu’elle est pas installée au bon endroit. (Psychiatre, H2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eQu’on n’a pas de, nécessairement de locaux ou de salle d’attente. On est assez nomade. Fait que ça, c’est compliqué de trainer la grosse tablette, on n’a pas d’endroit fixe où la mettre. (Chef d’équipe, F2)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJ’parle pour chez nous, je sais pas ailleurs, mais y est dans la salle d’attente. Y est pas beaucoup aimé de la prévention des infections. (Gestionnaire, F1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRLHS protects the rights and privacy of service users and maintains equity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003e…we have to pay a lot of attention, introducing things slowly, slowly … step by step, and see what the problems are, what difficulties we encounter in using these tools. We also need to consult with IT services, with the ethics committee, with patients. Everyone needs to be comfortable with this, and we must do it in stages and make sure that we don’t begin with a big project and then it flops and doesn’t work. A few clinicians who are engaged in this type of activities, need to be included, a few patients, and then improve the quality of the experience as we increase our experience… We have been saying this for 20 years, but I hope we will get there. These types of platforms will probably be part of our everyday life in future. (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI think that, yes, it’s how these data are utilized, where they are stored. This is confidential information, that’s it. (Service user, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eBut I’m thinking now, she (daughter) thinks that her cell phone is being hacked, you know, so if she goes to the doctor tomorrow morning, the last thing she wants is to respond on the tablet… (Family member, p. 16)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI am trying to think of someone young, like me, who is in the streets; well, I don’t think they will use it. They have other things to do than to spend 10 minutes filling out a survey with him in the clinic. (Service user, Gr2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e...\u003cem\u003ey va falloir faire très attention c’est-à-dire en introduisant les choses tranquillement, doucement, pas à pas et voir un peu quels sont les problèmes, les difficultés qu’on rencontre dans l’application de ces outils-là. Donc y faut également consulter avec nos services informatiques. Y faut consulter avec le comité d’éthique. Y faut consulter avec les patients. Y faut que tout le monde soit à l’aise avec ça et puis y faut que faire ça par étape et s’assurer que … s’assurer de ne pas commencer avec un grand projet et puis par la suite ça flop et ça ne fonctionne pas. Y faut inclure quelques cliniciens qui sont engagés dans ce genre d’activités, quelques patients et puis par la suite améliorer l’expérience et augmenter notre expérience …. Cela on le dit depuis maintenant 20\u0026nbsp;ans, mais j’espère qu’on va y arriver et ce genre de plateformes vont faire probablement partie de notre quotidien dans l’avenir. (Psychiatre, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJe pense que oui, c’est comment ces données-là sont utilisées, où est-ce qu’elles sont stockées. C’est de l’information confidentielle, c’est ça. (Utilisateur de service, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eMais je suis en train de penser en ce moment, elle pense que son téléphone cellulaire est hacké, la, tu sais … fait que la si elle s’en va au médecin demain matin, répondre sur la tablette c’est la dernière affaire qu’elle veut. (Membre de la famille, p.16)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJ’imagine que mettons, j’essaie de penser à quelqu’un de jeune comme moi, mais qui est dans la rue, bien je ne pense pas qu’ils vont le prendre, il y a d’autres besoins que de prendre 10 minutes pour remplir le sondage dans la clinique. (Utilisateur de service, Gr 2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRHLS maintains and promotes learning, innovation, and discovery\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003e…that helped the team somewhat to decide on what data to collect (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eSometimes I say to myself, ‘heck, are we the only ones functioning like this? …So, when we met, I said to myself, ‘Okay, my reality isn’t so different from that of other regions or teams. So, when we met maybe twice in the past year … I found that it normalized our challenges as well as our successes, and the organization of our services. It was pleasant. (Team leader, F6)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI think that we started there (capacity-building activities). This is the first time that, as a community of psychiatrists, of carers, we are together to discuss these problems. Before that, nothing existed. (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI don’t think we have a sense of community for the managers … Or, maybe it’s ‘me’, because I just arrived… (Manager, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eIt allows us to keep a common thread. It allows exchanges between coordinators, team leaders … It’s like, yes, it keeps you aware of what’s going on elsewhere, and then it feels good to get out of your little belly button. (Team leader, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI think this has surely had an impact, because it offers a sharing forum between clinic … From what I’ve seen, there was more openness, more ideas. There was more communication between the different EIS, and that in addition to the fact that the 2015–2020 mental health plan spoke specifically about family and close ones (“entourage”), so this increased engagement with families. I think that this also helped. (Family member, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eÇa a aidé l’équipe de décider un peu sur quoi, qu’est-ce qu’on collecte un peu dans nos données. (Psychiatrist, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…des fois j’me disais coudonc on est-tu dans le champ, on est-tu tout seul à fonctionner comme. Fait que quand on se réunissait, j’me disais OK ma réalité est pas si différente des autres régions ou des autres équipes. Fait que moi j’ai apprécié quand on se réunissait peut-être… peut-être deux fois dans la dernière année. Mais ça normalisait, j’trouve, autant nos défis que nos bons coups, que l’organisation des services. C’était agréable. (Chef d’équipe, F6)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJe pense que là on commence. C’est la première fois que en tant que communauté de psychiatre, de soignants, de … on est ensemble pour discuter de ces problèmes-là. Avant ça n’existait pas. (Psychiatre, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJ’pense pas qu’on a un sentiment de communauté pour les gestionnaires … Ou c’est peut-être moi parce que j’viens d’arriver... (Gestionnaire, H1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eÇ’a permis de garder un fil conducteur. Ça permettait des échanges justement entre les coordonnateurs, chefs d’équipe … c’est comme oui à l’affût de qu’est-ce qui se passe ailleurs, puis ça fait du bien de sortir de son petit nombril. (Chef d’équipe, F7)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eJe pense que oui, moi. Ça a eu surement un impact parce que ça offre une tribune de partage entre les cliniques ... qu’est-ce que j’ai vu, c’est qu' il y avait plus d’ouvertures d'idées, il y avait plus de communication entre les PEPP, et ça avec le fait que le plan santé mentale de 2015–2020 a parlé spécifiquement de l’entourage, qu’on devrait engager l’entourage, et qu’on devrait élargir nos affaires avec eux, je pense que ça a aidé aussi. (Membre de la famille, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHow SARPEP sustains and improves the RLHS over time\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eBasically, SARPEP should be better known to clinicians, managers, and higher-level managers, even the general directors of our health organizations. I think it has to be recognized …. there has to be an awareness about SARPEP. (Manager, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eI think that … what will really help, is when they rewrite the Québec EIS guidelines. That will be super useful, and we have already been told about it… (Team leader, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…there still remain challenges for our clinic. We don’t fulfill all the requirements of the Québec EIS guidelines.. So, what we would like is that this … helps us perhaps to have a little more weight [credibility] with management to be able to continue to improve … (Manager, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…it (SARPEP) allows the clinic to improve and put forward the concerns of young people, while adapting services to their needs and vision, to their image… To provide the best chances for success, recovery and to always aim for excellence in treatment… all this (Service user, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eWe know that the data are going to be collated so changes can be made (Service user, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eÀ la base que ce soit mieux connu autant des intervenants, des gestionnaires, puis des gestionnaires de plus haut niveau voire même jusqu’aux directions générales. J’pense que faut que ça soit … faut que ce soit … pour être reconnu y faut que ce soit connu. (Gestionnaire, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eMoi j’pense que tout ce projet-là, c’qui va vraiment aider, c’est quand y vont réécrire le cadre de référence des EIS au Québec. Ça va être hyper utile parce que nous on l’a dit. (Chef d’équipe, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e…y reste encore des défis au sein de la clinique. On remplit pas toutes les exigences du cadre de référence. Donc ce qu’on aimerait c’est que ça, ça nous aide peut-être à avoir un peu plus de poids sur la gestion pour pouvoir continuer à améliorer. (Gestionnaire, F1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eÇa permet toujours à la clinique de s’améliorer et de mettre de l’avant la préoccupation des jeunes … et de justement adapter leur service à leurs besoins, à leur image … Donner les meilleures chances de réussite, de rétablissement et de toujours viser l’excellence au niveau des soins, tout ça. (Utilisateur de service, Gr1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eOn le sait que les données vont être colligées pour qu’il y ait des changements. (Utilisateur de service, Gr1)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eHow to improve SARPEP\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003e…often the peer support worker could, from his experience, help people to open up more about their symptoms, to express themselves, which was good. (Service user, Gr 1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eIn our program, we would like to include a patient partner, a peer support worker on the team … I think that could be very interesting for some clients who are a bit ambivalent about some of the proposed interventions. (Manager, F4)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eTo have a family peer support worker in all the clinics! And to meet him/her at the beginning! Because I am working hard on that… (Family member, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003e…souvent, le pair aidant pouvait de son expérience et ça aidait les gens à s’ouvrir davantage sur leurs symptômes et à oser s’exprimer, donc ça, c’est bien. (Utilisateur de service, Gr1)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eDu côté des Laurentides on aimerait bien pouvoir inclure un patient partenaire, un pair aidant dans l’équipe … J’pense que ça pourrait être intéressant pour certains clients qui peuvent être un peu ambivalents face à certaines propositions d’interventions. (Gestionnaire, F4)\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eAvoir un pair-aidant famille dans toutes les cliniques! … Et le rencontrer au début! Parce que là je suis en train de travailler fort…\u0026nbsp; (Membre de la famille, F2)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003e In response to increasing public expectations of evidence-based and patient-centered health care, the RLHS represents a new paradigm for revamping health systems to achieve evidence-based, patient-centered care grounded in safe, timely, efficient, and equitable practices. Yet very few RLHS have been developed or evaluated in the mental health context. The SARPEP project created one of few RLHS in EIS for psychosis worldwide\u003csup\u003e43, 44\u003c/sup\u003e, and the first in Canada.\u003c/p\u003e \u003cp\u003eThis study presented qualitative findings from the mixed-method SARPEP project, describing experiences and insights of stakeholders involved in the design and 2-year implementation of the Qu\u0026eacute;bec RLHS in EIS for psychosis. Findings confirmed that the EIS accepted and highly valued the introduction of this innovative, measurement-based, digital system supporting the routine collection and sharing of program-level data. The RLHS, including capacity-building activities, also met with success in creating a learning community where all stakeholders, particularly service users and families, felt welcome and motivated to target areas of improvement in services.\u003c/p\u003e \u003cp\u003eStudy participants viewed the eight selected key indicators as highly relevant and useful. Access to care stood out as most consequential for recovery, followed by service user and family satisfaction. For some, maintaining quality standards depended upon ensuring service intensity (engaging/training staff; appropriate provider-patient ratios). Participants relied on the RLHS to boost their capacity to capture and monitor precise indicators aligned with evidence-based and recovery-oriented practices (Indicator 7) and patient-reported outcomes (e.g., self-evaluation of recovery dimensions) (Indicator 8). Participants found the selected indicators meaningful (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e2\u003c/span\u003e) but disagreed somewhat on the ease or difficulty of collecting related data (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eIn this figure, + means indicators are considered easy to collect; - not as easy to collect)\u003c/em\u003e \u003c/p\u003e \u003cp\u003eRegarding the three technologies for data collection, aggregation, analysis and visualization, the REDCap digital platform was considered effective overall, although challenges persisted for managers and team leaders around dedicated time and resources for data collection. The much-appreciated graphical feedback allowed them to evaluate clinical performance, celebrate good results with staff, and set new objectives. Uptake of satisfaction questionnaires by service users, whether using \u0026ldquo;happy or not\u0026rdquo; terminals or REDCap, was somewhat weak, provoking considerable stakeholder discussion around how to encourage systematic, patient self-assessment and service evaluation practices. Psychiatrists and clinicians expressed concerns about biased results favoring extreme satisfaction/dissatisfaction with services. Extended discussion by providers, service users and family members addressed issues around competence and comfort with technology, the full representation of service user voices, and equity regarding access to digital tools and skills-building, leaving no service user behind. Service users, families and psychiatrists advised careful consideration of ethics and data security in shifting to a culture of measurement and in using digital technologies. Most crucial was the common concern for service user and family satisfaction, particularly the urgent need for more peer support and family engagement in EIS.\u003c/p\u003e \u003cp\u003eOur findings also reflected the five Lessard, et al\u003csup\u003e40\u003c/sup\u003e dimensions of learning health systems described above. Like other programs inspired by the learning health system vision, SARPEP prioritized safe, high-quality services\u003csup\u003e40, 45, 46\u003c/sup\u003e and better patient outcomes, goals that were achieved according to participants. For instance, all 11 EIS started to collect and share data on the indicators, improved their measurement-based practices, and committed to improve data quality and administrative/clinical practices. Moreover, the deployment of technologies by EIS demonstrated the capacity of the RLHS to support services and stakeholders, including service users and family members, as competent and equal partners in program design and evaluation activities\u003csup\u003e47\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eContrary to the view that successful RLHS implementation implies heavy data collection using sophisticated digital platforms and software, our findings support a growing body of research suggesting that access to an integrated data platform is a necessary, but insufficient, condition for a successful learning health system\u003csup\u003e30, 48\u003c/sup\u003e. Participants placed great importance on the emergence of an active learning community and sense of belonging, spearheaded by SARPEP, that enabled EIS to compare practices and learn from each other\u0026rsquo;s strategies, activities, and procedures. Lessard\u003csup\u003e40\u003c/sup\u003e also proposed a culture of continuous learning, citing transparency, collaboration, teamwork, and innovation as essential\u003csup\u003e49, 50\u003c/sup\u003e. The core elements needed to generate this culture include shared decision-making processes, alignment of stakeholder goals, requisite clinical and analytic expertise, and appropriate principles of governance and leadership\u003csup\u003e51\u003c/sup\u003e. SARPEP stakeholders appreciated this new community for sharing both problems and solutions, while adding credibility to their work and strengthening their representations to decision- and policy-makers.\u003c/p\u003e \u003cp\u003eEmpowering the social dimension of the RLHS involved harnessing the digital infrastructure, designed to support data gathering, sharing, visualization, and learning, to guide a shift in the healthcare ecosystem toward quality improvement and innovation. As findings described, the need for reliable and analyzable health data met with challenges: e.g., making the REDCap platform and \u0026ldquo;Happy or not\u0026rdquo; terminals more accessible and appealing to service users by introducing flexible digital solutions (QR code); technical issues involving data lifecycle management and verification, which ultimately resulted in a lengthening of the data collection-feedback cycles from three to four months to allow sufficient time for implementation; and challenges related to lack of time and resources for service providers, who recognized the need to involve multiple people in real-time data collection. Other studies have described similar challenges\u003csup\u003e52\u0026ndash;55\u003c/sup\u003e. Overall, service users and family members accepted both short and long satisfaction questionnaires well, although their level of use (adoption) varied. Motivation among all stakeholders to participate fully in the RLHS also remained high, fostered by their sense of belonging to a movement nourished by a community of practice, and recognition that the RLHS created the capacity to measure important indicators, and bring improvement.\u003c/p\u003e \u003cp\u003eThe scientific dimension focused on innovations for improved health outcomes\u003csup\u003e30\u003c/sup\u003e. According to Lessard\u003csup\u003e40\u003c/sup\u003e, this dimension incorporates social and technical elements of the learning health system into a \u003cem\u003econtinuous learning circle that moves from data aggregation and analysis to interpretation and practice change\u003c/em\u003e. New data are generated for integration within the learning system. SARPEP promoted discovery, revealing practice gaps (e.g., lack of peer support workers in Qu\u0026eacute;bec EIS), while informing Qu\u0026eacute;bec EIS guidelines related to the anticipated incidence of new cases, by region, which prompted funding adjustments for human resources; also, the modification of clinic opening hours based on available resources while maintaining flexibility and easy patient access for regular appointments, or crises. Supported by the Qu\u0026eacute;bec Ministry, SARPEP will embark on a larger-scale implementation, adding 2\u0026ndash;3 waves of EIS to the original 11 and aiming to include all 33 Qu\u0026eacute;bec EIS within approximately three years.\u003c/p\u003e \u003cp\u003eSARPEP has also inspired new research projects and collaborations. Researchers have proposed a CBT implementation study, using the RLHS infrastructure for data collection on EIS practices and participant evaluations, the LHS media library for training, community of practice knowledge exchange sessions, and service user evaluations. Second is a cognitive remediation project that integrates user self-assessments into the REDCap platform and features a co-designed, stepped care intervention with service users, and capacity training for clinicians. PAIRPEP is a third project on peer support and family peer support implementation involving the entire RLHS system. PAIRPEP engages multiple RLHS stakeholders in partnership with local peer support organizations. The project also features a digital intervention (DIALOG+) for peer support workers and digital training using avatars.\u003c/p\u003e \u003cp\u003eThe ethical dimension in RLHS is often unaccounted for\u003csup\u003e40\u003c/sup\u003e, although preliminary work has led to a proposed ethical framework for RLHS activities\u003csup\u003e56\u003c/sup\u003e. Our study contributes to this area, especially in revealing the strong interest of service users and clinicians in preserving confidentiality and data security. Service users also expressed concerns about inequities in accessing digital technologies due to lack of skills or economic deprivation (e.g., youth homelessness). One peer support worker was assisting service user participation by helping them complete satisfaction and recovery self-assessment questionnaires. Findings strongly suggest that the Qu\u0026eacute;bec RLHS is advancing toward a truly participatory and inclusive EIS ecosystem.\u003c/p\u003e \u003cp\u003eThe strengths and limitations of this study should be noted. Stakeholder engagement and co-design with ongoing input throughout the implementation period was a definite strength, as was data collection using stakeholder-specific focus groups. Member-checking, a full review of study findings and contents by study participants, was also employed. One notable weakness in this qualitative evaluation was that the participants did not include family members currently receiving care in the EIS, although two service user participants were currently receiving services. As our focus in this study was on implementation of the SARPEP project, we considered the inclusion of service users and family members who were research partners as more pertinent to the composition of focus groups. Finally, given the richness of the data we dedicated a full manuscript to the qualitative results, even though mixed methods designs usually entail a simultaneous discussion of both qualitative and quantitative findings (convergent analysis). A future paper presenting a convergent analysis of qualitative and quantitative findings from the SARPEP project is planned, following the completion of further quantitative analyses.\u003c/p\u003e"},{"header":"CONCLUSIONS","content":"\u003cp\u003eThis study described implementation of the first RLHS in early intervention services for psychosis in Canada, confirming the acceptability of the approach for promoting evidence-based care and measurement-based practices in EIS, despite initial challenges in deploying the technology and integrating data collection into routine care. The RLHS creates a welcoming community for learning and improvement. The findings support a range of provincial policies (system integration, healthcare monitoring, health informatics solutions, etc.). Most importantly, SARPEP has advanced our understanding of how the RLHS may be used and implemented in healthcare services and lays the foundations for extending this paradigm to other jurisdictions where existing EIS services are located, in Canada or elsewhere. Sharing common measures used in the RLHS operating in different provinces or countries, as a wider community of practice, may allow us to compare the impact of health policies and provincial health systems on the implementation of evidence-based practices.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eHuman ethics and consent to participate:\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe research ethics board of the Centre de Recherche du Centre Hospitalier de l\u0026rsquo;Université de Montréal approved the study in December 2019 (reference: 19-282 and MP-02-2020-8627), followed by institutional approval from all 11 participating sites. Individuals provided written informed consent before participating in the focus groups. Only service users and family members received compensation, whereas other stakeholders participated during regular work hours.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e: All participants consent in their consent forms for anonymized and non-personal data publication\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability declaration\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and/or analysed during the current study are not publicly available due privacy and confidentiality reasons but could be available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests as defined by BMC, or other interests that might be perceived to influence the results and/or discussion reported in this paper.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding declaration\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis project is funded by \u003cem\u003eFonds de recherche du Qu\u0026eacute;bec-Sant\u0026eacute;\u003c/em\u003e (program name: \u003cem\u003eDonn\u0026eacute;es de recherche en contexte r\u0026eacute;el - Partenariat Innovation-Qu\u0026eacute;bec-JANSSEN - Volet Schizophr\u0026eacute;nie - Cliniques int\u0026eacute;gr\u0026eacute;es en sant\u0026eacute; mentale dans un contexte r\u0026eacute;el\u003c/em\u003e, grant 284385).The aforementioned funding agency (which is the Quebec governmental health research funding agency) was not involved in the design of the study or data collection.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eDr. Ferrari has a salary award from the Fonds de Recherche du Qu\u0026eacute;bec\u0026ndash;Sant\u0026eacute;.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe researchers (AAB, SI, Annie Leblanc, MF, MAR - alphabetical order) designed the study. Researchers (MF, MSS) analyzed the data and developed the standardized code book. At the writing stage, the quotations were translated into English (Judith Sabetti), with oversight by KM and AAB who are fluent in both languages. The principal investigators (AAB, SI, Annie Leblanc, MF, MAR - alphabetical order) reviewed and refined the themes and subthemes after exposure to the various analytical phases, including meetings to discuss previously shared analyses. All authors elaborated drafts of the manuscript and reviewed the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors gratefully acknowledge the service users, family members, and clinical staff working in Quebec early intervention services for their involvement in the SARPEP project and focus groups. The authors also greatly appreciate the contribution of Camille Arbaud, Kevin MacDonald, and Judith Sabetti (alphabetical order) for their support in data collection, analysis, manuscript writing and formatting.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBreitborde NJ, Bell EK, Dawley D, et al. The Early Psychosis Intervention Center (EPICENTER): development and six-month outcomes of an American first-episode psychosis clinical service. BMC Psychiatry Oct 28 2015;15:266.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDixon LB, Goldman HH, Bennett ME, et al. Implementing Coordinated Specialty Care for Early Psychosis: The RAISE Connection Program. Psychiatr Serv Jul 2015;66(7):691\u0026ndash;698.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNorth CS, Simic Z, Burress J. Design, implementation, and assessment of a public comprehensive specialty care program for early psychosis. Journal of Psychiatric Practice 2019;25(2):91\u0026ndash;102.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBertelsen M, Jeppesen P, Petersen L, et al. Five-year follow-up of a randomized multicenter trial of intensive early intervention vs standard treatment for patients with a first episode of psychotic illness: the OPUS trial. Archives of general psychiatry Jul 2008;65(7):762\u0026ndash;771.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCraig TK, Garety P, Power P, Rahaman N, Colbert S, Fornells-Ambrojo M, Dunn G. The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis. BMJ (Clinical research ed) Nov 6 2004;329(7474):1067.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnderson KK, Norman R, MacDougall A, Edwards J, Palaniyappan L, Lau C, Kurdyak P. Effectiveness of Early Psychosis Intervention: Comparison of Service Users and Nonusers in Population-Based Health Administrative Data. Am J Psychiatry May 1 2018;175(5):443\u0026ndash;452.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHarvey PO, Lepage M, Malla A. Benefits of enriched intervention compared with standard care for patients with recent-onset psychosis: a metaanalytic approach. Canadian journal of psychiatry Revue canadienne de psychiatrie Jul 2007;52(7):464\u0026ndash;472.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHumensky JL, Bello I, Malinovsky I, et al. OnTrackNY's learning healthcare system. J Clin Transl Sci Apr 6 2020;4(4):301\u0026ndash;306.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMalla A, McGorry P. Early Intervention in Psychosis in Young People: A Population and Public Health Perspective. Am J Public Health Jun 2019;109(S3):S181-S184.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMalla AK, Norman RM, Joober R. First-episode psychosis, early intervention, and outcome: what have we learned? Canadian journal of psychiatry Revue canadienne de psychiatrie Dec 2005;50(14):881\u0026ndash;891.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIyer S, Jordan G, MacDonald K, Joober R, Malla A. Early intervention for psychosis: a Canadian perspective. The Journal of nervous and mental disease May 2015;203(5):356\u0026ndash;364.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChandra A, E. P, Hodge S, eds. \u003cem\u003eStandards for early intervention in psychosis services \u0026ndash;\u0026thinsp;1st Edition\u003c/em\u003e: Royal College of Psychiatrists, Early Intervention in Psychosis Network (EIPN); 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEarly Psychosis Guidelines Writing Group and EPPIC National Support Program. \u003cem\u003eAustralian Clinical Guidelines for Early Psychosis, 2nd edition update\u003c/em\u003e, 2016, \u003cem\u003eOrygen.\u003c/em\u003e Melbourne: The National Centre of Excellence in Youth Mental Health; 2016.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEhmann T. A Quiet Evolution: Early Psychosis Services in British Columbia. A Survey of Hospital and Community Resources 2003\u0026ndash;2004.. 2004.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOntario Ministry of Health and Long-Term Care. Early Psychosis Intervention Program Standards 2011.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQu\u0026eacute;bec. Minist\u0026egrave;re de la Sant\u0026eacute; et des Services sociaux LDdc. 2017. Cadre de r\u0026eacute;f\u0026eacute;rence. Programmes d\u0026rsquo;interventions pour premiers \u0026eacute;pisodes psychotiques (PIPEP). \u003cem\u003ePeriodical\u003c/em\u003e 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThe EPI Advanced Practice Project (EPI APP). Available at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.earlypsychosis.ca/about-us/\u003c/span\u003e\u003cspan address=\"https://www.earlypsychosis.ca/about-us/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEarly Psychosis Intervention Ontario Network (EPION). Available at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://help4psychosis.ca/\u003c/span\u003e\u003cspan address=\"https://help4psychosis.ca/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAssociation qu\u0026eacute;b\u0026eacute;coise des programmes pour premiers \u0026eacute;pisodes psychotiques (AQPPEP). Available at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://aqppep.ca/aqppep\u003c/span\u003e\u003cspan address=\"https://aqppep.ca/aqppep\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCanadian Consortium for Early Intervention in Psychosis (CCEIP). Available at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.earlypsychosisintervention.ca/schizophrenia-resources-websites#:~:text=CCEIP%20\u003c/span\u003e\u003cspan address=\"https://www.earlypsychosisintervention.ca/schizophrenia-resources-websites#:~:text=CCEIP%20\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003eWEBSITE,associated%20with%20early%20psychosis%20programs. Accessed July 27, 2023.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKotlicka-Antczak M, Podgorski M, Oliver D, Maric NP, Valmaggia L, Fusar-Poli P. Worldwide implementation of clinical services for the prevention of psychosis: The IEPA early intervention in mental health survey. Early Interv Psychiatry Dec 2020;14(6):741\u0026ndash;750.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMaric NP, Andric Petrovic S, Rojnic-Kuzman M, Riecher-Rossler A. Implementation of early detection and intervention services for psychosis in Central and Eastern Europe: Current status. Early Interv Psychiatry Oct 2019;13(5):1283\u0026ndash;1288.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDurbin J, Selick A, Hierlihy D, Moss S, Cheng C. A first step in system improvement: a survey of Early Psychosis Intervention Programmes in Ontario. Early Interv Psychiatry Dec 2016;10(6):485\u0026ndash;493.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBertulies-Esposito B, Nolin M, Iyer S, Malla A, Tibbo P, Banks N, al. e. Implementation of Early Intervention Services for Psychosis in Quebec: A cross sectional study. \u003cem\u003eCanadian Journal of Psychiatry\u003c/em\u003e Canadian Journal of Psychiatry.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNolin M, Malla A, Tibbo P, Norman R, Abdel-Baki A. Early Intervention for Psychosis in Canada: What Is the State of Affairs? Canadian journal of psychiatry Revue canadienne de psychiatrie Mar 2016;61(3):186\u0026ndash;194.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAddington DE, McKenzie E, Norman R, Wang J, Bond GR. Essential evidence-based components of first-episode psychosis services. Psychiatr Serv May 1 2013;64(5):452\u0026ndash;457.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYong PL, Olsen L, McGinnis M. \u003cem\u003eValue in Health Care. Accounting for cost, quality, safety, outcomes and innovation\u003c/em\u003e. Washington, D.C.: The National Academies Press; 2010.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAgency for Healthcare Research and Quality. About Learning Health Systems. Content last reviewed May 2019. Available at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ahrq.gov/learning-health-systems/about.html\u003c/span\u003e\u003cspan address=\"https://www.ahrq.gov/learning-health-systems/about.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGreene SM, Reid RJ, Larson EB. Implementing the Learning Health System: From Concept to Action. Annals of Internal Medicine 2012;157(3):207\u0026ndash;210.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFriedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc Jan 2015;22(1):43\u0026ndash;50.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMenear M, Blanchette MA, Demers-Payette O, Roy D. A framework for value-creating learning health systems. Health Res Policy Syst Aug 9 2019;17(1):79.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbernethy AP, Etheredge LM, Ganz PA, Wallace P, German RR, Neti C, Bach PB, Murphy SB. Rapid-learning system for cancer care. J Clin Oncol Sep 20 2010;28(27):4268\u0026ndash;4274.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDelaney BC, Curcin V, Andreasson A, et al. Translational Medicine and Patient Safety in Europe: TRANSFoRm\u0026ndash;Architecture for the Learning Health System in Europe. Biomed Res Int 2015;2015:961526.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePlatt R, Wilson MG, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network -- improving the evidence of medical-product safety. New England Journal of Medicine 2009;361(7):645\u0026ndash;647.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWallace PJ. Reshaping cancer learning through the use of health information technology. Health Affairs 2007;26(2):w169-177.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFerrari M, Iyer S, LeBlanc A, Roy MA, Abdel-Baki A. A Rapid-Learning Health System to Support Implementation of Early Intervention Services for Psychosis in Quebec, Canada: Protocol. JMIR Res Protoc Jul 19 2022;11(7):e37346.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQu\u0026eacute;bec. Minist\u0026egrave;re de la Sant\u0026eacute; et des Services sociaux. \u003cem\u003eProgramme pour premiers \u0026eacute;pisodes paychotiques. Cadre de r\u0026eacute;f\u0026eacute;rence\u003c/em\u003e; 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGlasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG, Estabrooks PA. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health 2019;7:64.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKreuger RA, Casey MA. \u003cem\u003eFocus groups: A practical guide for applied research\u003c/em\u003e. 5th ed. Thousand Oaks, CA: Sage Publications; 2014.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz A. Architectural frameworks: defining the structures for implementing learning health systems. Implement Sci Jun 23 2017;12(1):78.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBraun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006/01/01 2006;3(2):77\u0026ndash;101.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLincoln YS, Guba EG. \u003cem\u003eNaturalistic inquiry\u003c/em\u003e. Newbury Park, CA: SAGE; 1985.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlanco C, Heinssen RK, Tenhula WN. Public Sector Learning Health Care Systems\u0026mdash;Improving Patient Experience, Workforce Well-Being, and Recovery Outcomes. JAMA psychiatry 2024;81(1):9\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeinssen RK, Azrin ST. A national learning health experiment in early psychosis research and care. Psychiatric Services 2022;73(9):962\u0026ndash;964.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFriedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010;2(57):1\u0026ndash;3.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYong PL, Olsen LA, McGinnis JM, eds. \u003cem\u003eRoundtable on value \u0026amp; science-driven health care\u003c/em\u003e. Washington (DC): National Academies Press; 2010.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeid M, Margolis PA, Opipari-Arrigan L. Engagement, peer production, and the learning healthcare system. JAMA Pediatr Mar 2014;168(3):201\u0026ndash;202.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDixon-Woods M, Campbell A, Chang T, et al. A qualitative study of design stakeholders' views of developing and implementing a registry-based learning health system. Implement Sci Mar 6 2020;15(1):16.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLowes LP, Noritz GH, Newmeyer A, Embi PJ, Yin H, Smoyer WE, Learn from Every Patient' Study G. 'Learn From Every Patient': implementation and early results of a learning health system. Dev Med Child Neurol Feb 2017;59(2):183\u0026ndash;191.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOlsen L, Saunders RS, McGinnis JM, eds. \u003cem\u003ePatients charting the course: Citizen engagement and the learning health system\u003c/em\u003e. Washington (DC): National Academies Press; 2011.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbernethy AP. Demonstrating the learning health system through practical use cases. Pediatrics 2014;134(1):171\u0026ndash;172.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGrant RW, Uratsu CS, Estacio KR, et al. Pre-Visit Prioritization for complex patients with diabetes: Randomized trial design and implementation within an integrated health care system. Contemp Clin Trials Mar 2016;47:196\u0026ndash;201.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNash DM, Brown JB, Thorpe C, Rayner J, Zwarenstein M. The Alliance for Healthier Communities as a Learning Health System for primary care: A qualitative analysis in Ontario, Canada. J Eval Clin Pract Dec 2022;28(6):1106\u0026ndash;1112.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThuraisingam S, Chondros P, Dowsey MM, Spelman T, Garies S, Choong PF, Gunn J, Manski-Nankervis JA. Assessing the suitability of general practice electronic health records for clinical prediction model development: a data quality assessment. BMC Med Inform Decis Mak Oct 30 2021;21(1):297.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWeir CR, Butler J, Thraen I, et al. Veterans Healthcare Administration providers' attitudes and perceptions regarding pragmatic trials embedded at the point of care. Clin Trials 2014;11(3):292\u0026ndash;299.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFaden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep Jan-Feb 2013;Spec No:S16-27.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"health-research-policy-and-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"hrps","sideBox":"Learn more about [Health Research Policy and Systems](http://health-policy-systems.biomedcentral.com/)","snPcode":"12961","submissionUrl":"https://submission.nature.com/new-submission/12961/3","title":"Health Research Policy and Systems","twitterHandle":"@HarpsJournal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4216686/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4216686/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground:\u003c/strong\u003e The early intervention service for psychosis (EIS) model is increasingly implemented. Yet, despite the development of guidelines and standards for EIS in several countries, heterogeneity persists in implementing the essential evidence-based practices. One remedy to these implementation challenges entails the use of a rapid-learning health system (RLHS), an approach that involves ongoing, systematic data collection, analysis and visualization using an electronic platform to design capacity-building activities and promote learning. In 2019, our team implemented SARPEP (\u003cem\u003eSystème Apprenant Rapide pour les Programmes de Premiers Épisodes Psychotiques\u003c/em\u003e), the first Canadian RLHS in EIS. This paper presents findings from the qualitative phase of the mixed-method study that evaluated the feasibility and acceptability of implementing a RLHS in EIS, which was the primary objective of the SARPEP research project.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods:\u003c/strong\u003e We conducted six focus groups, two with service users, and one each with family members, psychiatrists, managers, and team leaders to ensure multistakeholder perspectives. Participants (n=24) were asked to share their experiences with implementation of the RLHS over three years. Focus groups drew on the five dimensions of the RE-AIM framework, as did the quantitative study phase, while Lessard’s five dimensions for a learning health system guided development of the focus group questions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e \u0026nbsp;Reach: invited members from all five stakeholder groups of the RLHS attended focus group meetings and shared their experiences: service users (n=5), family members (n=2), psychiatrists (n=4), managers (n=6), and team leaders (n=7). \u0026nbsp;Effectiveness: all stakeholder groups confirmed the ability of the RLHS to improve the capacity of programs to collect data on core indicators and to promote evidence-based practices. Adoption: \u0026nbsp;stakeholders acknowledged the value of selecting specific indicators for a RLHS and the need to improve technologies for data gathering. They also recognized challenges in deploying and integrating digital platforms into routine care. Implementation and maintenance: all stakeholders elaborated on how the RLHS enabled mutual learning, self-reflection, and shared practices among EIS in Québec.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions: \u003c/strong\u003eStakeholders agreed that a RLHS can promote evidence-based care in EIS. The RLHS created a sense of belonging in a community aiming to learn and improve.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTrial registration: N/A\u003c/strong\u003e\u003c/p\u003e","manuscriptTitle":"Implementing a rapid-learning health system in early intervention services for psychosis: Qualitative evaluation of its feasibility and acceptability","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-29 14:42:53","doi":"10.21203/rs.3.rs-4216686/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-11-02T19:28:36+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-30T21:46:38+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-28T23:12:11+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-24T05:33:22+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"336815823916131640213045740677822417744","date":"2024-10-11T16:17:31+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"140878059810119566157879259245823403047","date":"2024-10-08T12:06:31+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"278878645168889935951761195390576629597","date":"2024-10-07T12:10:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"258025867426477476897320841616473312266","date":"2024-05-20T23:02:14+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-05-20T23:00:49+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-04-22T02:26:47+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-04-22T02:26:47+00:00","index":"","fulltext":""},{"type":"submitted","content":"Health Research Policy and Systems","date":"2024-04-04T08:57:56+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"health-research-policy-and-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"hrps","sideBox":"Learn more about [Health Research Policy and Systems](http://health-policy-systems.biomedcentral.com/)","snPcode":"12961","submissionUrl":"https://submission.nature.com/new-submission/12961/3","title":"Health Research Policy and Systems","twitterHandle":"@HarpsJournal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"83b1fbcf-c1b1-438b-95b6-6ce36c5268aa","owner":[],"postedDate":"April 29th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-03-17T16:09:10+00:00","versionOfRecord":{"articleIdentity":"rs-4216686","link":"https://doi.org/10.1186/s12961-024-01281-w","journal":{"identity":"health-research-policy-and-systems","isVorOnly":false,"title":"Health Research Policy and Systems"},"publishedOn":"2025-03-14 15:58:45","publishedOnDateReadable":"March 14th, 2025"},"versionCreatedAt":"2024-04-29 14:42:53","video":"","vorDoi":"10.1186/s12961-024-01281-w","vorDoiUrl":"https://doi.org/10.1186/s12961-024-01281-w","workflowStages":[]},"version":"v1","identity":"rs-4216686","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4216686","identity":"rs-4216686","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}