Methods of Engaging Interest-Holders in Healthcare Evidence Syntheses: A Scoping Review.

We synthesized data for the whole group of included studies within tables and charts, with brief descriptive statements. For the group of evidence syntheses judged as “green” we summarized key characteristics of individual evidence syntheses and used the ACTIVE framework [ 5 ] to describe key features relating to engagement. We grouped “green” evidence syntheses according to the type of evidence synthesis and produced narrative summaries of what happened at which stage in the review process, providing an overview of the key ways of engaging interest‐holders within different types of evidence synthesis. We held team meetings to reflect on patterns or approaches to engagement, and how reviewers may have made informed choices about the methods of engagement to use within a planned review. We created tables comparing the results of this updated scoping review with the results of the previous scoping review.

Alex Todhunter‐Brown: conceptualization, funding acquisition, methodology, data curation, investigation, formal analysis, visualization, writing – review and editing, writing – original draft, project administration, supervision. Jennifer Petkovic: conceptualization, funding acquisition, methodology, data curation, investigation, writing – original draft, writing – review and editing, project administration, supervision. Christine Chang: methodology, investigation, formal analysis, writing – original draft, writing – review and editing. Ursula Griebler: investigation, formal analysis, writing – original draft, writing – review and editing. Ailish Hannigan: investigation, formal analysis, writing – original draft, writing – review and editing, methodology. Jennifer Hilgart: methodology, investigation, formal analysis, writing – original draft, writing – review and editing. Basharat Hussain: investigation, formal analysis, writing – original draft, writing – review and editing. Janet Jull: investigation, formal analysis, methodology, writing – original draft, writing – review and editing. Christina Koscher‐Kien: methodology, investigation, formal analysis, writing – original draft, writing – review and editing, data curation. Dominic Ledinger: data curation, investigation, formal analysis, writing – original draft, writing – review and editing. Barbara Nussbaumer‐Streit: investigation, formal analysis, writing – original draft, writing – review and editing. Oyekola Oloyede: methodology, data curation, investigation, formal analysis, writing – original draft, writing – review and editing. Eve Tomlinson: methodology, investigation, formal analysis, writing – original draft, writing – review and editing. Shoba Dawson: methodology, investigation, writing – review and editing. Omar Dewidar: methodology, investigation, writing – review and editing. Sean Grant: investigation, writing – review and editing. Lyubov Lytvyn: investigation, writing – review and editing. Thomas W. Concannon: methodology, writing – review and editing. Leonila Dans: methodology, writing – review and editing. Denny John: methodology, writing – review and editing. Zoe Jordan: methodology, writing – review and editing. Evan Mayo‐Wilson: methodology, writing – review and editing. Chris McCutcheon: methodology, writing – review and editing. Francesco Nonino: methodology, writing – review and editing. Danielle Pollock: methodology, investigation, writing – review and editing. Karine Toupin April: methodology, writing – review and editing. Pauline Campbell: conceptualization, funding acquisition, methodology, data curation, writing – review and editing, project administration, supervision. Joanne Khabsa: methodology, writing – review and editing, project administration, supervision. Olivia Magwood: methodology, data curation, investigation, writing – review and editing, project administration, supervision. Vivian Welch: funding acquisition, conceptualization, methodology, writing – review and editing, supervision, project administration. Peter Tugwell: conceptualization, methodology, funding acquisition, writing – review and editing, project administration, supervision.

The protocol for this scoping review is available at https://osf.io/8t6jn (published 23/02/2023). This scoping review is reported according to PRISMA‐ScR Checklist (available on request) and GRIPP2 template (provided as Appendix  2 ).

We used a similar methodological approach to the previous review [ 12 , 16 ], conducting a broad scoping review, followed by a descriptive synthesis of publications that provide the most comprehensive description of methods of engaging interest‐holders. We followed JBI guidance [ 17 , 18 ] for scoping review methods and reported according to guidance for reporting of scoping reviews (PRISMA‐ScR [ 19 ]). Our full protocol was pre‐published and describes all details of our methods [ 15 ]. Deviations from protocol are reported in Appendix  1 . A description of the engagement of interest‐holders in this review is reported using the GRIPP2 tool [ 20 ] in Appendix  2 , with review author identification as interest‐holders provided in Appendix  3 . We conducted a comprehensive database search (MEDLINE (OVID), CINAHL (EBSCO), EMBASE (OVID), PsycInfo (OVID), and SCOPUS). Search strategies are listed in Appendix  4 . Search dates were from January 2016 to February 2024. Publications eligible for inclusion were any documents, in any language, that reported engagement of interest‐holders in evidence syntheses (for detailed definitions see protocol [ 15 ]). This included: Evidence syntheses relating to any health and/or social care topic that reported engagement of interest‐holders, including those conducted to inform guidelines (where these report explicit methods). Evidence syntheses could address any type of healthcare‐related question (e.g., intervention effectiveness, prevalence, diagnostic test accuracy, patient experiences, volume and nature of evidence, evidence gaps), within any context (i.e., any geographical location or specific setting). Methods studies, commentaries, or supplementary material or accompanying articles to evidence syntheses in which the engagement of interest‐holders is described. Evidence syntheses relating to any health and/or social care topic that reported engagement of interest‐holders, including those conducted to inform guidelines (where these report explicit methods). Evidence syntheses could address any type of healthcare‐related question (e.g., intervention effectiveness, prevalence, diagnostic test accuracy, patient experiences, volume and nature of evidence, evidence gaps), within any context (i.e., any geographical location or specific setting). Methods studies, commentaries, or supplementary material or accompanying articles to evidence syntheses in which the engagement of interest‐holders is described. We excluded publications from 2015 or earlier, abstracts with insufficient information, and protocols for systematic reviews. As we had multiple reviewers applying eligibility criteria, decision support trees were used to support consistent application of eligibility criteria and reasons for exclusion (see Appendix  5 ). Search results were screened in Covidence, following removal of duplicates. Two independent reviewers (from a team of 20 reviewers) applied the selection criteria to titles/abstracts, and then to full papers. Disagreements were assessed by a third reviewer, with discussion between reviewers if necessary to reach a decision. Two independent reviewers (from a team of 17 reviewers) extracted data on all included publications using the data extraction tool within Covidence, with disagreements resolved through discussion. Data extraction and coding included: Bibliographic information. Type of publication (i.e., methods paper, systematic review, other) Stated aim/objective For systematic reviews: ◦ Topic/focus of systematic review ◦ Methodological focus/study methodology ◦ Type of evidence synthesized ◦ Why were they engaged (aim of engagement) ◦ Who was engaged? (categorized as patients or public members (including carers or family members) or other interest‐holders (professionals); number of interest‐holders engaged) ◦ Where were they engaged? ◦ When were interest‐holders engaged? (Phase of review) ◦ How were interest‐holders engaged Bibliographic information. Type of publication (i.e., methods paper, systematic review, other) Stated aim/objective For systematic reviews: ◦ Topic/focus of systematic review ◦ Methodological focus/study methodology ◦ Type of evidence synthesized ◦ Why were they engaged (aim of engagement) ◦ Who was engaged? (categorized as patients or public members (including carers or family members) or other interest‐holders (professionals); number of interest‐holders engaged) ◦ Where were they engaged? ◦ When were interest‐holders engaged? (Phase of review) ◦ How were interest‐holders engaged Topic/focus of systematic review Methodological focus/study methodology Type of evidence synthesized Why were they engaged (aim of engagement) Who was engaged? (categorized as patients or public members (including carers or family members) or other interest‐holders (professionals); number of interest‐holders engaged) Where were they engaged? When were interest‐holders engaged? (Phase of review) How were interest‐holders engaged Full details of data extraction domains and coding are available in Appendix  6 . These domains and codes were built on the previous scoping review [ 12 ], the ACTIVE Framework [ 5 ], and the lived experiences of interest‐holders from the MuSE Consortium who contributed to the protocol development. In addition, the independent reviewers made a judgment of the comprehensiveness of the description of methods of engagement, using a “traffic light” categorization system developed and used for the previous review [ 12 , 16 ]. Publications judged to provide a comprehensive description of one (or more) method or approach to interest‐holder engagement in a single evidence synthesis were coded as “Green”; evidence syntheses judged to provide a brief or partial description were coded as “Amber” and those providing few details were coded as “Red.” Full definitions of the traffic light categorizations and examples from the previous scoping review are provided in Table  1 . Where there were disagreements between “Amber” and “Red” categorizations, the decision was made by the first author (A. T.‐B.). Where one reviewer had categorized an evidence synthesis as “Green” and another had selected “Amber” or “Red,” consensus was reached through discussion between members of the review team. Traffic‐light categorization of comprehensiveness of description of interest‐holder engagement. Studies judged to be in the “green” category for the comprehensiveness of the description of methods of engagement had additional data extraction and coding, including details relating to: How were interest‐holders invited to be engaged? Who was involved (Type of interest‐holders, based on the 11 “P's” [ 7 ]; PROGRESS‐Plus characteristics [ 25 ]; geographical location; country income level) What happened when? (Methods, timing, amount of engagement at different review stages; see protocol for categories and definitions) Level of engagement (see protocol for categories and definitions) Ethical approval/statement Acknowledgment (e.g., authorship) and compensation (e.g., payment of expenses, financial compensation, other incentives) Tool or method of reporting engagement Processes to facilitate engagement Reflexivity Disclosure of conflicts of interest How were interest‐holders invited to be engaged? Who was involved (Type of interest‐holders, based on the 11 “P's” [ 7 ]; PROGRESS‐Plus characteristics [ 25 ]; geographical location; country income level) What happened when? (Methods, timing, amount of engagement at different review stages; see protocol for categories and definitions) Level of engagement (see protocol for categories and definitions) Ethical approval/statement Acknowledgment (e.g., authorship) and compensation (e.g., payment of expenses, financial compensation, other incentives) Tool or method of reporting engagement Processes to facilitate engagement Reflexivity Disclosure of conflicts of interest These data were extracted by 1 reviewer and checked by a second reviewer (with 12 reviewers working in pairs). Appendix  6 provides further details of data extraction domains and codes. In line with guidance for scoping reviews [ 26 ], we did not conduct a formal assessment of the methodological quality or risk of bias of the studies included in this review.

Figure  1 summarizes the results of the search. We identified 302 publications (with 311 references) [ 5 , 12 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 , 152 , 153 , 154 , 155 , 156 , 157 , 158 , 159 , 160 , 161 , 162 , 163 , 164 , 165 , 166 , 167 , 168 , 169 , 170 , 171 , 172 , 173 , 174 , 175 , 176 , 177 , 178 , 179 , 180 , 181 , 182 , 183 , 184 , 185 , 186 , 187 , 188 , 189 , 190 , 191 , 192 , 193 , 194 , 195 , 196 , 197 , 198 , 199 , 200 , 201 , 202 , 203 , 204 , 205 , 206 , 207 , 208 , 209 , 210 , 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , 219 , 220 , 221 , 222 , 223 , 224 , 225 , 226 , 227 , 228 , 229 , 230 , 231 , 232 , 233 , 234 , 235 , 236 , 237 , 238 , 239 , 240 , 241 , 242 , 243 , 244 , 245 , 246 , 247 , 248 , 249 , 250 , 251 , 252 , 253 , 254 , 255 , 256 , 257 , 258 , 259 , 260 , 261 , 262 , 263 , 264 , 265 , 266 , 267 , 268 , 269 , 270 , 271 , 272 , 273 , 274 , 275 , 276 , 277 , 278 , 279 , 280 , 281 , 282 , 283 , 284 , 285 , 286 , 287 , 288 , 289 , 290 , 291 , 292 , 293 , 294 , 295 , 296 , 297 , 298 , 299 , 300 , 301 , 302 , 303 , 304 , 305 , 306 , 307 , 308 , 309 , 310 , 311 , 312 , 313 , 314 , 315 , 316 , 317 , 318 , 319 , 320 , 321 , 322 , 323 , 324 , 325 , 326 , 327 , 328 , 329 , 330 , 331 , 332 , 333 , 334 , 335 ] that met our criteria for inclusion. Results of the search.

Evidence syntheses are central to evidence‐informed practice, providing reliable and credible information to support healthcare decisions, policies, and future research [ 1 , 2 ]. Evidence syntheses identify and compile data that address specific research questions using a variety of rigorous methodological approaches [ 3 ]. There is increasing recognition towards engaging a range of interest‐holders (as defined in Box  1 ) in healthcare evidence syntheses as they can enhance the relevance and applicability of the findings while addressing barriers to evidence implementation [ 10 , 11 ]. However, uncertainty remains regarding optimal methods to ensure meaningful engagement to produce evidence syntheses that are most useful, relevant, and accessible [ 12 ]. Evidence syntheses: Research studies that synthesize the research evidence to address health care‐related questions. They use rigorous, explicit, and transparent methods. There are a range of different types of evidence syntheses; definitions of quantitative systematic reviews, qualitative evidence syntheses, rapid reviews, realist reviews, scoping reviews, mixed method reviews, and living reviews are provided elsewhere [ 4 ]. Interest‐holders: “groups with legitimate interests in the health issue under consideration. The interests arise and draw their legitimacy from the fact that people from these groups are responsible for or affected by health‐related decisions that can be informed by research evidence.” This term has been introduced to replace the word “stakeholder” which may be perceived as disrespectful due to colonial connotations, and to replace other alternatives (e.g., knowledge users, consumers, partners, etc.) as all of these are considered to have limitations; full justification and explanation is provided elsewhere [ 6 ]. Eleven broad groups of interest‐holders have been identified (referred to as the “11 P's” [ 7 ]). These include patients and caregivers, the public, providers of care, policy makers, program managers, payers of health research, payers of health services, peer review editors, product makers, producers and commissioners, and principal investigators. These have been fully defined elsewhere [ 4 ]. Within this paper, the nine categories of providers of care, policy makers, program managers, payers of health research, payers of health services, peer review editors, product makers, producers and commissioners, and principal investigators are sometimes collectively referred to as “professionals” to denote their employed roles, while the two categories of patients and caregivers and public are collectively referred to as “patients/public.” Engagement: A bi‐directional relationship, or collaboration, between interested people and groups and a research team that results in informed decision‐making about the selection, conduct, and use of research [ 8 , 9 ]. We use this term in preference to alternatives which are in common usage in some parts of the world (e.g., involvement). It is important to note that the term involvement (often “patient and public involvement” or PPI) is used in other parts of the world (e.g., UK) with the same meaning. Engagement in research refers to research being carried out “with” or “by” interested groups (rather than research carried out “about” or “for” interested people or groups). Co‐production and other terms starting with co‐ (e.g., co‐creation) are considered to be related to, but to potentially go beyond, engagement. Definitions of key terms related to engagement, and notes relating to our preferred terms, are presented elsewhere [ 4 ]. Interest‐holder recruitment: Two broad approaches to the identification and recruitment of interest‐holders to be engaged in an evidence synthesis have been defined. We use these terms: Open recruitment: provision of “opportunities for involvement through advertisement to the general population, allowing anyone to volunteer to get involved. Open recruitment may result in ‘fixed’ membership, where, once group members have volunteered, the membership remains the same, or in ‘flexible’ membership, where different people attend different events or contribute to different activities” [ 5 ]. Closed recruitment: strategies which “focus on inviting only specific people to participate. Closed strategies include invitation of known individuals or recognized experts, recruitment from membership of an existing group, or purposive sampling to achieve representation of people with key pre‐determined characteristics, experience or expertise” [ 5 ]. To address this uncertainty, the MuSE Consortium is conducting a series of evidence syntheses to collate the most up‐to‐date evidence relating to engagement of interest‐holders. An overview of this work has been presented elsewhere [ 4 , 7 ]. Definitions of key terms used within this series of papers relating to interest‐holder engagement in evidence syntheses have been published in full elsewhere [ 4 , 6 ] and are briefly summarized in Box  1 . A previous scoping review has collated 291 evidence syntheses, published up until 2016, which described the engagement of interest‐holders and explored the range of methods of engagement [ 12 ]. Given key developments in this field since this previous scoping review, including the establishment of best practice standards and increasing mandatory requirements from research funders for patient and/or public engagement [ 13 , 14 ], one of the MuSE evidence syntheses was this planned update of the previous scoping review [ 7 , 15 ].

Our previous scoping review [ 15 ] included 291 publications; our updated search has identified a further 302 publications. Details of the 291 publications are provided in the supplementary material to the previous scoping review paper [ 15 ]. Tables summarizing the results from both the updated review and the previous review are provided in Supporting Information  S3 .

This scoping review has collated and provided an overview of 302 publications that describe the engagement of interest‐holders in a single evidence synthesis. We have described key details relating to the engagement in 51 evidence syntheses, which were judged to provide the most comprehensive description of engagement. Box  2 provides an overview of the findings of this scoping review, in relation to each of the pre‐stated review objectives. Overview of key findings in relation to each pre‐stated review objective Objective 1. Identify and collate resources that describe the engagement of interest‐holders in evidence syntheses We identified 302 publications, published since the previous scoping review, which describe the engagement of interest‐holders in a single evidence synthesis. Most publications (272/302, 90%) reported interest‐holder engagement in a single evidence synthesis. The remaining 10% (30/302) are publications providing an account or description of engagement, or methods of engagement, relevant to evidence. We judged 19% (51/272) of evidence syntheses to provide a comprehensive description of engagement, and we have described these, providing examples of how interest‐holders have been engaged in a range of evidence syntheses. Objective 2. Describe the methods that have been used to engage interest‐holders in evidence syntheses (see objectives 2a–2h): Objective 2a. Who was engaged? Of 272 evidence syntheses in which there was interest‐holder engagement, around three‐quarters involved patients and/or carers, while 17% did not involve patients and/or carers, and for the remainder, this information was unclear. For the “green” evidence syntheses, judged to have the most comprehensive reporting, 84% (43/51) engaged patients/caregivers, although it was most common to engage both patients/caregivers and other interest‐holders (professionals)(30/51, 59%). The professional interest‐holders were most likely to be providers (95% of publications reporting this information, 33/35), but engagement of principal investigators, producers/commissioners, program managers, policy makers, and payers was also reported. Less than half of the “green” evidence syntheses described any demographic characteristics of the interest‐holders, with small numbers presenting some information about gender, age, place or residence, occupation, race/ethnicity/culture, disability, and/or education. Objective 2b. How were they invited to be engaged? The method of recruitment tended to be very poorly reported. Even in the “green” evidence syntheses, with the most comprehensive reporting, one‐third (17/51) did not clearly report how interest‐holders were invited to be engaged. Most commonly, interest‐holders were recruited using a “closed” group (30/51, 59%), while few (3/51, 6%) reported use of an “open” approach to recruitment. Objective 2c. Where were they engaged (geographical and cultural context)? Most interest‐holders were engaged from English‐speaking countries with over three‐quarters of all evidence syntheses (215/272, 79%) from the United Kingdom, the United States, Canada, or Australia, and 84.3% (43/51) of the “green” publications from Europe or North America. Objective 2d. Why were they engaged (what was the aim of engagement)? The stated aim of engagement (if provided) was often specifically related to gaining perspectives of interest‐holders on the topic of the evidence synthesis. Around one‐fifth (60/272) of the evidence syntheses focussed on evidence relating to mental, behavioral, or neurodevelopmental disorders, and 14% (38/272) focussed on neoplasms. Sixteen percent (45/272) focussed on a specific intervention (but not population), and around 12% (32/272) had a focus on research or research methods. However, in many cases, the aim of engagement was expressed in relation to the roles and activities in which interest‐holders were engaged, and to the stage in the evidence synthesis process (see Objectives 2e and 2f below). Objective 2e. What did they do (how they were engaged)? Objective 2f. When in the review process were they engaged? Information extracted from all identified 272 evidence syntheses was limited to the stage of the review process and the focus of the evidence synthesis. In 42% (113/272), there was a “top and tail” approach, while in around one quarter (67/272) of evidence syntheses, the stage of the review process was unclear. Exploration of the 51 “green” evidence syntheses demonstrated that engagement was reported at all stages and within all review activities, but was most common at the stages of interpretation of findings and data synthesis. As above, this was frequently a “top and tail” approach. Generally, the engaged interest‐holders attended meetings, supplemented with emails/electronic communication, and formal research methods were rarely used. Objective 2g. What level of influence, or control, did they have in decision‐making? We made judgments on the potential level of influence or control for the 51 “green” evidence syntheses. Most commonly, interest‐holders were judged to have some influence over decisions and outcomes of each activity. For a small number of evidence syntheses, there was engagement in activities for which it was judged that the interest‐holders may have had overall control over final decisions (possibly sharing equal power with the research team). There were no obvious patterns in relation to different forms of engagement, methods used, or level of control during different stages of an evidence synthesis. Objective 2h. Were there processes, strategies, or tools to support engagement and evaluation of impact? We explored reported processes, strategies, or tools for the 51 “green” evidence syntheses. Twelve percent (6/51) used the ACTIVE framework, and 22% (11/51) the GRIPP2 checklist. A range of other checklists, tools, or frameworks were used by individual evidence syntheses. One‐third (16/51) of evidence syntheses were judged to provide some reflexivity on the process of engagement. In over half of these evidence syntheses (27/51), interest‐holders were co‐authors, but there was substantial variation in the roles and responsibilities of interest‐holder co‐authors. Supporting Information  S3 provides results from the previous scoping review and this updated scoping review. The previous review identified 291 evidence syntheses that described interest‐holder engagement, of which 60% were judged to provide few details and/or an inadequate description of the method of engagement. Of the 272 newly identified evidence syntheses, only 29% were judged to provide similarly few details. This suggests that the quality of reporting of interest‐holder engagement may have improved in evidence syntheses published from 2017 onwards. However, while there was some increase in the proportion of evidence syntheses considered to provide a comprehensive account of the methods of engagement (from 11% to 19%), the majority (52%) of the new evidence syntheses in the updated review were judged to fall in the “amber” category. Only 29% of evidence syntheses in the original review clearly engaged patients/carers/family members as interest‐holders, but this increased to 74% amongst the newly identified evidence syntheses, indicating a potential change in practice over time, with the engagement of patients or public becoming more common amongst those evidence syntheses which report interest‐holder engagement. The original review noted inadequate reporting of key features, even amongst the “green” evidence syntheses, which were considered to provide the most comprehensive reporting. This pattern of inadequate reporting for features such as financial compensation and ethical approval remains common. However, a small number of the new “green” evidence syntheses did report comprehensive information, supported by the ACTIVE Framework and/or GRIPP2 guidance (e.g., [ 5 , 20 ]). These evidence syntheses suggest that the use of existing tools can enhance the quality and consistency of reporting. Compared to the previous review, we observed an increase (from 6.8% to 41.4% of included evidence syntheses) in the use of a “top and tail” approach. A possible explanation for this is that the concept of engaging interest‐holders using a “top and tail” approach was first described within the ACTIVE framework [ 5 ], which was published after the previous review (building on the results of that review [ 12 ]); this may have prompted more researchers to adopt this approach to engagement when conducting evidence syntheses. The findings of this scoping review align with descriptions of poor reporting of interest‐holder engagement within other types of health research [ 343 ]. A recent evaluation of interest‐holder engagement in 360 RCTs (2015–2025) found that, in the 18% that reported on engagement, descriptions of “specific roles and contributions were generally broad and vaguely described” [ 344 ]. Of the RCTs reporting engagement, only 15% (10/64) were judged to have “highly detailed” descriptions, with 45% (29/64) having “moderately detailed” and 39% (25/64) “not detailed” descriptions [ 344 ]. This broadly aligns with our judgments of 19% “green,” 52% “amber,” and 29% “red,” suggesting similar patterns of poor reporting of descriptions of engagement in RCTs and evidence syntheses. Similar to our findings, Vanneste et al. [ 344 ] found that engagement could either occur “throughout” or within specific stages of research planning, conduct, or reporting. However, in contrast to our finding that engagement was most common during the data analysis/synthesis and interpretation of findings stages of an evidence synthesis, within RCTs, engagement was most common during the development phase, suggesting that the study design is influencing decisions about when to engage people. The conclusions from this evaluation of RCTs align with the recommendations supported by our scoping review, focussing on the necessity for clear, detailed reporting and use of standardized reporting tools [ 344 ]. Vanneste et al. [ 344 ] also reported an inconsistency in the reporting of engagement in the protocol and final report; this is something that we did not explore, and that may merit further investigation in relation to evidence syntheses. The evidence that we have synthesized demonstrates that there are many different approaches to engagement and that interest‐holders have been involved at all stages of evidence syntheses. Despite the review team hypothesizing potentially different patterns of engagement between types or the focus of evidence synthesis, we found no evidence of such patterns. For example, we considered that realist reviews and scoping reviews may be likely to have unique patterns of engagement as interest‐holder engagement is a recognized methodological component for these types of evidence syntheses. The lack of observable patterns and high level of heterogeneity in approaches to interest‐holder engagement in evidence synthesis could be due to a range of factors, such as the review topic and question (and existing knowledge in that field), the funding/resources available and timelines, and the values, knowledge, and experience of the research team. While we found no generalizable evidence about how different approaches may impact the evidence synthesis, we did find that it is most common to: Form a “closed” group of interest‐holders (where interest‐holders are invited to take part, and these individuals form the membership of a group which contributes at one or more stage(s) of the review). Engage both patients/caregivers and other interest‐holders (professionals). Engage interest‐holders at the start and/or end stages of an evidence synthesis. Engage interest‐holders in multiple activities throughout the review process (e.g., activities to clarify review question or inclusion criteria, identify outcomes, interpret findings, and/or plan dissemination). Engage interest‐holders in “meetings,” without using any formal methods of engagement, and supplement these using email communication. Enable interest‐holders to have some influence over decisions and outcomes for specific activities, with the sharing of power reported less commonly. Have one or more interest‐holders as a co‐author. Form a “closed” group of interest‐holders (where interest‐holders are invited to take part, and these individuals form the membership of a group which contributes at one or more stage(s) of the review). Engage both patients/caregivers and other interest‐holders (professionals). Engage interest‐holders at the start and/or end stages of an evidence synthesis. Engage interest‐holders in multiple activities throughout the review process (e.g., activities to clarify review question or inclusion criteria, identify outcomes, interpret findings, and/or plan dissemination). Engage interest‐holders in “meetings,” without using any formal methods of engagement, and supplement these using email communication. Enable interest‐holders to have some influence over decisions and outcomes for specific activities, with the sharing of power reported less commonly. Have one or more interest‐holders as a co‐author. While the above lists what we found to be most common in relation to interest‐holder engagement in evidence synthesis, it is important to highlight the considerable variability between the examples we have explored. The extent of this may be exaggerated by the inconsistency in terminology used to describe the individual approaches to engagement. Reflecting across the different reported approaches, in relation to “what happened,” rather than the terms used within publications to describe this (e.g., a “PPI group,” a “reference group,” etc.), we can observe some distinguishable approaches to who may be engaged, and how they are recruited. These include: 1. Small number, often one or two, of people with lived experience are engaged as “co” researchers, working in partnership with the review team (in a manner which could be described as “co‐production”). These interest‐holders are often co‐authors on publications (and may be co‐grant‐holders on review funding). In many cases, the people engaged are known to the review team and have had previous engagement experience. 2. A group of interest‐holders, often in the range of 6–12 people, but sometimes more (maximum of 28 people within a group within the “green” evidence syntheses [ 270 ]), formed specifically as an “advisory” or “reference” group or panel for the evidence synthesis. The recruitment is generally closed, with the review team aiming to form a group that includes interest‐holders with specific knowledge, expertise, or experience. This group could be: a. A “patient and public” group, comprising only people with lived experience or the families, carers, or representatives of those with lived experience. b. A mixed group, comprising both patients/public and professional interest‐holders (most often “providers” of healthcare). 3. Open engagement of interest‐holders, often for a single workshop or activity (e.g., a survey), in which interest‐holders are engaged but no “group” is formed. This approach to engagement is often conducted in addition to one of the above approaches and is likely to have a specific aim in relation to the evidence synthesis (e.g., to select the outcomes, to explore the interpretation of findings). Small number, often one or two, of people with lived experience are engaged as “co” researchers, working in partnership with the review team (in a manner which could be described as “co‐production”). These interest‐holders are often co‐authors on publications (and may be co‐grant‐holders on review funding). In many cases, the people engaged are known to the review team and have had previous engagement experience. A group of interest‐holders, often in the range of 6–12 people, but sometimes more (maximum of 28 people within a group within the “green” evidence syntheses [ 270 ]), formed specifically as an “advisory” or “reference” group or panel for the evidence synthesis. The recruitment is generally closed, with the review team aiming to form a group that includes interest‐holders with specific knowledge, expertise, or experience. This group could be: a. A “patient and public” group, comprising only people with lived experience or the families, carers, or representatives of those with lived experience. b. A mixed group, comprising both patients/public and professional interest‐holders (most often “providers” of healthcare). A “patient and public” group, comprising only people with lived experience or the families, carers, or representatives of those with lived experience. A mixed group, comprising both patients/public and professional interest‐holders (most often “providers” of healthcare). Open engagement of interest‐holders, often for a single workshop or activity (e.g., a survey), in which interest‐holders are engaged but no “group” is formed. This approach to engagement is often conducted in addition to one of the above approaches and is likely to have a specific aim in relation to the evidence synthesis (e.g., to select the outcomes, to explore the interpretation of findings). Further, it is clear that during the planning of interest‐holder engagement, a number of decisions are made by the teams responsible for the planning and/or conduct of the evidence synthesis. These decisions include consideration of several factors, including (but not limited to) the stages of the evidence synthesis at which interest‐holders will be engaged, the roles the interest‐holders will have, and, implicit to this, the level of control that interest‐holders may hold over decision‐making in relation to specific parts of the evidence synthesis process. The decisions that are made in relation to these domains may be linked to the engagement approach that is selected. Conversely, within some published evidence syntheses, it appears possible that the approach to engagement was selected first, and this then dictated the level of control and stages of the evidence synthesis at which interest‐holders were involved. The potential relationship between when interest‐holders are engaged, the level of control that they have over decisions relevant to the evidence synthesis, and the approach to engagement is illustrated in Figure  2 (“ACTIVE Tool” to select engagement approach). This figure provides a potential tool to support the selection of approaches to engagement within an evidence synthesis. The examples identified in this scoping review demonstrate that it is common to engage interest‐holders both “throughout” the evidence synthesis process (i.e., the “top” horizontal route in Figure  2 ) and also within specific evidence synthesis activities (i.e., the “bottom” horizontal route in Figure  2 ). Thus, there may be more than one “approach” to engagement selected for a single evidence synthesis. ACTIVE tool to inform the selection of different approaches to engagement. This illustrates potential relationships between when interest‐holders are engaged, the level of control they have over decision making, the approach to engagement which is used, and how this may relate to the level of engagement (using previously defined terms [ 5 ]). Taken from left to right, this provides a possible route to inform the selection of different approaches to engagement. Our scoping review provides an overview of the current state of interest‐holder engagement in evidence syntheses and gives descriptions of different examples of interest‐holder engagement in evidence syntheses. However, this scoping review did not explore the impact or usability of different engagement methods and, consequently, is not able to provide specific recommendations relating to “best” ways of engaging interest‐holders in evidence syntheses. We only included publications in which interest‐holders had an active role during the planning, conduct, or reporting of an evidence synthesis; we did not include publications reporting on strategies to inform interest‐holders about the evidence synthesis, in which interest‐holders received information only. Sharing of information by review teams with interest‐holders could arguably be considered a useful form of engagement, but this was beyond the scope of our work. Our objective focussed on healthcare evidence syntheses, but our pre‐stated eligibility criteria explicitly included evidence syntheses relating to social care topics; this scoping would have been strengthened by greater clarity in this regard. Although we implemented a robust and comprehensive search of electronic databases and used two independent reviewers at all stages of study selection, a key limitation of this scoping review is that we are unlikely to have comprehensively identified all relevant publications. This is because poor (and absent) reporting and inconsistent terminology mean we will not have identified some evidence syntheses in which interest‐holders were engaged. While this is a limitation of our scoping review data, we believe that our strategy of selecting the evidence syntheses with the most “comprehensive” description of interest‐holder engagement is a strength. This subset of “green” evidence syntheses is not designed to be comprehensive, nor to reflect quality of engagement, but rather to provide examples from which lessons can potentially be learned in relation to methods of engaging interest‐holders within evidence syntheses. A strength of this updated scoping review was that we were able to build on the findings of and learning from the previous version of this scoping review. Thus, we extracted data according to domains identified by the ACTIVE Framework [ 5 ], as well as those considered important by a wide range of interest‐holders within the MuSE consortium who contributed to the development of the updated protocol. However, despite the clearly defined domains of interest for our scoping review, findings and conclusions were limited by poor reporting within the included publications. Importantly, we did find evidence of improvements in reporting between the publications identified by the previous and updated reviews, for example, in reporting details of the interest‐holders (previous version was unclear for 30%; updated version unclear for 10%), stage of evidence synthesis (previous version was unclear for 48%; updated version unclear for 25%). This improvement in reporting is also reflected in the proportion of publications judged to provide few or inadequate details (“red”), which decreased from 60% in the previous version of this review to 29% for this updated version. While these findings suggest an improvement in reporting, it is not possible to determine if there have been any changes in the absence of reporting (i.e., evidence syntheses in which there was engagement, but for which there is no information reported). However, despite the observed improvements in reporting, even amongst the publications judged to provide the most comprehensive description (i.e., “green” publications) there remain some domains for which reporting seriously limits conclusions being drawn; for example, of the “green” publications, less than half provided any information about ethics or ethical approval (56% not reported), payment of expenses (78% not reported), other forms of compensation (75% not reported), equity‐related characteristics of interest‐holders (51% reported no information on any domains of PROGRESS‐Plus). One‐third of the “green” publications reported that there was interest‐holder engagement “throughout” the evidence synthesis, but did not provide any further details; this limits our understanding of the activities that interest‐holders were engaged in. Similarly, while over half of the publications list an interest‐holder as a co‐author, many do not detail the specific roles of the interest‐holder co‐authors. New mandatory requirements from many funders and journals for reporting of patient and public engagement could be an important factor in changes in reporting of engagement over time. Exploration of the journals that published the “green” publications could potentially provide some insight into factors that influence comprehensiveness of reporting (see Table  9 ). However, without systematic consideration of issues such as the number of evidence syntheses published and journal word count, any interpretation must be cautious. Almost two‐fifths (9/51, 18%) of “green” papers were published in the journal Health Expectations ; this journal focusses on publishing articles directly relating to patient and public involvement and engagement, and reporting of engagement in published studies is mandatory, providing a logical explanation for the large proportion meeting our criteria for providing comprehensive descriptions of engagement. Only slightly fewer (8/51, 16%) were published in the NIHR Journals Library, where reports often have large word counts and where there is a mandatory requirement for all publications to “explain how patients and the public have been involved in the study outside of being study participants” [ 345 ], with recommendations to use the GRIPP2 reporting tool [ 20 ]. The next two journals publishing the greatest numbers of “green” evidence syntheses (three each) both explicitly support engagement, although reporting is not mandatory. The remaining 26 journals all published one or two “green” evidence syntheses; we were unable to identify any information relating to reporting of interest‐holder engagement within author guidelines for any of these journals. This evidence supports the value of journals mandating reporting of interest‐holder engagement and/or having explicit statements or policies supporting the concept of interest‐holder engagement. Further systematic exploration of author guidelines, journal reporting requirements, and comprehensiveness of the reporting may be beneficial in informing approaches to improve reporting in future publications. A key limitation of our work is that we were dependent on the reporting provided in publications, many of which had a primary aim of reporting the findings of a specific evidence synthesis. Not only does this mean that our conclusions were limited by the lack of information provided by authors on interest‐holder engagement, but it also means that we are unable to make any assessment of whether the reported engagement is an accurate representation of what happened. Further, under‐reporting of engagement will mean that we will not have identified all evidence syntheses in which there was engagement. We also recognize that there is a risk that some articles may have overstated the amount and influence of patient and public involvement, and that the interest‐holder engagement may have been largely tokenistic in evidence syntheses where reporting is poor. We explored whether information on equity‐related characteristics of the engaged interest‐holders was reported. However, we did not explicitly gather data on whether review teams specifically sought to engage interest‐holders with diverse characteristics, experiences, or opinions. A strength of our findings is that we identified publications reporting interest‐holder engagement from at least 30 different countries. However, a large majority are from English‐speaking and/or high‐income countries, potentially limiting the applicability of findings to other contexts and affecting the generalizability of findings.

As stated in our protocol [ 15 ], our goal was to update a previous scoping review that collated and summarized information published up to 2016 relating to the different ways in which interest‐holders have been engaged in evidence syntheses [ 12 ]. The objectives of this updated scoping review were to: 1. Identify and collate resources that describe the engagement of interest‐holders in evidence syntheses; 2. Describe reported methods to engage interest‐holders in evidence syntheses, including: a. Who was engaged (type and characteristics of interest‐holder)? b. How were they invited to be engaged? c. Where were they engaged (geographical and cultural context)? d. Why were they engaged (what was the aim of engagement)? e. What did they do (how they were engaged)? f. When in the review process were they engaged? g. What level of influence, or control, did they have in decision‐making? h. Were there processes, strategies, or tools to support engagement and evaluation of impact (e.g., training, ethical approval, compensation, reporting frameworks, evaluation strategies)? Identify and collate resources that describe the engagement of interest‐holders in evidence syntheses; Describe reported methods to engage interest‐holders in evidence syntheses, including: a. Who was engaged (type and characteristics of interest‐holder)? b. How were they invited to be engaged? c. Where were they engaged (geographical and cultural context)? d. Why were they engaged (what was the aim of engagement)? e. What did they do (how they were engaged)? f. When in the review process were they engaged? g. What level of influence, or control, did they have in decision‐making? h. Were there processes, strategies, or tools to support engagement and evaluation of impact (e.g., training, ethical approval, compensation, reporting frameworks, evaluation strategies)? Who was engaged (type and characteristics of interest‐holder)? How were they invited to be engaged? Where were they engaged (geographical and cultural context)? Why were they engaged (what was the aim of engagement)? What did they do (how they were engaged)? When in the review process were they engaged? What level of influence, or control, did they have in decision‐making? Were there processes, strategies, or tools to support engagement and evaluation of impact (e.g., training, ethical approval, compensation, reporting frameworks, evaluation strategies)? In addition, we sought to compare the results from the previous review with those arising from the newly identified evidence syntheses within the updated review.

Our international team from the MuSE consortium has updated a previous scoping review, bringing together up‐to‐date evidence relating to interest‐holder engagement in evidence syntheses. We provide an overview of 302 publications and describe in more detail the methods used by 51 publications, which we judged to provide the most comprehensive description of methods of interest‐holder engagement. Many approaches to engagement have been used, with no evidence of patterns according to different types or focus of evidence syntheses. Interest‐holders have been engaged at all stages of the evidence synthesis process. We conclude that there may be a potential relationship between when interest‐holders are engaged, the level of control they have over decision‐making, and the approach to engagement that is used. This information may usefully inform the selection of different approaches to engagement in evidence synthesis. Conclusions arising from this scoping review are limited by poor reporting of interest‐holder engagement in evidence syntheses, and improvements in transparent reporting are essential. Most evidence syntheses engaging interest‐holders originate from a small number of high‐income countries, potentially limiting the generalizability of the findings. Our team considers the following recommendations to be supported by the findings of this scoping review. Those conducting evidence syntheses should adhere to relevant standards and principles related to good practice in engagement in research (e.g., UK Standards for Public Involvement [ 13 ]), involve key interest‐holders as early as possible, and pre‐plan their engagement approach, with clear terms of reference relating to roles, responsibilities, and issues such as compensation and acknowledgment. Additionally, they should consider the topic, resources, and timeline when planning engagement, as well as the types and characteristics of interest‐holder and their relationship to underrepresented or marginalized groups. This aligns with a growing awareness of the importance of considering these personal characteristics, both when engaging interest‐holders in research [ 346 ] and when conducting evidence syntheses [ 347 , 348 ]. The ACTIVE Framework may be used to guide decisions on who to involve and at what stages, and our newly proposed ACTIVE Tool (Figure  2 ) may support the selection of an appropriate engagement approach, but testing of this is required. When reporting an evidence synthesis, it is essential that key elements of interest‐holder engagement are transparently documented. The ACTIVE Framework [ 5 ] and GRIPP2 Guidance [ 20 ] provide useful resources to support reporting. We encourage journals to proactively mandate reporting of interest‐holder engagement, as seen in Cochrane's recent requirement for reporting consumer engagement in systematic reviews [ 349 ]. We recommend that reporting guidelines, such as PRISMA [ 350 ] and PRISMA extensions [ 351 ], be updated to include the reporting of interest‐holder engagement, covering who was engaged, at what stages, and what activities or decisions they contributed to. It is important that the roles of all co‐authors, including interest‐holders, should be reported following the CREDIT contributor taxonomy [ 352 ], and that the conflicts of interest (if any) of all interest‐holders are disclosed. Recommendations for future research include the development of tools to support the planning and conduct of interest‐holder engagement in evidence syntheses and research on the impact and usability of different engagement approaches. Future research should also assess the effects on engaged individuals, research teams, project resources, timelines, and the overall evidence synthesis and dissemination process. It is important that funding bodies are cognizant of and that research teams have access to adequate resources to support meaningful interest‐holder engagement in evidence syntheses. These recommendations are summarized in Box  3 . Recommendations for engagement of interest‐holders in evidence syntheses. Those conducting evidence syntheses should: ∘ Adhere to relevant standards and principles relating to engagement in research (e.g., [ 13 , 338 ]). ∘ Engage key interest‐holders (even if this is just 1 or 2 people) at the earliest opportunity. ∘ Pre‐plan their engagement approach. ∘ Consider the evidence synthesis topic, resources, and timeline when planning engagement. ∘ Consider using the ACTIVE Framework [ 5 ] as a tool to support decisions relating to who to involve, at what stages in the review, and in what way, and the ACTIVE Tool (Figure  2 ) to support selection of an approach to engagement of interest‐holders within a specific evidence synthesis. ∘ Consider factors relating to diversity, equity, and inclusion when planning and engaging interest‐holders. ∘ Have access to adequate resources to support interest‐holder engagement (funding bodies should provide appropriate resources to enable meaningful engagement). Recommendations relating to the reporting of engagement of interest‐holders in evidence syntheses. ∘ It is essential that evidence synthesis authors report key elements of interest‐holder engagement within final reports. The ACTIVE Framework [ 5 ] and GRIPP2 Guidance [ 20 ] may be useful in guiding this. Journals should be proactive in mandating this reporting. Cochrane has recently introduced new mandatory reporting of consumer involvement within Cochrane systematic reviews [ 349 ]. ∘ Reporting guidelines aimed at enhancing the quality and transparency of healthcare evidence syntheses, including PRISMA [ 350 ] and PRISMA extensions [ 351 ], should be updated to include the reporting of interest‐holder engagement. ∘ Conflicts of interest (if any) of interest‐holders should be reported. ∘ Roles of all co‐authors, including those who are interest‐holders, should be reported according to the CREDIT contributor taxonomy [ 352 ]. Recommendations for research. There is a need to: ∘ Develop tools to further support the planning and conduct of interest‐holder engagement in evidence syntheses; these should focus on practical elements of engagement. ∘ Conduct research to explore the impact and usability of different engagement approaches, considering the impact on the individuals engaged, as well as the research team, project resources, and timelines, and the final evidence synthesis and dissemination activities. ∘ Consider how project resources impact decisions around interest‐holder engagement, and how to prioritize engagement activities when resources are limited. Recommendations for engagement of interest‐holders in evidence syntheses. Those conducting evidence syntheses should: ∘ Adhere to relevant standards and principles relating to engagement in research (e.g., [ 13 , 338 ]). ∘ Engage key interest‐holders (even if this is just 1 or 2 people) at the earliest opportunity. ∘ Pre‐plan their engagement approach. ∘ Consider the evidence synthesis topic, resources, and timeline when planning engagement. ∘ Consider using the ACTIVE Framework [ 5 ] as a tool to support decisions relating to who to involve, at what stages in the review, and in what way, and the ACTIVE Tool (Figure  2 ) to support selection of an approach to engagement of interest‐holders within a specific evidence synthesis. ∘ Consider factors relating to diversity, equity, and inclusion when planning and engaging interest‐holders. ∘ Have access to adequate resources to support interest‐holder engagement (funding bodies should provide appropriate resources to enable meaningful engagement). Adhere to relevant standards and principles relating to engagement in research (e.g., [ 13 , 338 ]). Engage key interest‐holders (even if this is just 1 or 2 people) at the earliest opportunity. Pre‐plan their engagement approach. Consider the evidence synthesis topic, resources, and timeline when planning engagement. Consider using the ACTIVE Framework [ 5 ] as a tool to support decisions relating to who to involve, at what stages in the review, and in what way, and the ACTIVE Tool (Figure  2 ) to support selection of an approach to engagement of interest‐holders within a specific evidence synthesis. Consider factors relating to diversity, equity, and inclusion when planning and engaging interest‐holders. Have access to adequate resources to support interest‐holder engagement (funding bodies should provide appropriate resources to enable meaningful engagement). Recommendations relating to the reporting of engagement of interest‐holders in evidence syntheses. ∘ It is essential that evidence synthesis authors report key elements of interest‐holder engagement within final reports. The ACTIVE Framework [ 5 ] and GRIPP2 Guidance [ 20 ] may be useful in guiding this. Journals should be proactive in mandating this reporting. Cochrane has recently introduced new mandatory reporting of consumer involvement within Cochrane systematic reviews [ 349 ]. ∘ Reporting guidelines aimed at enhancing the quality and transparency of healthcare evidence syntheses, including PRISMA [ 350 ] and PRISMA extensions [ 351 ], should be updated to include the reporting of interest‐holder engagement. ∘ Conflicts of interest (if any) of interest‐holders should be reported. ∘ Roles of all co‐authors, including those who are interest‐holders, should be reported according to the CREDIT contributor taxonomy [ 352 ]. It is essential that evidence synthesis authors report key elements of interest‐holder engagement within final reports. The ACTIVE Framework [ 5 ] and GRIPP2 Guidance [ 20 ] may be useful in guiding this. Journals should be proactive in mandating this reporting. Cochrane has recently introduced new mandatory reporting of consumer involvement within Cochrane systematic reviews [ 349 ]. Reporting guidelines aimed at enhancing the quality and transparency of healthcare evidence syntheses, including PRISMA [ 350 ] and PRISMA extensions [ 351 ], should be updated to include the reporting of interest‐holder engagement. Conflicts of interest (if any) of interest‐holders should be reported. Roles of all co‐authors, including those who are interest‐holders, should be reported according to the CREDIT contributor taxonomy [ 352 ]. Recommendations for research. There is a need to: ∘ Develop tools to further support the planning and conduct of interest‐holder engagement in evidence syntheses; these should focus on practical elements of engagement. ∘ Conduct research to explore the impact and usability of different engagement approaches, considering the impact on the individuals engaged, as well as the research team, project resources, and timelines, and the final evidence synthesis and dissemination activities. ∘ Consider how project resources impact decisions around interest‐holder engagement, and how to prioritize engagement activities when resources are limited. Develop tools to further support the planning and conduct of interest‐holder engagement in evidence syntheses; these should focus on practical elements of engagement. Conduct research to explore the impact and usability of different engagement approaches, considering the impact on the individuals engaged, as well as the research team, project resources, and timelines, and the final evidence synthesis and dissemination activities. Consider how project resources impact decisions around interest‐holder engagement, and how to prioritize engagement activities when resources are limited. The MuSE Consortium is currently developing guidance for interest‐holder engagement [ 7 ], which will build on the findings and recommendations of this review; the planned guidance has the potential to provide clear evidence‐informed recommendations to support both conduct and reporting in this field.

Alex Todhunter‐Brown: Co‐lead of Cochrane Coproduction Methods Group (unpaid role). Christine Chang: Participation in US Food and Drug Administration, Drug Safety Oversight Board (unpaid role); Member of the Board of Trustees of Guidelines International Network (payment for travel to meetings received). Janet Jull: Canada Research Chair (some funding for Canadian Institute of Health Research); grant funding received from Canadian Institutes of Health Research, Social Sciences and Humanities Research, and New Frontiers Research (all Canadian Tri‐Council funders). Dominic Ledinger: PhD scholarship from Gesellschaft für Forschungsföderung Niederösterreich m.b.H., project number FTI23‐D‐042; PhD topic: Co‐production and knowledge user involvement in evidence synthesis. Sean Grant: Spouse is an employee of Eli Lilly and owns stock. Thomas W. Concannon ‐ recipient of grants or contracts from: Centers for Disease Control and Prevention/NIOSH and National Institutes of Health/NCAT. Zoe Jordan: Royalties received from Wolters Kluwer Health for the sale of JBI tools and resources to support evidence based decision making. Evan Mayo‐Wilson: Grant funding received from Agency for Healthcare Research and Quality (AHRQ), National Institutes of Health (NIH), Knowledge Works Global (subcontract with Patient Centered Outcomes Research Institute (PCORI)). Payment/Honoraria from National Institutes of Health; support for travel/attending meetings from Monash University. Danielle Pollock: Grant funding from Stillbirth CRE /Stillbirth Foundation. Peter Tugwell: Consulting fees from Reformulary Group (Providing independent medical consultation professional services to the firms listed in this section); I am [unpaid] Chair of the Management Group of a registered non‐profit independent medical research organization, OMERACT, whose goal is to improve and advance the health outcomes for patients suffering from musculo skeletal conditions. OMERACT receives arms‐length funding from 11 companies (Abbvie, Astra Zenaca, Aurinia, BMS, Centrexion, GSK, Horizon PharmaInc, Janssen, Novartis, Pfizer & Sparrow). Other authors declare no conflicts of interest.

Characteristics of the included studies are provided in the Table of included studies, Supporting Information  S1 , and a brief summary is described below. Two hundred and seventy‐two publications (272/302, 90.1%) reported engagement of interest‐holders in a single evidence synthesis. The remaining publications (30/302, 9.9%) did not describe engagement in a single evidence synthesis but were judged to have some relevance to interest‐holder engagement in evidence syntheses (referred to as “methods” papers). Fourteen of the 272 evidence syntheses reporting engagement in a single evidence synthesis were also considered to provide information that met our criteria for “methods” papers. Consequently, within this scoping review, we include 302 unique publications, of which 272 report engagement in a single evidence synthesis and 44 of which are “methods” papers (see Table  2 ). Types of included papers. Includes 14 publications which are also counted within the above Evidence Synthesis categories, taking the total publications categorized to over 100%. Almost three‐quarters (200/272, 73.5%) of the evidence syntheses involved patients or public members (including carers or family members) within the evidence synthesis process (with or without other professional, interest‐holders), while 16.9% (46/272) involved other interest‐holders only (e.g., health professionals, academic experts, representatives of patient organizations). In around one‐tenth of the included evidence syntheses (26/272, 9.6%), it was not clear who the interest‐holders engaged in the review were, and whether this included patients or the public (see Table  3 ). Interest‐holders engaged. Most evidence syntheses engaged interest‐holders from English‐speaking countries, with 40.1% (109/272) from the United Kingdom, 19.5% (53/272) from the United States, 13.2% (36/272) from Canada, and 6.2% (17/272) from Australia. Seventeen (17/272, 6.2%) evidence syntheses each engaged interest‐holders from multiple countries, of which three included interest‐holders from African countries (see Table  4 ). Countries from which interest‐holders engaged. Almost two‐thirds (169/272, 62.1%) of the included evidence syntheses were judged to be focussed on one of the International Statistical Classification of Diseases and Related Health Problems 11th Revision (ICD‐11) categories. Most frequently, this was “mental, behavioural or neurodevelopmental disorders” (60/272, 22.1%) and neoplasms (38/272, 14.0%). Evidence syntheses that did not fit one of the ICD‐11 categories included those focussed on a specific intervention (45/272, 16.5%) or research methods (32/272, 11.8%). Almost one‐tenth (26/272, 9.6%) were unable to be categorized within any of these groups and focussed on, for example, areas such as service delivery, teaching, data protection, and criminal justice (see Table  5 ). Focus of the included evidence syntheses. Note: 01–24 are ICD‐11 categories [ 334 ]. Most commonly (113/272, 41.5%), interest‐holders were involved in a “top and tail” approach, with engagement both at the start of the review process, during the planning stage, and at the end, interpreting the results after the evidence had been synthesized, but the stage of the review process at which interest‐holders were involved was unclear in around one quarter of evidence syntheses (67/272, 24.6%) (see Table  6 ). Stage of the evidence synthesis process at which interest‐holders were engaged. Table  7 shows the assigned judgments of the comprehensiveness of the description of the method or approach to engagement. In total, 29.0% (79/272) of the included evidence syntheses were judged to provide few or inadequate details (“red”), 52.2% (142/272) judged to provide a brief or partial description (“amber”), and 18.8% (51/272) judged to provide a comprehensive description of one, or more, method(s) or approach(es) to engagement (“green”). Comprehensiveness of description of method or approach to involvement. Table  8 provides an overview of engagement described within the 51 evidence syntheses categorized as “green.” These comprise 10 quantitative systematic reviews, 8 qualitative and mixed‐method evidence syntheses, 13 scoping reviews, 14 realist reviews, 3 overviews of reviews, and 3 publications that each reported more than one single evidence synthesis. The 51 evidence syntheses were published in 30 unique journals (see Table  9 ). Narrative summaries, grouped according to type of evidence synthesis (Appendix  7 ), bring together descriptions of who was engaged, how they were recruited, what happened at what stage of an evidence synthesis, and the level of control that engaged interest‐holders had over the evidence synthesis. Key characteristics are briefly described below. Overview of 51 “green” papers. Journals in which “green” evidence syntheses were published. Patients, caregivers, family members, representatives of patient/caregiver organizations, or general members of the public (abbreviated to “patients/public” below) were engaged in most of the evidence syntheses (43/51, 84.3%). For over half (30/51, 58.8%) of the “green” evidence syntheses, other interest‐holders (professionals) were engaged alongside the patients/public members, although around one‐quarter (13/51, 25.5%) only engaged patients/public members. For a small number (6/51, 11.8%), there were only professional interest‐holders engaged. For two of the “green” evidence syntheses, patients/caregivers were engaged, but it was unclear if there were also professionals engaged. Table  10 summarizes the types of interest‐holders engaged and provides publication references. Summary of interest‐holders engaged in “green” evidence syntheses. We attempted to categorize the type of interest‐holder using the 11 “P's” [ 7 ] (see Box  1 ). Over two‐thirds of “green” evidence syntheses (68.6%, 35/51) provided some information about the type of professional interest‐holders. Of those reporting this information, most commonly the professional interest‐holders were “providers” (95.3%, 33/35), but engagement of interest‐holders who were researchers (“principal investigators”) was also common (60.0%, 21/35). The engagement of producers/commissioners, program managers, and policy makers was reported in 22.9% (8/35), 20.0% (7/35), and 8.6% (3/35) of “green” evidence syntheses, respectively, whilst one evidence synthesis clearly reported the engagement of a payer of health research, and none reported engagement of peer‐reviewed journal editors. Several “other” interest‐holders, whom it was challenging to categorize using the 11 Ps, were reported to be engaged; these included people involved in education and law, care home managers, administrators, and students. The Table of Included studies (Supporting Information  S1 ) provides the author description of engagement, including, where provided, the aim or goal of engagement. Generally, the aim of engagement was articulated in relation to the roles and activities that the interest‐holders were engaged in, and to the stage in the evidence synthesis process, and it was challenging to synthesize these in relation to the aim of engagement. Commonly, broad terms and phrases were used, for example: “to gain feedback” [ 102 ], “to help define the scope” [ 117 ], “to interpret” [ 175 ], “to get an expert opinion” [ 178 ], “to discuss findings” [ 222 ]. Several “green” evidence syntheses referred to the goal of including the voice or perspectives of the interest‐holders, for example: “informed by both patient and healthcare professional perspectives” [ 236 ]; “to ensure patient voice is heard throughout the review” [ 153 ]. Although an overall aim was often not provided, aims were commonly stated in relation to the tasks or activities that the interest‐holders were engaged in, for example: “to comment on preliminary themes, and guide final analysis, to identify areas not covered by the literature)” [ 153 ], “to ensure that our findings were relevant to the people who would eventually use them” [ 206 ], “prioritize clinically relevant outcomes” [ 72 ]. We explored whether the demographic characteristics of the interest‐holders engaged were reported according to the PROGRESS‐Plus domains [ 25 ]. Half of the “green” evidence syntheses (51.0%, 26/51) did not report any characteristics of the interest‐holders engaged, and none of them reported any information relating to domains of religion, socioeconomic status, or social capital. Over a quarter (27.5%, 14/51) provided some information about gender, 23.5% (12/51) about age and 21.6% (11/51) about place of residence. One‐fifth (19.6%, 10/51) reported occupation, but this often related to the professional interest‐holders. Only 9.8% (5/51) reported any information about race/ethnicity/culture, 9.8% (5/51) about disability and 5.9% (3/51) about education. In one‐third of “green” evidence syntheses (17/51, 33.3%) it was not clearly reported how interest‐holders were recruited. In 58.8% (30/51) there was a “closed” recruitment strategy (see Box  1 ). The membership of the closed group was formed by individually inviting interest‐holders to the group (14/51, 27.5%), engaging members of an existing group (8/51, 15.7%), or using a purposeful sampling approach (8/51, 15.7%). The recruitment approach was only considered to be “open” in 3/51 (5.9%), with opportunities for any interest‐holder(s) to volunteer to contribute. One “green” evidence synthesis used multiple recruitment approaches, comprising both open and closed strategies, for engaging interest‐holders in different parts of their process. The countries in which engagement was conducted are reported in Table  8 . In most, these were European (28/51, 54.9%) or North American (15/51, 29.4%) countries, with few in East Asian/Pacific countries (5/51, 9.8%). In three evidence syntheses (3/51, 5.9%), engagement occurred across several different countries. Almost half of the “green” evidence syntheses (24/51, 47.1%) did not report any information relating to ethics or ethical approval. A quarter (13/51, 25.5%) reported that ethical approval was sought and approved, and 17.3% (14/81) reported that ethical approval was not required. Almost all (12/13, 92.3%) “green” evidence syntheses that sought ethical approval involved patients, carers, and/or family members, while the type of interest‐holder was unclear for the remaining evidence synthesis. Table  11 summarizes information relating to payment and other forms of compensation given to interest‐holders. Over three‐quarters of “green” evidence syntheses did not report whether expenses were paid to engaged interest‐holders (40/51, 78.4%), with less than one‐fifth clearly reporting that out‐of‐pocket expenses were paid (9/51, 17.6%), and two evidence syntheses (2/51, 3.9%) explicitly stating that interest‐holders were not paid. Similar proportions report information relating to other forms of compensation (including financial payments, vouchers, or other rewards provided to interest‐holders), with three‐quarters not providing any information (39/51, 76.5%), one‐fifth reporting that some form of compensation was provided (11/51, 21.5%), and one stating that no compensation was provided (1/51, 2.0%). Summary of information relating to compensation provided to interest‐holders reported in 51 “green” papers. Twelve percent of “green” evidence syntheses (6/51, 11.8%) used or referred to the ACTIVE Framework, 21.6% (11/51) used the GRIPP/GRIPP2 checklist. Other checklists/tools used included the Concannon et al.'s 7Ps framework [ 335 ], “Linkage and exchange model” [ 336 ], UK Standards for Public Involvement [ 13 ], Saskatchewan Center for Patient‐Oriented Research level of engagement tool (PORLET) [ 337 ], CIHR Guiding Principles for Patient Engagement Framework [ 338 ], an Involvement Matrix [ 339 ], and a range of tools specific to evidence synthesis methodology (e.g., RAMESES [ 340 ]; COREQ [ 341 ]). Table  12 summarizes at what stage in the evidence synthesis process there were reported engagement activities within each of the “green” evidence syntheses. Engagement was most frequently reported at the stage of interpretation of findings (39/51, 76.5%). Next most frequent stage at which engagement was reported was during the data synthesis stage (9. Analyze/synthesize data; 31/51, 60.8%) and at the beginning of the evidence synthesis (1. Develop question (26/51, 51.0%) and 2. Plan methods (21/51, 41.2%)). It was common for interest‐holders to be engaged at both the beginning and end stages of an evidence synthesis, that is, using a “top and tail” approach [ 5 ]. In addition to the activities reported at specific evidence synthesis stages, around one‐third of the evidence syntheses described one or more interest‐holders who were engaged “throughout,” but without detailing specific activities. Summary of when the ES engagement was reported to occur within the 51 “green” papers. Note: The darker the green shade, the greater the proportion of ES reporting engagement at that stage of an ES. Appendix  8 provides a more detailed synopsis of the stages in the evidence synthesis process at which there was engagement. This detailed breakdown of activities shows that there was engagement reported within every activity, which may be part of an evidence synthesis, although only one “green” evidence synthesis specifically described engagement during the activity of “run search.” We judged that one‐third of “green” publications (31.4%, 16/51) reported on their interest‐holder engagement using a process of reflexivity, providing a reflective discussion relating to the engagement activities that had been undertaken during the evidence synthesis. All evidence syntheses using the GRIPP2 reporting tool [ 20 ] were considered to provide some reflexivity. Almost all “green” evidence syntheses (96.1%, 49/51) reported conflicts of interest of the authors, but only one‐third (33.3%, 17/51) reported any conflicts of interest of the engaged interest‐holders. Where conflicts of interest of interest‐holders engaged were reported, this generally occurred when all of those engaged were co‐authors. In over half (52.3%, 27/51) of the “green” evidence syntheses, at least one interest‐holder was a co‐author. The role and responsibilities of a co‐author varied considerably between different evidence syntheses; some co‐authors worked as equal members of a review team, and other co‐authors contributed to specific, and often limited, tasks only (e.g., commenting on written drafts of manuscripts). Often, in addition to having one or two patient/public co‐authors, there were additional interest‐holders engaged in specific tasks throughout the evidence synthesis process. Supporting Information  S2 provides an overview of the different forms of engagement, methods used to facilitate engagement, and the level of control that interest‐holders were judged to have during the different stages/activities at which there was engagement in the 51 “green” evidence syntheses. A brief summary of how interest‐holders were engaged in the most common types of evidence syntheses is provided here (for more details, see Appendix  7 ): Quantitative systematic reviews ( n  = 10): In four evidence syntheses, interest‐holders were involved in multiple activities throughout the review process [ 64 , 158 , 182 , 300 ]; in three, engagement occurred during the early stages [ 72 , 179 , 282 ]; and three adopted a “top and tail” approach [ 93 , 164 , 173 ]. Commonly, activities comprised meetings (both face‐to‐face and online) and email communication, but other activities included larger workshops, ranking of outcomes, and surveys. Qualitative and mixed method evidence syntheses ( n  = 8): Five of the eight evidence syntheses reported that they had continuous engagement throughout, with four of these also having other engagement activities for specific evidence synthesis tasks [ 136 , 153 , 222 , 268 ], while one had continuous involvement of two patient partners [ 229 ]. Two used a “top and tail” approach [ 157 , 175 ], while one had engagement only during the final evidence synthesis stages [ 323 ]. None of these evidence syntheses reported the use of any formal research methods or techniques to engage people, with “meetings” being the most common form of engagement. Scoping reviews ( n  = 13): Eight of the 13 scoping reviews had engagement throughout the review [ 50 , 82 , 217 , 236 , 246 , 315 , 317 , 329 ]; three had engagement during the final stages only [ 71 , 302 , 320 ], one adopted a “top and tail” approach [ 240 ], and one engaged interest‐holders in rating potential evidence synthesis topics [ 167 ]. Five evidence syntheses used meetings or a mixture of meetings and electronic communication to engage interest‐holders [ 217 , 246 , 315 , 317 , 329 ]. Four evidence syntheses included interviews or focus groups [ 71 , 240 , 302 , 320 ] and two used electronic communication only [ 50 , 82 ]. Realist reviews ( n  = 14): All the included realist reviews reported engagement with topic experts and/or patient and public representatives to generate initial program theories [ 74 , 98 , 117 , 127 , 146 , 178 , 181 , 224 , 242 , 257 , 266 , 270 , 331 , 332 ]. The other main activity in which people were engaged was to refine and validate the final program theory. Engagement in this activity was reported in all the realist reviews. Interest‐holders were often asked to provide feedback on the preliminary analysis (or Context‐mechanism‐outcome configurations (CMOCs) in realist reviews) and to test or refine the program theories that were developed from the analysis. This was done primarily through interviews, workshops, and team meetings. Quantitative systematic reviews ( n  = 10): In four evidence syntheses, interest‐holders were involved in multiple activities throughout the review process [ 64 , 158 , 182 , 300 ]; in three, engagement occurred during the early stages [ 72 , 179 , 282 ]; and three adopted a “top and tail” approach [ 93 , 164 , 173 ]. Commonly, activities comprised meetings (both face‐to‐face and online) and email communication, but other activities included larger workshops, ranking of outcomes, and surveys. Qualitative and mixed method evidence syntheses ( n  = 8): Five of the eight evidence syntheses reported that they had continuous engagement throughout, with four of these also having other engagement activities for specific evidence synthesis tasks [ 136 , 153 , 222 , 268 ], while one had continuous involvement of two patient partners [ 229 ]. Two used a “top and tail” approach [ 157 , 175 ], while one had engagement only during the final evidence synthesis stages [ 323 ]. None of these evidence syntheses reported the use of any formal research methods or techniques to engage people, with “meetings” being the most common form of engagement. Scoping reviews ( n  = 13): Eight of the 13 scoping reviews had engagement throughout the review [ 50 , 82 , 217 , 236 , 246 , 315 , 317 , 329 ]; three had engagement during the final stages only [ 71 , 302 , 320 ], one adopted a “top and tail” approach [ 240 ], and one engaged interest‐holders in rating potential evidence synthesis topics [ 167 ]. Five evidence syntheses used meetings or a mixture of meetings and electronic communication to engage interest‐holders [ 217 , 246 , 315 , 317 , 329 ]. Four evidence syntheses included interviews or focus groups [ 71 , 240 , 302 , 320 ] and two used electronic communication only [ 50 , 82 ]. Realist reviews ( n  = 14): All the included realist reviews reported engagement with topic experts and/or patient and public representatives to generate initial program theories [ 74 , 98 , 117 , 127 , 146 , 178 , 181 , 224 , 242 , 257 , 266 , 270 , 331 , 332 ]. The other main activity in which people were engaged was to refine and validate the final program theory. Engagement in this activity was reported in all the realist reviews. Interest‐holders were often asked to provide feedback on the preliminary analysis (or Context‐mechanism‐outcome configurations (CMOCs) in realist reviews) and to test or refine the program theories that were developed from the analysis. This was done primarily through interviews, workshops, and team meetings. Across all types of evidence synthesis, most commonly, interest‐holders attended meetings at which no formal methods of engagement were used. Often, attendance at meetings was supplemented with electronic communication, such as emails. However, several “green” evidence syntheses described group discussion, sometimes referred to using the term “consultation.” Formal consensus activities, such as voting or ranking techniques, or the use of nominal group technique or Delphi surveys, were reported by some, while others reported holding larger “workshops,” “focus groups,” or individual interviews. It was most commonly considered that the interest‐holders had some influence over decisions and outcomes of each activity, though there were a small number of evidence syntheses in which it was judged that the interest‐holders had overall control over final decisions (possibly sharing equal power with the research team). With the exception of the engagement in activities relating to the generation, refinement, and validation of program theories within realist reviews, we were unable to identify any obvious patterns in relation to different forms of engagement, methods used, or level of control during different stages of an evidence synthesis. Across all types of evidence syntheses, it was common to engage interest‐holders in multiple activities throughout the review process, but there are also examples where interest‐holders are engaged for one activity only. Where interest‐holders contributed to one activity only, this is most likely to be an interpretation of review findings. The included “methods” papers (44/302, 14.6%) comprised an eclectic mix of publications, but have been broadly categorized into publications which: Describe a method or methods of engagement, or guidance relating to how to engage interest‐holders ( n  = 17) Focus on a specific methodological approach, tool, or framework ( n  = 9) Evaluate or reflect on engagement ( n  = 7) Describe methods to identify and/or prioritize new reviews ( n  = 7) Focus on review publication and dissemination ( n  = 2) Address other topics ( n  = 2) Describe a method or methods of engagement, or guidance relating to how to engage interest‐holders ( n  = 17) Focus on a specific methodological approach, tool, or framework ( n  = 9) Evaluate or reflect on engagement ( n  = 7) Describe methods to identify and/or prioritize new reviews ( n  = 7) Focus on review publication and dissemination ( n  = 2) Address other topics ( n  = 2) Details of these publications are provided in Table  13 . Summary of other publications providing relevant information relating to engagement in evidence syntheses (“Methods” papers; n  = 44).

Supplementary Material 1, Table of included studies ( n  = 302). Supplementary Material 2, How were interest‐holders engaged ( n =51). Supplementary Material 3, Comparison with previous scoping review results. PRISMA‐ScR‐Fillable‐Checklist.