Mixed Method Evaluation of the CEBHA+ Integrated Knowledge Translation Approach: A Protocol

Mixed Method Evaluation of the CEBHA+ Integrated Knowledge Translation Approach: A Protocol | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Study Protocol Mixed Method Evaluation of the CEBHA+ Integrated Knowledge Translation Approach: A Protocol Lisa Pfadenhauer, Tanja Grath, Peter Delobelle, Nasreen Jessani, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-117198/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 18 Jan, 2021 Read the published version in Health Research Policy and Systems → Version 1 posted 8 You are reading this latest preprint version Abstract Background The Collaboration for Evidence-based Healthcare and Public Health in Africa (CEBHA+) is a research consortium concerned with the prevention, diagnosis and treatment of non-communicable diseases. CEBHA+ seeks to engage policymakers and practitioners throughout the research process in order to build lasting relationships, enhance evidence uptake, and create long-term capacity among partner institutions in Ethiopia, Malawi, Rwanda, South Africa and Uganda in collaboration with two German universities. This integrated knowledge translation (IKT) approach includes the formal development, implementation and evaluation of country specific IKT strategies. Methods We have conceptualised the CEBHA+ IKT approach as a complex intervention in a complex system. We will employ a comparative case study (CCS) design and mixed methods to facilitate an in-depth evaluation. We will use quantitative surveys, qualitative interviews, quarterly updates, and a policy document analysis to capture the process and outcomes of IKT across the African CEBHA+ partner sites. We will conduct an early stage (early 2020) and a late-stage evaluation (early 2022), triangulate the data collected with various methods at each site and subsequently compare our findings across the five sites. Discussion Evaluating a complex intervention such as the CEBHA+ IKT approach is complicated, even more so when undertaken across five diverse countries. Despite conceptual, methodological and practical challenges, our comparative case study addresses important evidence gaps: While involving decision-makers in the research process is gaining traction worldwide, we still know very little regarding (i) whether this approach really makes a difference to evidence uptake, (ii) the mechanisms that make IKT successful, and (iii) relevant differences across socio-cultural contexts. The evaluation described here is intended to provide relevant insights on all of these aspects, notably in countries in Sub-Saharan Africa, and is expected to contribute to the science of IKT overall. Health Economics & Outcomes Research Health Policy Integrated Knowledge Translation co-production Sub-Saharan Africa non-communicable diseases evaluation mixed methods comparative case study complexity Figures Figure 1 Figure 2 Figure 3 Background Integrated Knowledge Translation (IKT) in public health and healthcare is defined as engagement of knowledge users (e.g., decision-makers) as active participants in the research process ( 1 ). Its purpose is to engage in an interactive, collaborative process with the overarching goal being the co-production of knowledge that is relevant to policy and practice ( 2 ). By integrating knowledge users in research, knowledge translation (KT) and research can be considered as two interwoven processes. The involved decision-makers minimise or address potential barriers that may occur when attempting to act upon results. IKT is a steadily evolving process, embedded in a complex web of contextual factors ( 3 ). IKT thus constitutes one of a series of related concepts in the field of research co-production, namely participatory research, research collaboration, public/patient involvement and engaged scholarship, with aims ranging from the production of more applicable and useful research to the democratisation of science ( 4 ). There are several presumed benefits linked to research that embraces IKT, the most important being that it generates knowledge that is more relevant to policy and practice and more likely to be used by decision-makers ( 5 ). An IKT approach integrates local knowledge and helps to translate research findings into policy and practice by addressing identified knowledge-practice gaps and providing easier-to-adopt research evidence ( 6 – 8 ). IKT could therefore be considered one step towards producing the right answers to the right questions ( 9 ). The involvement of various stakeholders in health care and public health research is gaining traction ( 10 ) with increasing funder interest, predominantly in high-income countries but also in low- and middle-income countries (LMICs) ( 4 ). While IKT shows promise in bridging the gap between research and practice, there may also be challenges encountered along the way ( 5 ). From the perspective of decision-makers, these include presumptions about academic knowledge, skills and capabilities ( 5 ). Functional collaborations require time to be established and effort to be maintained, resources that are often not allocated by funders ( 11 , 12 ). Stakeholders may have competing demands and agendas ( 11 ) and may consider a range of aspects before engaging in IKT ( 2 ). Also, networks relying on personal relationships might be affected by personnel changes ( 11 , 13 – 15 ) as well as the “maturity” of the relationships ( 8 , 16 ). Last but not least, it should be acknowledged that IKT is not a panacea and is unlikely to be a suitable approach for all research endeavours ( 11 ). Thus it is important to gauge where and when an IKT approach adds value to research. Considering that IKT has been assumed to have an influential role in evidence-informed policy-making, it is surprising that high-quality evaluations of its processes and outcomes are still scarce ( 17 ). Most evaluation research has been conducted in the form of case studies, using observational study designs ( 18 ). A scoping review concluded that IKT initiatives that were evaluated achieved one or more positive outcomes, however, due to the often poor quality and reporting of these evaluations, no recommendation with regards to what makes IKT strategies effective or not could be made ( 19 ). The IKT field should therefore emphasise a thorough description of IKT efforts and their implementation, as well as well-conducted evaluations ( 20 ). Also, formative research to develop and test outcome measures such as metrics of research use or research impact on policy making would help to advance the knowledge translation field ( 20 ). The field needs empirical and comparative research that tests the benefits that are attributed to IKT, such as the generation of social capital, new relations and cooperative behaviour, greater quality of health services and reduced costs and greater health benefits for those involved ( 21 , 22 ), including an increased consideration of contextual factors ( 23 ). Also, LMICs are seldom represented in IKT evaluations, with evidence to date derived from a small number of primarily Anglophone countries, notably the United States, United Kingdom, Canada, Australia and New Zealand ( 21 , 23 , 24 ). IKT represents an integral part of the Collaboration for Evidence-Based Healthcare and Public Health in Africa (CEBHA+), an African-German research consortium with a focus on non-communicable diseases (research tasks 1–3: prevention, screening and treatment of cardiovascular disease, diabetes, and hypertension; research task 4: road traffic injuries). CEBHA + is a five-year project funded by the German Federal Ministry of Education and Research (BMBF) as part of the Research Networks for Health Innovation in Sub-Saharan Africa funding initiative. It was set up in 2017 and aims to build long-term capacity and infrastructure for evidence-based health care and public health in Africa. The consortium consists of seven partner institutions in five Sub-Saharan African countries (Ethiopia, Malawi, Rwanda, Uganda, and South Africa; subsequently referred to as five “CEBHA + sites”) and two partner institutions in Germany. As one of its core goals, CEBHA + seeks to promote the use of contextualised research evidence in decision-making by applying a coordinated IKT approach across all its research activities. To this end, researchers at the five African CEBHA + sites are pursuing focused, long-term engagement with members of the policy-and-practice community (subsequently referred to as “stakeholders”). Objective This paper describes the evaluation procedures to assess the process and outcomes of the CEBHA + IKT approach implemented across five African CEBHA + sites. The evaluation is based on a programme theory and will be conducted semi-externally, i.e. one of the two German partner institutions, the LMU Munich (“the evaluation team”), will lead the evaluation while not being directly involved in the implementation of the IKT strategies at each site. The proposed outcome evaluation aims to assess whether an IKT approach ultimately contributes to increased uptake of contextualised research in policy and practice, while also paying attention to intermediate outcomes such as increased relevance or applicability of research in the respective context. The proposed process evaluation, conducted in parallel, aims to shed light on the dose, fidelity and quality of the IKT strategies implemented at each site ( 25 ). Continuous monitoring, conducted by each implementing African partner site, is envisioned to refine the IKT strategies as they are being implemented. Having conceptualised the CEBHA + IKT approach as a complex intervention in a complex system, our evaluation is expected to support an in-depth understanding of what works, for whom, and in which circumstances. We aim to enhance our understanding of IKT within the CEBHA + consortium and beyond; to enhance the IKT approach over the course of the project; and, to contribute to the IKT research field as a whole. Methods Intervention: Overarching CEBHA+ IKT approach and site-specific IKT strategies The IKT approach in CEBHA+ can be conceptualized as an intervention within a complex social system. By implementing a systematic approach to engaging with stakeholders, building new or strengthening existing relationships, we aim to disrupt traditional knowledge translation approaches (26). We do this by redistributing and transforming resources, for example by having dedicated personnel for IKT ("IKT focal points”, one or two staff per site) that coordinate and implement local IKT efforts. The IKT focal points and the IKT evaluation team at LMU Munich make up the CEBHA+ IKT team. At each CEBHA+ site, researchers engage with stakeholders. In the context of IKT in healthcare and public health, a concerned stakeholder can be defined as an entity responsible, involved or affected by health-related decisions that can be informed by research evidence (27, 28). Entities can be individuals, organisations, groups or networks operating at a local, regional, national or supra-national level. They can comprise the public, civil society organizations, the media, public health practitioners and service providers as well as public health policy-makers and industry actors across multiple sectors (29). These entities can be directly or indirectly affected – in positive or negative ways – by an effort or the actions of an entity (30). We target decision-makers, i.e. policy and practice stakeholders holding a position that allows them to make system-level, organizational or technical decisions that affect the general health of communities or populations. These decisions can be made on a more political or a more technical (i.e. programmatic) level within the health sector as well as in other sectors. Our primarily targeted stakeholder group might be extended as we proceed with the CEBHA+ IKT approach. Initially, the IKT evaluation team at LMU Munich conducted a literature review on IKT, its mechanisms and effectiveness. We conducted forward citation tracking from a recent scoping review (19). This search (Feb 2018) did not yield updates of the review or more recently published evaluations of IKT approaches. Informed by the literature review and discussions with the IKT focal points, we developed a programme theory of the overarching CEBHA+ IKT approach. This programme theory was operationalized into a menu of options of IKT activities that researchers and stakeholders could adapt to their respective research tasks and contexts. The overall CEBHA+ IKT approach comprises six steps, of which four are site-specific and include development of IKT strategies that are tailored to each CEBHA+ site. These site-specific IKT strategies recognise existing relationships at the different institutions, the local context, and the varying needs of different research tasks within CEBHA+. A coordinated IKT approach ensures that site-specific IKT strategies are harmonised to the extent reasonable. In terms of activities, the six steps towards a coordinated IKT approach in CEBHA+ are described in Table 1 and visualized in Figure 1. Table 1 Overview of Development, Implementation and Evaluation of the Integrated Knowledge Translation Approach in CEBHA+ Step Date Description Step 1 Nov 2018 In a foundational workshop developed and implemented by the Centre for Evidence Based Health Care, Stellenbosch University (31), one of the South African CEBHA+ partners, IKT focal points from each CEBHA+ site were introduced to the concepts of evidence-informed decision-making, IKT, and the overarching CEBHA+ IKT approach. Participants were introduced to the programme theory and received practical training on how to develop an IKT strategy (stakeholder mapping, stakeholder analysis, stakeholder engagement strategies) and draft a site-specific IKT strategy. Step 2 Nov 2018 Workshop participants acted as multipliers within their country teams and shared their knowledge and skills gained during the workshop with the rest of their team members. By using the tools introduced during the workshop, country teams were requested to jointly agree on priority stakeholders with whom to engage over the course of the CEBHA+ project and to refine the site-specific IKT strategy drafted during the foundational workshop. Step 3 Nov 2018 – Jan 2019 Each country team consulted priority stakeholders to introduce CEBHA+ and IKT and to gauge their interest in the research topic as well as their preferences for engaging with the CEBHA+ research team. Step 4 Feb 2019 Country teams met to review and synthesise the information obtained in previous steps and to finalise the site-specific stakeholder analysis and IKT strategy. Step 5 Feb 2019 – Dec 2022 Country teams are implementing and refining their site-specific IKT strategies and monitoring the indicators they defined as part of their site-specific IKT strategy. Step 6 Feb 2020 – Jul 2022 A two-step evaluation, by LMU Munich, of the coordinated IKT approach and the site-specific IKT strategies will assess the added value of the IKT approach in CEBHA+ (stage one ongoing, stage two scheduled for Mid-2022) Captions to Figure 1: This figure visualizes the process of developing, implementing and evaluating the Integrated Knowledge Translation Approach in CEBHA+. The dark shaded boxes describe steps which were pursued at the level of the research consortium, while the light shaded boxes describe steps pursued at the respective sites. Step 6 (evaluation) is pursued at both the consortium and individual site level. These steps are not necessarily pursued in a linear manner across all sites but require a certain degree of iteration, i.e. steps 2 to 5 may be repeated as the stakeholder landscape changes. Ideally, stakeholders are engaged in the research process from an early stage (11). This engagement can comprise shaping the research questions, deciding on the methodology, involvement in data collection and development of tools, interpreting the findings and supporting dissemination of research results at the research task level (2). At the level of the CEBHA+ consortium, this “IKT thinking” has been embraced since the initial development of the project proposal, where relevant stakeholders from the respective partner countries were included in the process of setting research priorities. Over the course of the project, practical experiences with the development and implementation of site-specific IKT strategies, insights gained from monitoring and evaluation, or new scientific evidence may lead to adaptations in the overall IKT approach and/or the site-specific IKT strategies. Programme theory In order to guide the development and evaluation of the CEBHA+ IKT approach, we developed a comprehensive programme theory (see Figure 2). This builds upon two main frameworks: the Context and Implementation of Complex Interventions (CICI) framework (32) and a visual representation of IKT approaches, influencing factors, and outcomes as described in a scoping review by Gagliardi and colleagues (19). The CICI framework was used to structure the context and implementation component of the evaluation by providing a priori categories which informed the development of both the qualitative interview guide and the survey. The framework by Gagliardi provides an overview of empirical evaluations of IKT approaches, relevant enablers and barriers, preconditions, and outcomes. Moreover, it flags relevant and partly validated evaluation tools which informed the development of our survey. The contributions of these two frameworks were integrated to show how the CEBHA+ IKT approach is assumed to increase the use of contextualized research evidence in policy and practice decision-making. As shown in Figure 2, the programme theory describes the coordinated IKT approach as an intervention implemented across the CEBHA+ consortium. In CEBHA+, a coordinated IKT approach is understood as an iterative process in which 1) relationship building and strengthening, 2) capacity building, and 3) collaborative research between CEBHA+ researchers and their priority stakeholders represent integral components of the intervention. Capacity building, relationship building, and their maintenance and collaborative research are expected to take place in an interactive manner and repeatedly over time, thereby reinforcing each other (3). The implementation of these intervention components is expected to result in several intervention outcomes. The literature indicates that strong relationships increase mutual understanding (e.g. of language, work style, needs, constraints) and trust among researchers and decision-makers engaging in a partnership and that these influence attitudes towards the partners’ professional environment (i.e. attitudes towards research among stakeholders versus attitudes towards the policy-and-practice field among researchers) (19, 33-36). With this in mind, CEBHA+ researchers seek to build and enhance relationships with priority stakeholders. The experience of conducting collaborative research would ideally lead to appreciation for the collaborative process (33), promote a more diverse range of partners to engage in, and encourage involvement throughout the research process. Mutual capacity-building can occur in the form of improved access to information and/or contacts due to the research partnership, broadened perspectives and skills (e.g. research skills), and improved capacity for collaboration (33-35). Moving on to intermediate outcomes , successful research partnerships between researchers and stakeholders is assumed to result in an increased perceived value of research evidence in terms of relevance, better applicability to the stakeholders’ field of activity as well as credibility. While both actual and perceived value are potentially affected by IKT efforts, studies show the individual perception of value has greater behavioural implications than the actual value of research evidence (37). We therefore focus on the perceived value of CEBHA+ research evidence with respect to its importance to the actual consideration and utilization of research evidence in decision-making. In turn, research evidence that is perceived as credible, relevant, and usable is likely to result in an increased intention to use that evidence in policy and practice decision-making. Subsequently, the intention to use research evidence is hypothesised to lead to the actual consideration of findings in a specific decision in another step towards the final outcome: the use of contextualized research in policy and practice decision-making. Evaluation of the CEBHA+ IKT approach Overview of evaluation approach This evaluation is theory-driven. The programme theory provides the main structure for the evaluation and informs our choice of methods for data collection as well as data analysis. A comparative case study (CCS) design (38) will be used, including mixed methods to facilitate in-depth examination of IKT evaluation in each site. Each site (Ethiopia, Malawi, Rwanda, South Africa, Uganda) is considered a case. A CCS is undertaken over time and emphasizes comparison within and across contexts and thus allows to explore, understand, and explain how context influences the success of an intervention (39). Studying each case with its distinct IKT strategy at two time points will provide an opportunity to compare and contrast findings within and across sites. As part of the process evaluation, we will look at how and why IKT works in the respective contexts. In line with the CICI framework, we will look at the different stages in the implementation process, the diverse implementation strategies employed in different sites and the respective implementation agents (32). We will moreover utilize our monitoring to deepen our understanding of the implementation process. This may be complemented by including quantitative implementation outcomes, such as acceptability and feasibility in the late-stage evaluation. As part of the outcome evaluation, we will investigate the intervention, intermediate, and final outcomes, as defined in the programme theory. As shown in Figure 3, the evaluation process will comprise two phases, i.e. an early-stage assessment (early 2020) and a late-stage assessment (early 2022). In addition to the data analyses performed at the end of each phase, we will conduct an integrated analysis after the completion of phases 1 and 2 (until mid-2022). The evaluation process will comprise five different procedures: (a) quantitative surveys with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (b) qualitative interviews with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (c) continuous monitoring to capture ongoing IKT activities (mainly covered by the documentation of interactions between CEBHA+ researchers and policy-and-practice stakeholders reported in quarterly updates); (d) policy document analysis; and (e) integrated and cross-case analysis at the end of the evaluation. Each of the procedures will be conducted in all five CEBHA+ sites. Data collection Given the complexity of the initial programme theory and the multitude of outcomes along the causal path from the intervention to the intended outcome, we will use a combination of qualitative (semi-structured interviews, qualitative analysis of policy documents, IKT updates) and quantitative data collection approaches (survey). The survey is intended to touch upon all dimensions covered by the programme theory. The qualitative interviews, however, will allow us to explore how IKT exerts an influence in more depth. Also, we will focus on external context factors (macro-context) that impact on IKT, the process of IKT, as well as intermediate and final outcome(s). Two distinct surveys and two interview guides target researchers and stakeholders, respectively. This is necessary to reflect the different perspectives of individuals engaged in a partnership. Development of data collection instruments The underlying programme theory was used to develop survey and interview instruments. We initially constructed the survey based on suitable pre-existing instruments included in the review by Gagliardi et al. (2015). Literature searches were then conducted and instruments identified to inform the development of additional survey items (e.g. SAGE (40), PreVAiL (41), SEER (42)). Where items were non-existent, we constructed them de novo. The survey comprises both multiple choice and Likert scale questions. We provide an overview of included constructs and sub-constructs as well as their source in the additional files (Additional file 1). The interview guides and surveys were pilot tested among two researchers (Rwanda and South Africa, one male, one female) who are not involved in the IKT implementation process. We assessed the tools for errors, comprehensibility, acceptability, and understandability. We were not able to pilot-test our tools with decision-makers from policy-and-practice due to their limited time availability and the high opportunity cost. Both the interview guide and survey were slightly adapted to account for the changed project context due to the COVID-19 pandemic and are available as additional files (Additional files 2 - 5). Insights gained during the early-stage evaluation will inform the interview guides and surveys used in the late-stage evaluation. In addition to the topics covered in the early-stage assessment, participants will also be asked to describe how the research partnership has changed over time and to describe, where applicable, examples of successful translation of research evidence into policy or practice because of the research partnership. Sampling Selection of study participants for the surveys and interviews will reflect the following criteria: - A CEBHA+ researcher or a priority stakeholder as identified by the respective CEBHA+ country team; AND - actively involved in a research-stakeholder partnership in CEBHA+; AND - 18 years or above; AND - lives and works in the respective CEBHA+ country; AND - is able to understand and articulate him/herself in English With respect to sample size, we aim to interview five to ten participants per site, ideally both researchers and key stakeholders. Recruitment The evaluators will invite all eligible CEBHA+ researchers to participate in the study via email and will introduce the study. Eligible stakeholders will be invited to participate by their respective research counterparts and will be provided with the same information. Participants willing to be enrolled will be contacted by the evaluators via email to arrange a date for the qualitative interview. A written informed consent form will be provided, detailing the study’s goals, estimated duration, scope, its voluntary nature, the pseudonymity and confidentiality of responses, and whom to contact regarding questions. Participants will be asked to return the digitally signed informed consent form or a photo/scan of the manually signed form. Participants will also receive an invitation link to access the online survey. Survey procedures The survey will be administered online using the LimeSurvey tool ( https://www.limesurvey.org/en ) and participants requested to complete the survey prior to the qualitative interviews. At the beginning of the questionnaire, participants will receive information regarding the study. Key information regarding the informed consent form will be reiterated. Participants who are not able to complete the survey before the qualitative interviews will receive reminders via email and orally during the interview. Interview procedures The interviews will be conducted by a member of the CEBHA+ evaluation team or trained research staff at the CEBHA+ site using an interview guide. Where feasible, notes will be taken by a second member of the CEBHA+ evaluation team or by trained research staff at the CEBHA+ site, who are not directly involved with the site-specific IKT strategy. We consider it important that the interviewer is not involved in the practical implementation of IKT at the respective CEBHA+ site and thus unknown to the interviewed stakeholder to reduce interviewer bias. All interviews will be conducted in English. In light of the COVID-19 pandemic interviews will be conducted virtually (e.g. via skype, zoom or other) unless the interviewee insists on a face-to-face interview, the pandemic situation permits, and precautionary measures are in place. Where conducted in person, interviews will be carried out in a place of the interviewee’s choice, e.g. the office of the stakeholder or researcher. As a minimum requirement, the chosen place should be a separate room or another quiet place to avoid disturbances and outside observers. The estimated duration of a single interview is 30 to 60 minutes. The interviewees will be asked to describe their perspective on the research partnership, contextual factors that hinder or facilitate the partnership or its outcomes, the expected benefits and ways to improve the partnership and/or outcomes. Interviews will be audio-recorded and transcribed verbatim. Interviewees will be given the opportunity to review the transcript, ask for modifications and give their approval for further use of the data. Further data sources Quarterly IKT updates serve as a means of continuous exchange among the IKT teams. The updates take place during a virtual call and are facilitated by a structured reporting tool. This tool captures noteworthy interactions between researchers and stakeholders and their outputs. It also contains questions on the learnings and challenges encountered at each site. All updates are submitted to the evaluation team before the calls and will be analysed in Phase 1 and 2. Policy documents will be identified and analysed for the late-stage assessment. Relevant policy documents are those produced between 2017 and 2022 (i.e. the course of the CEBHA+ project) and on topics related to CEBHA+ research activities at the five research sites. Documents concerned with laws, regulations, plans and major programmes from different tiers of government will be selected. Other documents considered will include those issued by multi-sectoral working groups or semi-independent bodies with a mandate to devise strategies or plans. These policy documents will be identified through the quarterly IKT updates, interviews and stakeholder websites (incl. meeting agendas, minutes, protocols, or reports). Data analysis Descriptive analysis of survey data All quantitative data will be analysed descriptively using IBM SPSS Statistics 27.0. Survey data will be analysed in conjunction with qualitative data for triangulation at the end of phase 1 and phase 2 (Figure 3). Qualitative analysis of interviews Interview transcripts will be analysed in MaxQDA using content analysis as described by Schreier (2012) (43). Codes will be developed both deductively (informed by the programme theory and resulting interview guide) and inductively (data driven). For each of the sites, the interviews will be coded independently by two coders, with one of them being from the same site with a comparable cultural background as the interviewee in order to include both an insider as well as outsider perspective (44). Codes will be discussed among the two coders and adapted until agreement is reached. For each site, there will be two sets of category systems, one for the researcher and one for the stakeholder dataset. While we expect overlaps, there are likely to be distinctions. This will ultimately lead to a code book that will be applied to all interviews, while allowing for inductively developed, site-specific codes. Document analysis of policy documents The policy documents identified in the late-stage evaluation phase will be analysed through document analysis, a systematic procedure for reviewing or evaluating printed or electronic documents, which entails finding, selecting, appraising and synthesising data contained in the retrieved documents (45). We will look for specific indicators of CEBHA+-generated research evidence uptake into practice or policymaking. Qualitative content analysis will be applied to relevant sections of the document (45), by using an a priori developed list of categories which corresponds to the outcome indicators in our programme theory. Text passages that discuss any of these a priori categories will be coded according to this list, while others will be coded under new categories. The resulting category systems will be shared and discussed among the CEBHA+ IKT team members undertaking the document analyses. Triangulation of data at each site At the end of phases 1 and 2, qualitative and quantitative data will be integrated for each case using the triangulation method as described by Moran-Ellis and colleagues (2006), “following a thread” (46). Triangulation is a general approach whereby the convergence, complementarity, and dissonance of results on related research questions, obtained through different methodological approaches, sources, theoretical perspective, or researchers are explored. With this method, an initial analysis of multiple datasets (survey, interviews, quarterly IKT updates, policy document analysis) will be undertaken separately for each case (i.e. each site). At this stage, themes and analytical questions that require further investigation will be identified. These themes or questions – the thread – are followed throughout the datasets to capture corresponding, contradicting as well as lacking findings within cases. The final outcome of this step is to create concise case descriptions for each of the five individual CEBHA+ sites. Cross-case analysis across sites After data collection and triangulation (2022), we will conduct a cross-case analysis across findings from the five CEBHA+ sites, the two points of data collection and various methods of data collection. Cross-case analysis is a research method that facilitates the comparison of commonalities and differences in the events, activities, and processes that are the units of analyses in case studies. The aim is to compare key findings across cases (i.e. across sites). Drawing upon the programme theory, we will contrast what we found at each site with regards to the components of the programme theory. This cross-case analysis will also serve as basis for reflections on the community of IKT practice across sites within the CEBHA+ consortium. Insights from the cross-case analysis are likely to lead to refinements of the initial programme theory, leading to a middle-range theory. Data Management Data collection, retention and analysis will be conducted adhering to EU data protection regulations. All data (digital records, transcripts, interview notes, survey data) will be kept confidentially on a secured institutional IT infrastructure provided by the Institute for Medical Information Processing, Biometry, and Epidemiology (IBE) at the LMU Munich, which meets the requirements of an institute for research on highly sensitive patient and study data. For analysis, researchers from the respective CEBHA+ sites will have access to the pseudonymized interview data after transcription. All analyses will be undertaken in the cloud and no data will be stored locally at any of the sites. Discussion To our knowledge, this protocol is one of the first attempts to evaluate an IKT approach across multiple sites and countries using a longitudinal design. It suggests an overall methodology and detailed methods which we deem appropriate to capture the complexities and context dependencies of the different IKT strategies employed across sites and highlight the important characteristics of individuals involved in delivering IKT. The opportunity that a thorough development, implementation and evaluation of IKT is possible within the context of CEBHA + can be attributed to the open funding call issued by the German Federal Ministry of Education and Research ( Bundesministerium für Bildung und Forschung , BMBF) in 2013. Whilst the funder’s interest in IKT as a means of enhancing research impact is highly commendable and in line with several other international funding agencies requiring that knowledge translation be integrated into proposals ( 12 ), a critical meta-discussion is needed on how IKT interventions can be aligned with the realities of international research collaborations and the contexts and capacities of the different partner institutions. We have encountered and will continue to encounter conceptual, methodological and practical challenges in the proposed evaluation. Conceptually, we are seeking to understand the context and individual dependency of IKT which will inevitably lead to heterogeneity. We aim to capture these heterogeneities by employing multiple methods; we endeavour to paint a holistic picture by integrating the perspectives of researchers and stakeholders in the evaluation. Methodologically, we have struggled to identify appropriate and valid ways of monitoring highly heterogeneous IKT strategies. This will be particularly important when determining the dose-response-relationship between the effort invested in stakeholder engagement and related outcomes. We are optimistic that the triangulation of different sources of data will provide valuable insights. We have conceptualized this evaluation as semi-external, with the evaluation team not being involved in the implementation of local IKT strategies. While this more distant perspective on the realities of IKT on the ground facilitates a largely independent evaluation, the interpretation of evaluation findings will need to be complemented by local researchers’ contextual understanding. This will be facilitated by integrating one researcher per site with the analysis and interpretation of the generated data. This researcher will not have participated in the evaluation. Practically, CEBHA + activities just like those of most other health research institutions or networks have been severely disrupted by the COVID-19 pandemic. Both on the side of the researchers and the stakeholders, resources initially intended for CEBHA + activities were focused towards responding to emerging needs due to the pandemic. Additionally, we had to defer from conducting this evaluation face-to-face and resort to virtual interviews. These developments led to delays and reiterations in our procedures. On the other hand, the COVID-19 pandemic has most clearly demonstrated the international need for research evidence in practice and policy decision making and made a very strong case for IKT. While we cannot anticipate the developments over the next year(s), we believe that the IKT experiences in CEBHA + thus far do contribute to the international research on IKT, especially given the intensive collaboration of researchers and policymakers during the pandemic. Conclusion While involving decision-makers in the research process is gaining traction worldwide, there is still very little knowledge regarding (i) whether a systematic approach makes a difference to evidence uptake, (ii) the mechanisms that make IKT successful, and (iii) relevant differences across socio-cultural contexts. The CEBHA + IKT approach represents a carefully developed intervention designed to increase the use of contextualized research in policy and practice decision-making. As an approach that is implemented in a coordinated manner across five African countries, it represents a rare opportunity for a thorough evaluation and learning from differences across multiple sites. The evaluation described here is intended to provide relevant insights on all the above aspects, notably in Sub-Saharan Africa, and is expected to contribute to the science of IKT overall. List Of Abbreviations IKT/KT Integrated Knowledge Translation/Knowledge translation CEBHA+ Collaboration for Evidence-Based Healthcare and Public Health in Africa CCS Comparative Case Study LMIC Low- and Middle-Income Country Declarations Ethics approval and consent to participate We applied and were granted ethics approval from the following institutions: - University of Munich (LMU), Germany (19-633) - Research and Ethics Committee, School of Public Health, College of Health Sciences, Makerere University, Uganda (IRB00011353; study protocol number 469) - Armauer Hansen Research Institute (P0/31/20) - College of Medicine Research Ethics Committee, University of Malawi (P.11/19/2850) - Rwanda National Ethics Committee, Ministry of Health, Kigali, Rwanda (074/RNEC/2020) - Human Research Ethics Committee, Faculty of Health Sciences, University of Cape Town (026/2020) Consent for publication Not applicable Availability of data and materials Not applicable Competing interests The authors declare that they have no competing interests. All authors are part of the CEBHA+ consortium. Funding This work was supported by the German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung (BMBF)) as part of the Research Networks for Health Innovation in Sub-Saharan Africa funding initiative. The funder had no role in design of the study and in writing the manuscript. Authors' contributions All authors were actively involved in designing the CEBHA+ IKT approach, the evaluation tools, and developing this protocol by providing critical feedback and engaging in numerous discussions. All authors except for LMP, TG, JJM, KS, IT and ER are actively implementing the IKT strategies at the respective sites. LMP, TG, KS and ER drafted the manuscript. All authors critically revised the draft and read and approved the final manuscript. Acknowledgements We would like to thank Jacob Burns for valuable input on the development of the survey. We would like to thank Dr. Irakoze Magnifique from University of Rwanda and Lynn Hendricks from Stellenbosch University for testing and providing valuable feedback on our data collection tools. We would moreover acknowledge all IKT focal points and all CEBHA+ researchers who are actively implementing and monitoring the IKT strategies at each site. References Lawrence LM, Bishop A, Curran J. Integrated Knowledge Translation with Public Health Policy Makers: A Scoping Review. Healthc Policy. 2019;14(3):55-77. Kothari A, Wathen CN. A critical second look at integrated knowledge translation. Health Policy. 2013;109(2):187-91. Kreindler SA. Advancing the evaluation of integrated knowledge translation. Health research policy and systems. 2018;16(1):104-. Graham ID, McCutcheon C, Kothari A. Exploring the frontiers of research co-production: the Integrated Knowledge Translation Research Network concept papers. Health Research Policy and Systems. 2019;17(1):88. Campbell H, Vanderhoven D. Knowledge that matters: realising the potential of co-production. . Manchester; 2016. Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: what we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105-6. Orr K, Bennett M. Public Administration Scholarship and the Politics of Coproducing Academic–Practitioner Research. Public Administration Review. 2012;72(4):487-95. Kothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface: managing policymaker-researcher collaboration. Knowledge Management Research & Practice. 2011;9(3):203-14. Van de Ven AH, Johnson PE. Knowledge for theory and practice. Academy of management review. 2006;31(4):802-21. Rehfuess EA, Durao S, Kyamanywa P, Meerpohl JJ, Young T, Rohwer A, et al. An approach for setting evidence-based and stakeholder-informed research priorities in low- and middle-income countries. Bull World Health Organ. 2016;94(4):297-305. Rycroft-Malone J, Burton CR, Bucknall T, Graham ID, Hutchinson AM, Stacey D. Collaboration and Co-Production of Knowledge in Healthcare: Opportunities and Challenges. Int J Health Policy Manag. 2016;5(4):221-3. McLean RKD, Graham ID, Bosompra K, Choudhry Y, Coen SE, MacLeod M, et al. Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science. 2012;7(1):57. Shearer JC, Abelson J, Kouyaté B, Lavis JN, Walt G. Why do policies change? Institutions, interests, ideas and networks in three cases of policy reform. Health Policy and Planning. 2016;31(9):1200-11. Jessani NS, Boulay MG, Bennett SC. Do academic knowledge brokers exist? Using social network analysis to explore academic research-to-policy networks from six schools of public health in Kenya. Health Policy and Planning. 2015;31(5):600-11. Jessani NS, Babcock C, Siddiqi S, Davey-Rothwell M, Ho S, Holtgrave DR. Relationships between public health faculty and decision makers at four governmental levels: a social network analysis. Evidence & Policy: A Journal of Research, Debate and Practice. 2018;14(3):499-522. Jessani NS, Valmeekanathan A, Babcock C, Ling B, Davey-Rothwell MA, Holtgrave DR. Exploring the evolution of engagement between academic public health researchers and decision-makers: from initiation to dissolution. Health Research Policy and Systems. 2020;18(1):15. Durose C, Needham C, Mangan C, Rees J. Generating ‘good enough’ evidence for coproduction. Evidence and Policy. 2015. Durose C, Mangan C, Needham C, Rees J, editors. Evaluating co-production: pragmatic approaches to building the evidence base Political Studies Association Conference; 2014; Manchester. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38. Gollust SE, Seymour JW, Pany MJ, Goss A, Meisel ZF, Grande D. Mutual Distrust: Perspectives From Researchers and Policy Makers on the Research to Policy Gap in 2013 and Recommendations for the Future. Inquiry. 2017;54:46958017705465. Cairney P, Oliver K. Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Res Policy Syst. 2017;15(1):35. Cottrell E, Whitlock E, Kato E, Uhl S, Belinson S, Chang C, et al. AHRQ Methods for Effective Health Care. Defining the Benefits of Stakeholder Engagement in Systematic Reviews. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014. Edwards A, Zweigenthal V, Olivier J. Evidence map of knowledge translation strategies, outcomes, facilitators and barriers in African health systems. Health Research Policy and Systems. 2019;17(1):16. Tricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open. 2016;6(12):e013929-e. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134-47. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. American journal of community psychology. 2009;43(3-4):267-76. Lemke AA, Harris-Wai JN. Stakeholder engagement in policy development: challenges and opportunities for human genomics. Genetics in medicine : official journal of the American College of Medical Genetics. 2015;17(12):949-57. Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. Journal of General Internal Medicine. 2014;29(12):1692-701. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, Knowledge Transfer Study G. How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers? Milbank Quarterly. 2003;81(2):221-48. Community Tool Box. Section 8. Identifying and Analyzing Stakeholders and Their Interests 2017 [Available from: https://ctb.ku.edu/en/table-of-contents/participation/encouraging-involvement/identify-stakeholders/main . Jessani NS, Hendricks L, Nicol L, Young T. University Curricula in Evidence-Informed Decision Making and Knowledge Translation: Integrating Best Practice, Innovation, and Experience for Effective Teaching and Learning. Frontiers in Public Health. 2019;7(313). Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implementation Science. 2017;12(1):21. Soper B, Yaqub O, Hinrichs S, Marjanovich S, Drabble S, Hanney S, et al. CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation. Journal of Health Services Research & Policy. 2013;18(3_suppl):53-64. Van Olphen J, Ottoson J, Green L, Barlow J, Koblick K, Hiatt R. Evaluation of a partnership approach to translating research on breast cancer and the environment. Prog Community Health Partnersh. 2009;3(3):199. Bowen S, Martens P. Demystifying knowledge translation: learning from the community. J Health Serv Res Policy. 2005;10(4):203-11. Eriksson CC-G, Fredriksson I, Fröding K, Geidne S, Pettersson C. Academic practice–policy partnerships for health promotion research: Experiences from three research programs. Scandinavian Journal of Public Health. 2014;42(15_suppl):88-95. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. Campbell S. Comparative case study. Encyclopedia of case study research. 2010;1:174-6. Bartlett L, Vavrus F. Comparative Case Studies: An Innovative Approach. Nordic Journal of Comparative and International Education (NJCIE). 2017;1(1). Makkar SR, Brennan S, Turner T, Williamson A, Redman S, Green S. The development of SAGE: A tool to evaluate how policymakers’ engage with and use research in health policymaking. Research Evaluation. 2016;25(3):315-28. Kothari A, Sibbald SL, Wathen CN. Evaluation of partnerships in a transnational family violence prevention network using an integrated knowledge translation and exchange model: a mixed methods study. Health Res Policy Syst. 2014;12:25. Brennan SE, McKenzie JE, Turner T, Redman S, Makkar S, Williamson A, et al. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research. Health research policy and systems. 2017;15(1):1-. Schreier M. Qualitative Content Analysis in Practice: SAGE Publications; 2012. The SAGE Encyclopedia of Qualitative Research Methods. 2008. Bowen GA. Document analysis as a qualitative research method. Qualitative research journal. 2009;9(2):27-40. Moran-Ellis J, Alexander VD, Cronin A, Dickinson M, Fielding J, Sleney J, et al. Triangulation and integration: processes, claims and implications. Qualitative Research. 2006;6(1):45-59. Supplementary Files AdditionalFile1.docx Evaluation Domains, Constructs and Data Sources AdditionalFile2SurveyResearcher.pdf CEBHA+ Evaluation Survey for Researchers AdditionalFile3SurveyStakeholder.pdf CEBHA+ Evaluation Survey for Stakeholders AdditionalFile4InterviewguideresearcherversionV7.docx CEBHA+ Evaluation Interview Guide for Researchers AdditionalFile5InterviewguidestakeholderversionV9.docx CEBHA+ Evaluation Interview Guide for Stakeholders Cite Share Download PDF Status: Published Journal Publication published 18 Jan, 2021 Read the published version in Health Research Policy and Systems → Version 1 posted Editorial decision: Accept 20 Dec, 2020 Reviewer # 1 agreed at journal 19 Dec, 2020 Review # 1 received at journal 19 Dec, 2020 Reviewers invited by journal 17 Dec, 2020 First submitted to journal 23 Nov, 2020 Editor assigned by journal 23 Nov, 2020 Submission checks completed at journal 23 Nov, 2020 Editor invited by journal 23 Nov, 2020 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-117198","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Study Protocol","associatedPublications":[],"authors":[{"id":5450448,"identity":"0abbebb7-8508-486c-954a-ed28fed55317","order_by":0,"name":"Lisa Pfadenhauer","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABVUlEQVRIie2PMUvDQBSAXwhclmQ/KegvKKQIVwO1/pUegbgEW+jipAXhOUVXO/gf0kUyRgLJcj/gBqHpkqmCXQoZpCYtamJxcHPIB3cc773vvXcADQ3/ETUEChCWTyXlk23MLA7dpZVtZFBTyOBLUc0/K4Qq3wr8qnSJupCr4OXITLzschH0oH1746tGcALtBztO86B3AZodVhQLybE1FVnHFwmTXDjARDxSDUGBSee84wlnDHpWHWPOJ6xlYKT40iGSY1RUuqZqIL1+ki5rKRjxCXXNqhJr69Y7Rmf+PCMjjhtg8+VWKdzhulA2hTJ8qyn6rpUvCQGOYVGpfyouKVJhOaX6fQv1seVhZE+Fo1IubJ0JZ/T8WCoiYwce2hz1rLpYl2gzmWN0epfEyioP+ocsiWbpK14BS+yM5tjn95qdVheDH+jlFdZCpF6xpzQ0NDQ07PEBouWEiyJOnKIAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0001-5038-8072","institution":"University of Munich: Ludwig-Maximilians-Universitat Munchen","correspondingAuthor":true,"prefix":"","firstName":"Lisa","middleName":"","lastName":"Pfadenhauer","suffix":""},{"id":5450449,"identity":"65b49ee1-e3b0-430f-9226-a544d9d5151f","order_by":1,"name":"Tanja Grath","email":"","orcid":"","institution":"Institute for Medical Processing, Biometry and Epidemiology, LMU Munich","correspondingAuthor":false,"prefix":"","firstName":"Tanja","middleName":"","lastName":"Grath","suffix":""},{"id":5450450,"identity":"5769a705-8efe-422c-9a1e-c4437626965e","order_by":2,"name":"Peter Delobelle","email":"","orcid":"","institution":"Chronic Disease Initiative (CDIA), University of Cape Town, Groote Schuur Hospital","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"","lastName":"Delobelle","suffix":""},{"id":5450451,"identity":"2253b281-fc0d-4290-85bd-a653642fc10f","order_by":3,"name":"Nasreen Jessani","email":"","orcid":"","institution":"Centre for Evidence Based Health Care, Department of Global Health, Faculty of Medicine and Health Sciences, Division of Epidemiology and Biostatistics, Stellenbosch University","correspondingAuthor":false,"prefix":"","firstName":"Nasreen","middleName":"","lastName":"Jessani","suffix":""},{"id":5450452,"identity":"23451ba5-773b-4b47-bda4-5538fea5eb21","order_by":4,"name":"Joerg J Meerpohl","email":"","orcid":"","institution":"Institute for Evidence in Medicine, Medical Center, Faculty of Medicine, University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Joerg","middleName":"J","lastName":"Meerpohl","suffix":""},{"id":5450453,"identity":"2dee9c70-5018-4341-822f-b933cc430015","order_by":5,"name":"Anke Rohwer","email":"","orcid":"","institution":"Center for Evidence Based Health Care, Division of Epidemiology and Biostatistics, Department of Global Health, Faculty of Medicine and Health Sciences, Stellenbosch University","correspondingAuthor":false,"prefix":"","firstName":"Anke","middleName":"","lastName":"Rohwer","suffix":""},{"id":5450454,"identity":"c0450ba8-41b7-477a-a219-826397269303","order_by":6,"name":"Bey-Marrie Schmidt","email":"","orcid":"","institution":"Cochrane South Africa","correspondingAuthor":false,"prefix":"","firstName":"Bey-Marrie","middleName":"","lastName":"Schmidt","suffix":""},{"id":5450455,"identity":"a7a4d1e7-99a3-4ca1-9f04-3dda20d3faec","order_by":7,"name":"Ingrid Toews","email":"","orcid":"","institution":"Cochrane Germany","correspondingAuthor":false,"prefix":"","firstName":"Ingrid","middleName":"","lastName":"Toews","suffix":""},{"id":5450456,"identity":"d3017471-f363-411b-a82d-e74c4ca7301e","order_by":8,"name":"Ann R Akiteng","email":"","orcid":"","institution":"Makerere University College of Health Sciences","correspondingAuthor":false,"prefix":"","firstName":"Ann","middleName":"R","lastName":"Akiteng","suffix":""},{"id":5450457,"identity":"d53b49d1-afec-4711-9055-ebd31f02804b","order_by":9,"name":"Gertrude Chapotera","email":"","orcid":"","institution":"School of Public Health and Family Medicine, College of Medicine, University of Malawi","correspondingAuthor":false,"prefix":"","firstName":"Gertrude","middleName":"","lastName":"Chapotera","suffix":""},{"id":5450458,"identity":"0a73ddf7-a7d4-4151-8e8a-1596858bacf3","order_by":10,"name":"Tamara Kredo","email":"","orcid":"","institution":"Cochrane South Africa","correspondingAuthor":false,"prefix":"","firstName":"Tamara","middleName":"","lastName":"Kredo","suffix":""},{"id":5450459,"identity":"08b8d054-faf1-4d94-9362-4ebc3040fbe3","order_by":11,"name":"Naomi Levitt","email":"","orcid":"","institution":"Groote Schuur Hospital, Chronic Disease Initiative for Africa, University of Cape Town","correspondingAuthor":false,"prefix":"","firstName":"Naomi","middleName":"","lastName":"Levitt","suffix":""},{"id":5450460,"identity":"63156082-7cd7-4470-8750-0505a39a7ec1","order_by":12,"name":"Seleman Ntawuyirushintege","email":"","orcid":"","institution":"University of Rwanda College of Medicine and Health Sciences","correspondingAuthor":false,"prefix":"","firstName":"Seleman","middleName":"","lastName":"Ntawuyirushintege","suffix":""},{"id":5450461,"identity":"23d12d86-686e-4007-8a97-3416e5713bc7","order_by":13,"name":"Kerstin Sell","email":"","orcid":"","institution":"Institute for Medical Information Processing, Biometry and Epidemiology, University of Munich (LMU)","correspondingAuthor":false,"prefix":"","firstName":"Kerstin","middleName":"","lastName":"Sell","suffix":""},{"id":5450462,"identity":"8d417909-4990-4fff-91ed-75ce5889891c","order_by":14,"name":"Eva A Rehfuess","email":"","orcid":"","institution":"Institute of Medical Information Procession, Biometry and Epidemioloy, LMU Munich","correspondingAuthor":false,"prefix":"","firstName":"Eva","middleName":"A","lastName":"Rehfuess","suffix":""}],"badges":[],"createdAt":"2020-11-27 13:26:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-117198/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-117198/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12961-020-00675-w","type":"published","date":"2021-01-18T15:02:16+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":3993391,"identity":"fb4ff09d-0329-4938-adcb-202e19333aff","added_by":"auto","created_at":"2020-12-03 17:18:24","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":32864,"visible":true,"origin":"","legend":"Development, Implementation and Evaluation of Integrated Knowledge Translation Approach in CEBHA+","description":"","filename":"fig1.png","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/ec832dde39189010597829dc.png"},{"id":3993393,"identity":"aedc2f2b-8523-4a17-b9f5-083f7a4ca671","added_by":"auto","created_at":"2020-12-03 17:18:24","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":84673,"visible":true,"origin":"","legend":"Programme Theory of Integrated Knowledge Translation Approach in CEBHA+","description":"","filename":"fig2.png","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/f4f1aab3b3aeacb3c1ddb945.png"},{"id":3993395,"identity":"16d5e2c8-f169-4f56-a99d-f04acbac14b7","added_by":"auto","created_at":"2020-12-03 17:18:25","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":30927,"visible":true,"origin":"","legend":"Overview of Evaluation of Integrated Knowledge Translation Approach in CEBHA+","description":"","filename":"fig3.png","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/a22f86b0f8c58dd0e3f3308e.png"},{"id":13621826,"identity":"cb633481-13e4-4c6c-831d-f294a12720d0","added_by":"auto","created_at":"2021-09-17 07:12:12","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":580572,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/147a91b3-e1e9-4ec5-8411-4858dd6a424d.pdf"},{"id":3993392,"identity":"a634b605-4117-4091-821b-47235e942366","added_by":"auto","created_at":"2020-12-03 17:18:24","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":38769,"visible":true,"origin":"","legend":"Evaluation Domains, Constructs and Data Sources","description":"","filename":"AdditionalFile1.docx","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/dc9a2a284996c5784305dd3d.docx"},{"id":3993394,"identity":"b49d0c69-cee3-4a01-ad13-36f61ebeb86d","added_by":"auto","created_at":"2020-12-03 17:18:25","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":4525585,"visible":true,"origin":"","legend":"CEBHA+ Evaluation Survey for Researchers","description":"","filename":"AdditionalFile2SurveyResearcher.pdf","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/3de6dc00b1b591f784fce61c.pdf"},{"id":3993396,"identity":"6eb0b779-ee2e-4b58-aa3f-fe98b21f9254","added_by":"auto","created_at":"2020-12-03 17:18:25","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":4530520,"visible":true,"origin":"","legend":"CEBHA+ Evaluation Survey for Stakeholders","description":"","filename":"AdditionalFile3SurveyStakeholder.pdf","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/f64ed4f874517e9bfbd9fbc3.pdf"},{"id":3993397,"identity":"7142f78e-fe58-441d-a51f-2af73d487296","added_by":"auto","created_at":"2020-12-03 17:18:26","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":20414,"visible":true,"origin":"","legend":"CEBHA+ Evaluation Interview Guide for Researchers","description":"","filename":"AdditionalFile4InterviewguideresearcherversionV7.docx","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/ec0b397e6c974ec843b3ad10.docx"},{"id":3993398,"identity":"466c2a90-9fe0-498c-9be3-255386cf5dd1","added_by":"auto","created_at":"2020-12-03 17:18:26","extension":"docx","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":21317,"visible":true,"origin":"","legend":"CEBHA+ Evaluation Interview Guide for Stakeholders","description":"","filename":"AdditionalFile5InterviewguidestakeholderversionV9.docx","url":"https://assets-eu.researchsquare.com/files/rs-117198/v1/be18869dc91b200f18c2688c.docx"}],"financialInterests":"","formattedTitle":"\u003cp\u003eMixed Method Evaluation of the CEBHA+ Integrated Knowledge Translation Approach: A Protocol\u003c/p\u003e","fulltext":[{"header":"Background","content":" \u003cp\u003eIntegrated Knowledge Translation (IKT) in public health and healthcare is defined as engagement of knowledge users (e.g., decision-makers) as active participants in the research process (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e). Its purpose is to engage in an interactive, collaborative process with the overarching goal being the co-production of knowledge that is relevant to policy and practice (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). By integrating knowledge users in research, knowledge translation (KT) and research can be considered as two interwoven processes. The involved decision-makers minimise or address potential barriers that may occur when attempting to act upon results. IKT is a steadily evolving process, embedded in a complex web of contextual factors (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). IKT thus constitutes one of a series of related concepts in the field of research co-production, namely participatory research, research collaboration, public/patient involvement and engaged scholarship, with aims ranging from the production of more applicable and useful research to the democratisation of science (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThere are several presumed \u003cb\u003ebenefits\u003c/b\u003e linked to research that embraces IKT, the most important being that it generates knowledge that is more relevant to policy and practice and more likely to be used by decision-makers (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). An IKT approach integrates local knowledge and helps to translate research findings into policy and practice by addressing identified knowledge-practice gaps and providing easier-to-adopt research evidence (\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e). IKT could therefore be considered one step towards producing the right answers to the right questions (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e). The involvement of various stakeholders in health care and public health research is gaining traction (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e) with increasing funder interest, predominantly in high-income countries but also in low- and middle-income countries (LMICs) (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWhile IKT shows promise in bridging the gap between research and practice, there may also be \u003cb\u003echallenges\u003c/b\u003e encountered along the way (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). From the perspective of decision-makers, these include presumptions about academic knowledge, skills and capabilities (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). Functional collaborations require time to be established and effort to be maintained, resources that are often not allocated by funders (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). Stakeholders may have competing demands and agendas (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e) and may consider a range of aspects before engaging in IKT (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). Also, networks relying on personal relationships might be affected by personnel changes (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e) as well as the \u0026ldquo;maturity\u0026rdquo; of the relationships (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e). Last but not least, it should be acknowledged that IKT is not a panacea and is unlikely to be a suitable approach for all research endeavours (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). Thus it is important to gauge where and when an IKT approach adds value to research.\u003c/p\u003e \u003cp\u003eConsidering that IKT has been assumed to have an influential role in evidence-informed policy-making, it is surprising that high-quality evaluations of its processes and outcomes are still scarce (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e). Most evaluation research has been conducted in the form of case studies, using observational study designs (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e). A scoping review concluded that IKT initiatives that were evaluated achieved one or more positive outcomes, however, due to the often poor quality and reporting of these evaluations, no recommendation with regards to what makes IKT strategies effective or not could be made (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). The IKT field should therefore emphasise a thorough description of IKT efforts and their implementation, as well as well-conducted evaluations (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). Also, formative research to develop and test outcome measures such as metrics of research use or research impact on policy making would help to advance the knowledge translation field (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). The field needs empirical and comparative research that tests the benefits that are attributed to IKT, such as the generation of social capital, new relations and cooperative behaviour, greater quality of health services and reduced costs and greater health benefits for those involved (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e), including an increased consideration of contextual factors (\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e). Also, LMICs are seldom represented in IKT evaluations, with evidence to date derived from a small number of primarily Anglophone countries, notably the United States, United Kingdom, Canada, Australia and New Zealand (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIKT represents an integral part of the \u003cem\u003eCollaboration for Evidence-Based Healthcare and Public Health in Africa\u003c/em\u003e (CEBHA+), an African-German research consortium with a focus on non-communicable diseases (research tasks 1\u0026ndash;3: prevention, screening and treatment of cardiovascular disease, diabetes, and hypertension; research task 4: road traffic injuries). CEBHA\u0026thinsp;+\u0026thinsp;is a five-year project funded by the German Federal Ministry of Education and Research (BMBF) as part of the Research Networks for Health Innovation in Sub-Saharan Africa funding initiative. It was set up in 2017 and aims to build long-term capacity and infrastructure for evidence-based health care and public health in Africa. The consortium consists of seven partner institutions in five Sub-Saharan African countries (Ethiopia, Malawi, Rwanda, Uganda, and South Africa; subsequently referred to as five \u0026ldquo;CEBHA\u0026thinsp;+\u0026thinsp;sites\u0026rdquo;) and two partner institutions in Germany. As one of its core goals, CEBHA\u0026thinsp;+\u0026thinsp;seeks to promote the use of contextualised research evidence in decision-making by applying a coordinated IKT approach across all its research activities. To this end, researchers at the five African CEBHA\u0026thinsp;+\u0026thinsp;sites are pursuing focused, long-term engagement with members of the policy-and-practice community (subsequently referred to as \u0026ldquo;stakeholders\u0026rdquo;).\u003c/p\u003e "},{"header":"Objective","content":" \u003cp\u003eThis paper describes the evaluation procedures to assess the process and outcomes of the CEBHA\u0026thinsp;+\u0026thinsp;IKT approach implemented across five African CEBHA\u0026thinsp;+\u0026thinsp;sites. The evaluation is based on a programme theory and will be conducted semi-externally, i.e. one of the two German partner institutions, the LMU Munich (\u0026ldquo;the evaluation team\u0026rdquo;), will lead the evaluation while not being directly involved in the implementation of the IKT strategies at each site.\u003c/p\u003e \u003cp\u003eThe proposed outcome evaluation aims to assess whether an IKT approach ultimately contributes to increased uptake of contextualised research in policy and practice, while also paying attention to intermediate outcomes such as increased relevance or applicability of research in the respective context. The proposed process evaluation, conducted in parallel, aims to shed light on the dose, fidelity and quality of the IKT strategies implemented at each site (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). Continuous monitoring, conducted by each implementing African partner site, is envisioned to refine the IKT strategies as they are being implemented.\u003c/p\u003e \u003cp\u003eHaving conceptualised the CEBHA\u0026thinsp;+\u0026thinsp;IKT approach as a complex intervention in a complex system, our evaluation is expected to support an in-depth understanding of what works, for whom, and in which circumstances. We aim to enhance our understanding of IKT within the CEBHA\u0026thinsp;+\u0026thinsp;consortium and beyond; to enhance the IKT approach over the course of the project; and, to contribute to the IKT research field as a whole.\u003c/p\u003e "},{"header":"Methods","content":"\u003ch2\u003eIntervention: Overarching CEBHA+ IKT approach and site-specific IKT strategies\u003c/h2\u003e\n\u003cp\u003eThe IKT approach in CEBHA+ can be conceptualized as an intervention within a complex social system. By implementing a systematic approach to engaging with stakeholders, building new or strengthening existing relationships, we aim to disrupt traditional knowledge translation approaches (26). We do this by redistributing and transforming resources, for example by having dedicated personnel for IKT (\"IKT focal points\u0026rdquo;, one or two staff per site) that coordinate and implement local IKT efforts. The IKT focal points and the IKT evaluation team at LMU Munich make up the CEBHA+ IKT team.\u003c/p\u003e\n\u003cp\u003eAt each CEBHA+ site, researchers engage with stakeholders. In the context of IKT in healthcare and public health, a concerned stakeholder can be defined as an entity responsible, involved or affected by health-related decisions that can be informed by research evidence (27, 28). Entities can be individuals, organisations, groups or networks operating at a local, regional, national or supra-national level. They can comprise the public, civil society organizations, the media, public health practitioners and service providers as well as public health policy-makers and industry actors across multiple sectors (29). These entities can be directly or indirectly affected \u0026ndash; in positive or negative ways \u0026ndash; by an effort or the actions of an entity (30). We target decision-makers, i.e. policy and practice stakeholders holding a position that allows them to make system-level, organizational or technical decisions that affect the general health of communities or populations. These decisions can be made on a more political or a more technical (i.e. programmatic) level within the health sector as well as in other sectors. Our primarily targeted stakeholder group might be extended as we proceed with the CEBHA+ IKT approach.\u003c/p\u003e\n\u003cp\u003eInitially, the IKT evaluation team at LMU Munich conducted a literature review on IKT, its mechanisms and effectiveness. We conducted forward citation tracking from a recent scoping review (19). This search (Feb 2018) did not yield updates of the review or more recently published evaluations of IKT approaches. Informed by the literature review and discussions with the IKT focal points, we developed a programme theory of the overarching CEBHA+ IKT approach. This programme theory was operationalized into a menu of options of IKT activities that researchers and stakeholders could adapt to their respective research tasks and contexts.\u003c/p\u003e\n\u003cp\u003eThe overall CEBHA+ IKT approach comprises six steps, of which four are site-specific and include development of IKT strategies that are tailored to each CEBHA+ site. These site-specific IKT strategies recognise existing relationships at the different institutions, the local context, and the varying needs of different research tasks within CEBHA+. A coordinated IKT approach ensures that site-specific IKT strategies are harmonised to the extent reasonable. In terms of activities, the six steps towards a coordinated IKT approach in CEBHA+ are described in Table 1 and visualized in Figure 1.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab1\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eOverview of Development, Implementation and Evaluation of the Integrated Knowledge Translation Approach in CEBHA+\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003e\u003cstrong\u003eStep\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003e\u003cstrong\u003eDate\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eNov 2018\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eIn a foundational workshop developed and implemented by the Centre for Evidence Based Health Care, Stellenbosch University (31), one of the South African CEBHA+ partners, IKT focal points from each CEBHA+ site were introduced to the concepts of evidence-informed decision-making, IKT, and the overarching CEBHA+ IKT approach. Participants were introduced to the programme theory and received practical training on how to develop an IKT strategy (stakeholder mapping, stakeholder analysis, stakeholder engagement strategies) and draft a site-specific IKT strategy.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eNov 2018\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eWorkshop participants acted as multipliers within their country teams and shared their knowledge and skills gained during the workshop with the rest of their team members. By using the tools introduced during the workshop, country teams were requested to jointly agree on priority stakeholders with whom to engage over the course of the CEBHA+ project and to refine the site-specific IKT strategy drafted during the foundational workshop.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eNov 2018 \u0026ndash; Jan 2019\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eEach country team consulted priority stakeholders to introduce CEBHA+ and IKT and to gauge their interest in the research topic as well as their preferences for engaging with the CEBHA+ research team.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eFeb 2019\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eCountry teams met to review and synthesise the information obtained in previous steps and to finalise the site-specific stakeholder analysis and IKT strategy.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 5\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eFeb 2019 \u0026ndash; Dec 2022\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eCountry teams are implementing and refining their site-specific IKT strategies and monitoring the indicators they defined as part of their site-specific IKT strategy.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"9%\"\u003e\n\u003cp\u003eStep 6\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"10%\"\u003e\n\u003cp\u003eFeb 2020 \u0026ndash; Jul 2022\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"79%\"\u003e\n\u003cp\u003eA two-step evaluation, by LMU Munich, of the coordinated IKT approach and the site-specific IKT strategies will assess the added value of the IKT approach in CEBHA+ (stage one ongoing, stage two scheduled for Mid-2022)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCaptions to Figure 1: This figure visualizes the process of developing, implementing and evaluating the Integrated Knowledge Translation Approach in CEBHA+. The dark shaded boxes describe steps which were pursued at the level of the research consortium, while the light shaded boxes describe steps pursued at the respective sites. Step 6 (evaluation) is pursued at both the consortium and individual site level.\u003c/p\u003e\n\u003cp\u003eThese steps are not necessarily pursued in a linear manner across all sites but require a certain degree of iteration, i.e. steps 2 to 5 may be repeated as the stakeholder landscape changes. Ideally, stakeholders are engaged in the research process from an early stage (11). This engagement can comprise shaping the research questions, deciding on the methodology, involvement in data collection and development of tools, interpreting the findings and supporting dissemination of research results at the research task level (2). At the level of the CEBHA+ consortium, this \u0026ldquo;IKT thinking\u0026rdquo; has been embraced since the initial development of the project proposal, where relevant stakeholders from the respective partner countries were included in the process of setting research priorities. Over the course of the project, practical experiences with the development and implementation of site-specific IKT strategies, insights gained from monitoring and evaluation, or new scientific evidence may lead to adaptations in the overall IKT approach and/or the site-specific IKT strategies.\u003c/p\u003e\n\u003ch2\u003eProgramme theory\u003c/h2\u003e\n\u003cp\u003eIn order to guide the development and evaluation of the CEBHA+ IKT approach, we developed a comprehensive programme theory (see Figure 2). This builds upon two main frameworks: the Context and Implementation of Complex Interventions (CICI) framework (32) and a visual representation of IKT approaches, influencing factors, and outcomes as described in a scoping review by Gagliardi and colleagues (19). The CICI framework was used to structure the context and implementation component of the evaluation by providing a priori categories which informed the development of both the qualitative interview guide and the survey. The framework by Gagliardi provides an overview of empirical evaluations of IKT approaches, relevant enablers and barriers, preconditions, and outcomes. Moreover, it flags relevant and partly validated evaluation tools which informed the development of our survey. The contributions of these two frameworks were integrated to show how the CEBHA+ IKT approach is assumed to increase the use of contextualized research evidence in policy and practice decision-making.\u003c/p\u003e\n\u003cp\u003eAs shown in Figure 2, the programme theory describes the coordinated IKT approach as an \u003cem\u003eintervention\u003c/em\u003e implemented across the CEBHA+ consortium. In CEBHA+, a coordinated IKT approach is understood as an iterative process in which 1) relationship building and strengthening, 2) capacity building, and 3) collaborative research between CEBHA+ researchers and their priority stakeholders represent integral components of the intervention. Capacity building, relationship building, and their maintenance and collaborative research are expected to take place in an interactive manner and repeatedly over time, thereby reinforcing each other (3).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe implementation of these intervention components is expected to result in several \u003cem\u003eintervention outcomes. \u003c/em\u003eThe literature indicates that strong relationships increase mutual understanding (e.g. of language, work style, needs, constraints) and trust among researchers and decision-makers engaging in a partnership and that these influence attitudes towards the partners\u0026rsquo; professional environment (i.e. attitudes towards research among stakeholders versus attitudes towards the policy-and-practice field among researchers) (19, 33-36). With this in mind, CEBHA+ researchers seek to build and enhance relationships with priority stakeholders. The experience of conducting collaborative research would ideally lead to appreciation for the collaborative process (33), promote a more diverse range of partners to engage in, and encourage involvement throughout the research process. Mutual capacity-building can occur in the form of improved access to information and/or contacts due to the research partnership, broadened perspectives and skills (e.g. research skills), and improved capacity for collaboration (33-35).\u003c/p\u003e\n\u003cp\u003eMoving on to \u003cem\u003eintermediate outcomes\u003c/em\u003e, successful research partnerships between researchers and stakeholders is assumed to result in an increased perceived value of research evidence in terms of relevance, better applicability to the stakeholders\u0026rsquo; field of activity as well as credibility. While both actual and perceived value are potentially affected by IKT efforts, studies show the individual perception of value has greater behavioural implications than the actual value of research evidence (37). We therefore focus on the perceived value of CEBHA+ research evidence with respect to its importance to the actual consideration and utilization of research evidence in decision-making. In turn, research evidence that is perceived as credible, relevant, and usable is likely to result in an increased intention to use that evidence in policy and practice decision-making. Subsequently, the intention to use research evidence is hypothesised to lead to the actual consideration of findings in a specific decision in another step towards the \u003cem\u003efinal outcome:\u003c/em\u003e the use of contextualized research in policy and practice decision-making.\u003c/p\u003e\n\u003ch2\u003eEvaluation of the CEBHA+ IKT approach\u003c/h2\u003e\n\u003ch3\u003eOverview of evaluation approach\u003c/h3\u003e\n\u003cp\u003eThis evaluation is theory-driven. The programme theory provides the main structure for the evaluation and informs our choice of methods for data collection as well as data analysis. A comparative case study (CCS) design (38) will be used, including mixed methods to facilitate in-depth examination of IKT evaluation in each site. Each site (Ethiopia, Malawi, Rwanda, South Africa, Uganda) is considered a case. A CCS is undertaken over time and emphasizes comparison within and across contexts and thus allows to explore, understand, and explain how context influences the success of an intervention (39). Studying each case with its distinct IKT strategy at two time points will provide an opportunity to compare and contrast findings within and across sites.\u003c/p\u003e\n\u003cp\u003eAs part of the process evaluation, we will look at how and why IKT works in the respective contexts. In line with the CICI framework, we will look at the different stages in the implementation process, the diverse implementation strategies employed in different sites and the respective implementation agents (32). We will moreover utilize our monitoring to deepen our understanding of the implementation process. This may be complemented by including quantitative implementation outcomes, such as acceptability and feasibility in the late-stage evaluation. As part of the outcome evaluation, we will investigate the intervention, intermediate, and final outcomes, as defined in the programme theory.\u003c/p\u003e\n\u003cp\u003eAs shown in Figure 3, the evaluation process will comprise two phases, i.e. an early-stage assessment (early 2020) and a late-stage assessment (early 2022). In addition to the data analyses performed at the end of each phase, we will conduct an integrated analysis after the completion of phases 1 and 2 (until mid-2022). The evaluation process will comprise five different procedures: (a) quantitative surveys with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (b) qualitative interviews with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (c) continuous monitoring to capture ongoing IKT activities (mainly covered by the documentation of interactions between CEBHA+ researchers and policy-and-practice stakeholders reported in quarterly updates); (d) policy document analysis; and (e) integrated and cross-case analysis at the end of the evaluation. Each of the procedures will be conducted in all five CEBHA+ sites.\u0026nbsp;\u003c/p\u003e\n\u003ch3\u003eData collection\u003c/h3\u003e\n\u003cp\u003eGiven the complexity of the initial programme theory and the multitude of outcomes along the causal path from the intervention to the intended outcome, we will use a combination of qualitative (semi-structured interviews, qualitative analysis of policy documents, IKT updates) and quantitative data collection approaches (survey). The survey is intended to touch upon all dimensions covered by the programme theory. The qualitative interviews, however, will allow us to explore how IKT exerts an influence in more depth. Also, we will focus on external context factors (macro-context) that impact on IKT, the process of IKT, as well as intermediate and final outcome(s). Two distinct surveys and two interview guides target researchers and stakeholders, respectively. This is necessary to reflect the different perspectives of individuals engaged in a partnership.\u003c/p\u003e\n\u003ch2\u003eDevelopment of data collection instruments\u003c/h2\u003e\n\u003cp\u003eThe underlying programme theory was used to develop survey and interview instruments. We initially constructed the survey based on suitable pre-existing instruments included in the review by Gagliardi et al. (2015). Literature searches were then conducted and instruments identified to inform the development of additional survey items (e.g. SAGE (40), PreVAiL (41), SEER (42)). Where items were non-existent, we constructed them de novo. The survey comprises both multiple choice and Likert scale questions. We provide an overview of included constructs and sub-constructs as well as their source in the additional files (Additional file 1).\u003c/p\u003e\n\u003cp\u003eThe interview guides and surveys were pilot tested among two researchers (Rwanda and South Africa, one male, one female) who are not involved in the IKT implementation process. We assessed the tools for errors, comprehensibility, acceptability, and understandability. We were not able to pilot-test our tools with decision-makers from policy-and-practice due to their limited time availability and the high opportunity cost. Both the interview guide and survey were slightly adapted to account for the changed project context due to the COVID-19 pandemic and are available as additional files (Additional files 2 - 5).\u003c/p\u003e\n\u003cp\u003eInsights gained during the early-stage evaluation will inform the interview guides and surveys used in the late-stage evaluation. In addition to the topics covered in the early-stage assessment, participants will also be asked to describe how the research partnership has changed over time and to describe, where applicable, examples of successful translation of research evidence into policy or practice because of the research partnership.\u003c/p\u003e\n\u003ch2\u003eSampling\u003c/h2\u003e\n\u003cp\u003eSelection of study participants for the surveys and interviews will reflect the following criteria:\u003c/p\u003e\n\u003cp\u003e- A CEBHA+ researcher or a priority stakeholder as identified by the respective CEBHA+ country team; AND\u003c/p\u003e\n\u003cp\u003e- actively involved in a research-stakeholder partnership in CEBHA+; AND\u003c/p\u003e\n\u003cp\u003e- 18 years or above; AND\u003c/p\u003e\n\u003cp\u003e- lives and works in the respective CEBHA+ country; AND\u003c/p\u003e\n\u003cp\u003e- is able to understand and articulate him/herself in English\u003c/p\u003e\n\u003cp\u003eWith respect to sample size, we aim to interview five to ten participants per site, ideally both researchers and key stakeholders.\u003c/p\u003e\n\u003ch2\u003eRecruitment\u003c/h2\u003e\n\u003cp\u003eThe evaluators will invite all eligible CEBHA+ researchers to participate in the study via email and will introduce the study. Eligible stakeholders will be invited to participate by their respective research counterparts and will be provided with the same information.\u003c/p\u003e\n\u003cp\u003eParticipants willing to be enrolled will be contacted by the evaluators via email to arrange a date for the qualitative interview. A written informed consent form will be provided, detailing the study\u0026rsquo;s goals, estimated duration, scope, its voluntary nature, the pseudonymity and confidentiality of responses, and whom to contact regarding questions. Participants will be asked to return the digitally signed informed consent form or a photo/scan of the manually signed form. Participants will also receive an invitation link to access the online survey.\u003c/p\u003e\n\u003ch2\u003eSurvey procedures\u003c/h2\u003e\n\u003cp\u003eThe survey will be administered online using the LimeSurvey tool (\u003ca href=\"https://www.limesurvey.org/en\"\u003ehttps://www.limesurvey.org/en\u003c/a\u003e) and participants requested to complete the survey prior to the qualitative interviews. At the beginning of the questionnaire, participants will receive information regarding the study. Key information regarding the informed consent form will be reiterated. Participants who are not able to complete the survey before the qualitative interviews will receive reminders via email and orally during the interview.\u003c/p\u003e\n\u003ch2\u003eInterview procedures\u003c/h2\u003e\n\u003cp\u003eThe interviews will be conducted by a member of the CEBHA+ evaluation team or trained research staff at the CEBHA+ site using an interview guide. Where feasible, notes will be taken by a second member of the CEBHA+ evaluation team or by trained research staff at the CEBHA+ site, who are not directly involved with the site-specific IKT strategy. We consider it important that the interviewer is not involved in the practical implementation of IKT at the respective CEBHA+ site and thus unknown to the interviewed stakeholder to reduce interviewer bias. All interviews will be conducted in English.\u003c/p\u003e\n\u003cp\u003eIn light of the COVID-19 pandemic interviews will be conducted virtually (e.g. via skype, zoom or other) unless the interviewee insists on a face-to-face interview, the pandemic situation permits, and precautionary measures are in place. Where conducted in person, interviews will be carried out in a place of the interviewee\u0026rsquo;s choice, e.g. the office of the stakeholder or researcher. As a minimum requirement, the chosen place should be a separate room or another quiet place to avoid disturbances and outside observers. The estimated duration of a single interview is 30 to 60 minutes.\u003c/p\u003e\n\u003cp\u003eThe interviewees will be asked to describe their perspective on the research partnership, contextual factors that hinder or facilitate the partnership or its outcomes, the expected benefits and ways to improve the partnership and/or outcomes. Interviews will be audio-recorded and transcribed verbatim. Interviewees will be given the opportunity to review the transcript, ask for modifications and give their approval for further use of the data.\u003c/p\u003e\n\u003ch2\u003eFurther data sources\u003c/h2\u003e\n\u003ch3\u003eQuarterly IKT updates\u003c/h3\u003e\n\u003cp\u003eserve as a means of continuous exchange among the IKT teams. The updates take place during a virtual call and are facilitated by a structured reporting tool. This tool captures noteworthy interactions between researchers and stakeholders and their outputs. It also contains questions on the learnings and challenges encountered at each site. All updates are submitted to the evaluation team before the calls and will be analysed in Phase 1 and 2.\u003c/p\u003e\n\u003ch3\u003ePolicy documents\u003c/h3\u003e\n\u003cp\u003ewill be identified and analysed for the late-stage assessment. Relevant policy documents are those produced between 2017 and 2022 (i.e. the course of the CEBHA+ project) and on topics related to CEBHA+ research activities at the five research sites. Documents concerned with laws, regulations, plans and major programmes from different tiers of government will be selected. Other documents considered will include those issued by multi-sectoral working groups or semi-independent bodies with a mandate to devise strategies or plans. These policy documents will be identified through the quarterly IKT updates, interviews and stakeholder websites (incl. meeting agendas, minutes, protocols, or reports).\u003c/p\u003e\n\u003ch2\u003eData analysis\u003c/h2\u003e\n\u003ch3\u003eDescriptive analysis of survey data\u003c/h3\u003e\n\u003cp\u003eAll quantitative data will be analysed descriptively using IBM SPSS Statistics 27.0. Survey data will be analysed in conjunction with qualitative data for triangulation at the end of phase 1 and phase 2 (Figure 3).\u003c/p\u003e\n\u003ch3\u003eQualitative analysis of interviews\u003c/h3\u003e\n\u003cp\u003eInterview transcripts will be analysed in MaxQDA using content analysis as described by Schreier (2012) (43). Codes will be developed both deductively (informed by the programme theory and resulting interview guide) and inductively (data driven).\u003c/p\u003e\n\u003cp\u003eFor each of the sites, the interviews will be coded independently by two coders, with one of them being from the same site with a comparable cultural background as the interviewee in order to include both an insider as well as outsider perspective (44). Codes will be discussed among the two coders and adapted until agreement is reached. For each site, there will be two sets of category systems, one for the researcher and one for the stakeholder dataset. While we expect overlaps, there are likely to be distinctions. This will ultimately lead to a code book that will be applied to all interviews, while allowing for inductively developed, site-specific codes.\u003c/p\u003e\n\u003ch3\u003eDocument analysis of policy documents\u003c/h3\u003e\n\u003cp\u003eThe policy documents identified in the late-stage evaluation phase will be analysed through document analysis, a systematic procedure for reviewing or evaluating printed or electronic documents, which entails finding, selecting, appraising and synthesising data contained in the retrieved documents (45). We will look for specific indicators of CEBHA+-generated research evidence uptake into practice or policymaking. Qualitative content analysis will be applied to relevant sections of the document (45), by using an a priori developed list of categories which corresponds to the outcome indicators in our programme theory. Text passages that discuss any of these a priori categories will be coded according to this list, while others will be coded under new categories. The resulting category systems will be shared and discussed among the CEBHA+ IKT team members undertaking the document analyses.\u003c/p\u003e\n\u003ch3\u003eTriangulation of data at each site\u003c/h3\u003e\n\u003cp\u003eAt the end of phases 1 and 2, qualitative and quantitative data will be integrated for each case using the triangulation method as described by Moran-Ellis and colleagues (2006), \u0026ldquo;following a thread\u0026rdquo; (46). Triangulation is a general approach whereby the convergence, complementarity, and dissonance of results on related research questions, obtained through different methodological approaches, sources, theoretical perspective, or researchers are explored. With this method, an initial analysis of multiple datasets (survey, interviews, quarterly IKT updates, policy document analysis) will be undertaken separately for each case (i.e. each site). At this stage, themes and analytical questions that require further investigation will be identified. These themes or questions \u0026ndash; the thread \u0026ndash; are followed throughout the datasets to capture corresponding, contradicting as well as lacking findings \u003cem\u003ewithin cases. \u003c/em\u003eThe final outcome of this step is to create concise case descriptions for each of the five individual CEBHA+ sites.\u003c/p\u003e\n\u003ch3\u003eCross-case \u003cem\u003eanalysis\u003c/em\u003e across sites\u003c/h3\u003e\n\u003cp\u003eAfter data collection and triangulation (2022), we will conduct a cross-case analysis across findings from the five CEBHA+ sites, the two points of data collection and various methods of data collection. Cross-case analysis is a research method that facilitates the comparison of commonalities and differences in the events, activities, and processes that are the units of analyses in case studies. The aim is to compare key findings \u003cem\u003eacross cases\u003c/em\u003e (i.e. across sites). Drawing upon the programme theory, we will contrast what we found at each site with regards to the components of the programme theory. This cross-case analysis will also serve as basis for reflections on the community of IKT practice across sites within the CEBHA+ consortium. Insights from the cross-case analysis are likely to lead to refinements of the initial programme theory, leading to a middle-range theory.\u003c/p\u003e\n\u003ch2\u003eData Management\u003c/h2\u003e\n\u003cp\u003eData collection, retention and analysis will be conducted adhering to EU data protection regulations. All data (digital records, transcripts, interview notes, survey data) will be kept confidentially on a secured institutional IT infrastructure provided by the Institute for Medical Information Processing, Biometry, and Epidemiology (IBE) at the LMU Munich, which meets the requirements of an institute for research on highly sensitive patient and study data. For analysis, researchers from the respective CEBHA+ sites will have access to the pseudonymized interview data after transcription. All analyses will be undertaken in the cloud and no data will be stored locally at any of the sites.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"Discussion","content":" \u003cp\u003eTo our knowledge, this protocol is one of the first attempts to evaluate an IKT approach across multiple sites and countries using a longitudinal design. It suggests an overall methodology and detailed methods which we deem appropriate to capture the complexities and context dependencies of the different IKT strategies employed across sites and highlight the important characteristics of individuals involved in delivering IKT. The opportunity that a thorough development, implementation and evaluation of IKT is possible within the context of CEBHA\u0026thinsp;+\u0026thinsp;can be attributed to the open funding call issued by the German Federal Ministry of Education and Research (\u003cem\u003eBundesministerium f\u0026uuml;r Bildung und Forschung\u003c/em\u003e, BMBF) in 2013. Whilst the funder\u0026rsquo;s interest in IKT as a means of enhancing research impact is highly commendable and in line with several other international funding agencies requiring that knowledge translation be integrated into proposals (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e), a critical meta-discussion is needed on how IKT interventions can be aligned with the realities of international research collaborations and the contexts and capacities of the different partner institutions.\u003c/p\u003e \u003cp\u003eWe have encountered and will continue to encounter conceptual, methodological and practical challenges in the proposed evaluation. Conceptually, we are seeking to understand the context and individual dependency of IKT which will inevitably lead to heterogeneity. We aim to capture these heterogeneities by employing multiple methods; we endeavour to paint a holistic picture by integrating the perspectives of researchers and stakeholders in the evaluation.\u003c/p\u003e \u003cp\u003eMethodologically, we have struggled to identify appropriate and valid ways of monitoring highly heterogeneous IKT strategies. This will be particularly important when determining the dose-response-relationship between the effort invested in stakeholder engagement and related outcomes. We are optimistic that the triangulation of different sources of data will provide valuable insights. We have conceptualized this evaluation as semi-external, with the evaluation team not being involved in the implementation of local IKT strategies. While this more distant perspective on the realities of IKT on the ground facilitates a largely independent evaluation, the interpretation of evaluation findings will need to be complemented by local researchers\u0026rsquo; contextual understanding. This will be facilitated by integrating one researcher per site with the analysis and interpretation of the generated data. This researcher will not have participated in the evaluation.\u003c/p\u003e \u003cp\u003ePractically, CEBHA\u0026thinsp;+\u0026thinsp;activities just like those of most other health research institutions or networks have been severely disrupted by the COVID-19 pandemic. Both on the side of the researchers and the stakeholders, resources initially intended for CEBHA\u0026thinsp;+\u0026thinsp;activities were focused towards responding to emerging needs due to the pandemic. Additionally, we had to defer from conducting this evaluation face-to-face and resort to virtual interviews. These developments led to delays and reiterations in our procedures. On the other hand, the COVID-19 pandemic has most clearly demonstrated the international need for research evidence in practice and policy decision making and made a very strong case for IKT. While we cannot anticipate the developments over the next year(s), we believe that the IKT experiences in CEBHA\u0026thinsp;+\u0026thinsp;thus far do contribute to the international research on IKT, especially given the intensive collaboration of researchers and policymakers during the pandemic.\u003c/p\u003e "},{"header":"Conclusion","content":" \u003cp\u003eWhile involving decision-makers in the research process is gaining traction worldwide, there is still very little knowledge regarding (i) whether a systematic approach makes a difference to evidence uptake, (ii) the mechanisms that make IKT successful, and (iii) relevant differences across socio-cultural contexts. The CEBHA\u0026thinsp;+\u0026thinsp;IKT approach represents a carefully developed intervention designed to increase the use of contextualized research in policy and practice decision-making. As an approach that is implemented in a coordinated manner across five African countries, it represents a rare opportunity for a thorough evaluation and learning from differences across multiple sites. The evaluation described here is intended to provide relevant insights on all the above aspects, notably in Sub-Saharan Africa, and is expected to contribute to the science of IKT overall.\u003c/p\u003e "},{"header":"List Of Abbreviations","content":"\u003ctable border=\"1\"\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eIKT/KT\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eIntegrated Knowledge Translation/Knowledge translation\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eCEBHA+\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eCollaboration for Evidence-Based Healthcare and Public Health in Africa\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eCCS\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eComparative Case Study\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eLMIC\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"302\"\u003e\n\u003cp\u003eLow- and Middle-Income Country\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Declarations","content":"\u003ch2\u003eEthics approval and consent to participate\u003c/h2\u003e\n\u003cp\u003eWe applied and were granted ethics approval from the following institutions:\u003c/p\u003e\n\u003cp\u003e- University of Munich (LMU), Germany (19-633)\u003c/p\u003e\n\u003cp\u003e- Research and Ethics Committee, School of Public Health, College of Health Sciences, Makerere University, Uganda (IRB00011353; study protocol number 469)\u003c/p\u003e\n\u003cp\u003e- Armauer Hansen Research Institute (P0/31/20)\u003c/p\u003e\n\u003cp\u003e- College of Medicine Research Ethics Committee, University of Malawi (P.11/19/2850)\u003c/p\u003e\n\u003cp\u003e- Rwanda National Ethics Committee, Ministry of Health, Kigali, Rwanda (074/RNEC/2020)\u003c/p\u003e\n\u003cp\u003e- Human Research Ethics Committee, Faculty of Health Sciences, University of Cape Town (026/2020)\u003c/p\u003e\n\u003ch2\u003eConsent for publication\u003c/h2\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch2\u003eAvailability of data and materials\u003c/h2\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch2\u003eCompeting interests\u003c/h2\u003e\n\u003cp\u003eThe authors declare that they have no competing interests. All authors are part of the CEBHA+ consortium.\u003c/p\u003e\n\u003ch2\u003eFunding\u003c/h2\u003e\n\u003cp\u003eThis work was supported by the German Federal Ministry of Education and Research (Bundesministerium f\u0026uuml;r Bildung und Forschung (BMBF)) as part of the Research Networks for Health Innovation in Sub-Saharan Africa funding initiative. The funder had no role in design of the study and in writing the manuscript.\u003c/p\u003e\n\u003ch2\u003eAuthors' contributions\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eAll authors were actively involved in designing the CEBHA+ IKT approach, the evaluation tools, and developing this protocol by providing critical feedback and engaging in numerous discussions. All authors except for LMP, TG, JJM, KS, IT and ER are actively implementing the IKT strategies at the respective sites. LMP, TG, KS and ER drafted the manuscript. All authors critically revised the draft and read and approved the final manuscript.\u003c/p\u003e\n\u003ch2\u003eAcknowledgements\u003c/h2\u003e\n\u003cp\u003eWe would like to thank Jacob Burns for valuable input on the development of the survey. We would like to thank Dr. Irakoze Magnifique from University of Rwanda and Lynn Hendricks from Stellenbosch University for testing and providing valuable feedback on our data collection tools. We would moreover acknowledge all IKT focal points and all CEBHA+ researchers who are actively implementing and monitoring the IKT strategies at each site.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eLawrence LM, Bishop A, Curran J. Integrated Knowledge Translation with Public Health Policy Makers: A Scoping Review. Healthc Policy. 2019;14(3):55-77.\u003c/li\u003e\n\u003cli\u003eKothari A, Wathen CN. A critical second look at integrated knowledge translation. Health Policy. 2013;109(2):187-91.\u003c/li\u003e\n\u003cli\u003eKreindler SA. Advancing the evaluation of integrated knowledge translation. Health research policy and systems. 2018;16(1):104-.\u003c/li\u003e\n\u003cli\u003eGraham ID, McCutcheon C, Kothari A. Exploring the frontiers of research co-production: the Integrated Knowledge Translation Research Network concept papers. Health Research Policy and Systems. 2019;17(1):88.\u003c/li\u003e\n\u003cli\u003eCampbell H, Vanderhoven D. Knowledge that matters: realising the potential of co-production. . Manchester; 2016.\u003c/li\u003e\n\u003cli\u003eGagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: what we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105-6.\u003c/li\u003e\n\u003cli\u003eOrr K, Bennett M. Public Administration Scholarship and the Politics of Coproducing Academic\u0026ndash;Practitioner Research. Public Administration Review. 2012;72(4):487-95.\u003c/li\u003e\n\u003cli\u003eKothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface: managing policymaker-researcher collaboration. Knowledge Management Research \u0026amp; Practice. 2011;9(3):203-14.\u003c/li\u003e\n\u003cli\u003eVan de Ven AH, Johnson PE. Knowledge for theory and practice. Academy of management review. 2006;31(4):802-21.\u003c/li\u003e\n\u003cli\u003eRehfuess EA, Durao S, Kyamanywa P, Meerpohl JJ, Young T, Rohwer A, et al. An approach for setting evidence-based and stakeholder-informed research priorities in low- and middle-income countries. Bull World Health Organ. 2016;94(4):297-305.\u003c/li\u003e\n\u003cli\u003eRycroft-Malone J, Burton CR, Bucknall T, Graham ID, Hutchinson AM, Stacey D. Collaboration and Co-Production of Knowledge in Healthcare: Opportunities and Challenges. Int J Health Policy Manag. 2016;5(4):221-3.\u003c/li\u003e\n\u003cli\u003eMcLean RKD, Graham ID, Bosompra K, Choudhry Y, Coen SE, MacLeod M, et al. Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science. 2012;7(1):57.\u003c/li\u003e\n\u003cli\u003eShearer JC, Abelson J, Kouyat\u0026eacute; B, Lavis JN, Walt G. Why do policies change? Institutions, interests, ideas and networks in three cases of policy reform. Health Policy and Planning. 2016;31(9):1200-11.\u003c/li\u003e\n\u003cli\u003eJessani NS, Boulay MG, Bennett SC. Do academic knowledge brokers exist? Using social network analysis to explore academic research-to-policy networks from six schools of public health in Kenya. Health Policy and Planning. 2015;31(5):600-11.\u003c/li\u003e\n\u003cli\u003eJessani NS, Babcock C, Siddiqi S, Davey-Rothwell M, Ho S, Holtgrave DR. Relationships between public health faculty and decision makers at four governmental levels: a social network analysis. Evidence \u0026amp; Policy: A Journal of Research, Debate and Practice. 2018;14(3):499-522.\u003c/li\u003e\n\u003cli\u003eJessani NS, Valmeekanathan A, Babcock C, Ling B, Davey-Rothwell MA, Holtgrave DR. Exploring the evolution of engagement between academic public health researchers and decision-makers: from initiation to dissolution. Health Research Policy and Systems. 2020;18(1):15.\u003c/li\u003e\n\u003cli\u003eDurose C, Needham C, Mangan C, Rees J. Generating \u0026lsquo;good enough\u0026rsquo; evidence for coproduction. Evidence and Policy. 2015.\u003c/li\u003e\n\u003cli\u003eDurose C, Mangan C, Needham C, Rees J, editors. Evaluating co-production: pragmatic approaches to building the evidence base Political Studies Association Conference; 2014; Manchester.\u003c/li\u003e\n\u003cli\u003eGagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38.\u003c/li\u003e\n\u003cli\u003eGollust SE, Seymour JW, Pany MJ, Goss A, Meisel ZF, Grande D. Mutual Distrust: Perspectives From Researchers and Policy Makers on the Research to Policy Gap in 2013 and Recommendations for the Future. Inquiry. 2017;54:46958017705465.\u003c/li\u003e\n\u003cli\u003eCairney P, Oliver K. Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Res Policy Syst. 2017;15(1):35.\u003c/li\u003e\n\u003cli\u003eCottrell E, Whitlock E, Kato E, Uhl S, Belinson S, Chang C, et al. AHRQ Methods for Effective Health Care. Defining the Benefits of Stakeholder Engagement in Systematic Reviews. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014.\u003c/li\u003e\n\u003cli\u003eEdwards A, Zweigenthal V, Olivier J. Evidence map of knowledge translation strategies, outcomes, facilitators and barriers in African health systems. Health Research Policy and Systems. 2019;17(1):16.\u003c/li\u003e\n\u003cli\u003eTricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open. 2016;6(12):e013929-e.\u003c/li\u003e\n\u003cli\u003eSaunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134-47.\u003c/li\u003e\n\u003cli\u003eHawe P, Shiell A, Riley T. Theorising interventions as events in systems. American journal of community psychology. 2009;43(3-4):267-76.\u003c/li\u003e\n\u003cli\u003eLemke AA, Harris-Wai JN. Stakeholder engagement in policy development: challenges and opportunities for human genomics. Genetics in medicine : official journal of the American College of Medical Genetics. 2015;17(12):949-57.\u003c/li\u003e\n\u003cli\u003eConcannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. Journal of General Internal Medicine. 2014;29(12):1692-701.\u003c/li\u003e\n\u003cli\u003eLavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, Knowledge Transfer Study G. How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers? Milbank Quarterly. 2003;81(2):221-48.\u003c/li\u003e\n\u003cli\u003eCommunity Tool Box. Section 8. Identifying and Analyzing Stakeholders and Their Interests 2017 [Available from: \u003cu\u003ehttps://ctb.ku.edu/en/table-of-contents/participation/encouraging-involvement/identify-stakeholders/main\u003c/u\u003e.\u003c/li\u003e\n\u003cli\u003eJessani NS, Hendricks L, Nicol L, Young T. University Curricula in Evidence-Informed Decision Making and Knowledge Translation: Integrating Best Practice, Innovation, and Experience for Effective Teaching and Learning. Frontiers in Public Health. 2019;7(313).\u003c/li\u003e\n\u003cli\u003ePfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implementation Science. 2017;12(1):21.\u003c/li\u003e\n\u003cli\u003eSoper B, Yaqub O, Hinrichs S, Marjanovich S, Drabble S, Hanney S, et al. CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation. Journal of Health Services Research \u0026amp; Policy. 2013;18(3_suppl):53-64.\u003c/li\u003e\n\u003cli\u003eVan Olphen J, Ottoson J, Green L, Barlow J, Koblick K, Hiatt R. Evaluation of a partnership approach to translating research on breast cancer and the environment. Prog Community Health Partnersh. 2009;3(3):199.\u003c/li\u003e\n\u003cli\u003eBowen S, Martens P. Demystifying knowledge translation: learning from the community. J Health Serv Res Policy. 2005;10(4):203-11.\u003c/li\u003e\n\u003cli\u003eEriksson CC-G, Fredriksson I, Fr\u0026ouml;ding K, Geidne S, Pettersson C. Academic practice\u0026ndash;policy partnerships for health promotion research: Experiences from three research programs. Scandinavian Journal of Public Health. 2014;42(15_suppl):88-95.\u003c/li\u003e\n\u003cli\u003eDamschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.\u003c/li\u003e\n\u003cli\u003eCampbell S. Comparative case study. Encyclopedia of case study research. 2010;1:174-6.\u003c/li\u003e\n\u003cli\u003eBartlett L, Vavrus F. Comparative Case Studies: An Innovative Approach. Nordic Journal of Comparative and International Education (NJCIE). 2017;1(1).\u003c/li\u003e\n\u003cli\u003eMakkar SR, Brennan S, Turner T, Williamson A, Redman S, Green S. The development of SAGE: A tool to evaluate how policymakers\u0026rsquo; engage with and use research in health policymaking. Research Evaluation. 2016;25(3):315-28.\u003c/li\u003e\n\u003cli\u003eKothari A, Sibbald SL, Wathen CN. Evaluation of partnerships in a transnational family violence prevention network using an integrated knowledge translation and exchange model: a mixed methods study. Health Res Policy Syst. 2014;12:25.\u003c/li\u003e\n\u003cli\u003eBrennan SE, McKenzie JE, Turner T, Redman S, Makkar S, Williamson A, et al. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research. Health research policy and systems. 2017;15(1):1-.\u003c/li\u003e\n\u003cli\u003eSchreier M. Qualitative Content Analysis in Practice: SAGE Publications; 2012.\u003c/li\u003e\n\u003cli\u003eThe SAGE Encyclopedia of Qualitative Research Methods. 2008.\u003c/li\u003e\n\u003cli\u003eBowen GA. Document analysis as a qualitative research method. Qualitative research journal. 2009;9(2):27-40.\u003c/li\u003e\n\u003cli\u003eMoran-Ellis J, Alexander VD, Cronin A, Dickinson M, Fielding J, Sleney J, et al. Triangulation and integration: processes, claims and implications. Qualitative Research. 2006;6(1):45-59.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"health-research-policy-and-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"hrps","sideBox":"Learn more about [Health Research Policy and Systems](http://health-policy-systems.biomedcentral.com/)","snPcode":"12961","submissionUrl":"https://submission.nature.com/new-submission/12961/3","title":"Health Research Policy and Systems","twitterHandle":"@HarpsJournal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Integrated Knowledge Translation, co-production, Sub-Saharan Africa, non-communicable diseases, evaluation, mixed methods, comparative case study, complexity","lastPublishedDoi":"10.21203/rs.3.rs-117198/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-117198/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground\u003c/p\u003e\u003cp\u003eThe Collaboration for Evidence-based Healthcare and Public Health in Africa (CEBHA+) is a research consortium concerned with the prevention, diagnosis and treatment of non-communicable diseases. CEBHA+ seeks to engage policymakers and practitioners throughout the research process in order to build lasting relationships, enhance evidence uptake, and create long-term capacity among partner institutions in Ethiopia, Malawi, Rwanda, South Africa and Uganda in collaboration with two German universities. This integrated knowledge translation (IKT) approach includes the formal development, implementation and evaluation of country specific IKT strategies.\u003c/p\u003e\u003cp\u003eMethods\u003c/p\u003e\u003cp\u003eWe have conceptualised the CEBHA+ IKT approach as a complex intervention in a complex system. We will employ a comparative case study (CCS) design and mixed methods to facilitate an in-depth evaluation. We will use quantitative surveys, qualitative interviews, quarterly updates, and a policy document analysis to capture the process and outcomes of IKT across the African CEBHA+ partner sites. We will conduct an early stage (early 2020) and a late-stage evaluation (early 2022), triangulate the data collected with various methods at each site and subsequently compare our findings across the five sites.\u003c/p\u003e\u003cp\u003eDiscussion\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/p\u003e\u003cp\u003eEvaluating a complex intervention such as the CEBHA+ IKT approach is complicated, even more so when undertaken across five diverse countries. Despite conceptual, methodological and practical challenges, our comparative case study addresses important evidence gaps: While involving decision-makers in the research process is gaining traction worldwide, we still know very little regarding (i) whether this approach really makes a difference to evidence uptake, (ii) the mechanisms that make IKT successful, and (iii) relevant differences across socio-cultural contexts. The evaluation described here is intended to provide relevant insights on all of these aspects, notably in countries in Sub-Saharan Africa, and is expected to contribute to the science of IKT overall.\u003c/p\u003e","manuscriptTitle":"Mixed Method Evaluation of the CEBHA+ Integrated Knowledge Translation Approach: A Protocol","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2020-12-03 17:18:22","doi":"10.21203/rs.3.rs-117198/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Accept","date":"2020-12-21T00:00:00+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-12-20T00:00:00+00:00","index":1,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-12-20T00:00:00+00:00","index":1,"fulltext":"Recommendation: Major Revision\nForm responses:\n---\n\nComments to Author:\n---\nThis is a great initiative with an overarchingly appropriate evaluation methodology and thoughtful program theory. It's great to see such work happening in African countries, and at such a scale. \n\nMy major concern is that there is some confusion about conducting a theory-driven evaluation that takes account of complexity:\n1. You quite rightly note that IKT strategies implemented at each site should be sensitive to local conditions and will therefore look different at each site, but on page 6 (lines 99-100) you say you will assess the \"dose\" and \"fidelity\" of CEBHA+ IKT as if it were a standardised intervention program. This concern was also triggered by your intention to evaluate \"a systematic approach to engaging with stakeholders\" (lines 111, 424). That suggests you will be looking for adherence to fixed, pre-planned method across all sites, so I was confused about the extent to which you will be responsive to different needs, preferences and infrastructures across the different sites (yet, as you point out, this initiative will not succeed without being responsive to context). I know it's very difficult to navigate the different, competing paradigms that shape norms in evaluation, but I suggest you embrace the complexity issue and make a good case for it without feeling you need to placate the positivists who crave standardisation. Please read more of Penny Hawe's work on what fidelity means for complex interventions in complex systems - she nails it!\n2. You seem to suggest you will be using a realist approach when you quote the realist mantra on page 6 (lines 103-105): \"what works, for whom, and in which circumstances\". You should probably reconsider what claims you wish to make about producing an \"in-depth understanding of what works, for whom, and in which circumstances\" and either rephrase that claim or demonstrate how realist evaluation techniques will be used in your work. You also mention the \"the mechanisms that make IKT successful\" on line 425. This would fit with a realist approach so perhaps you do intend to go down that track? Realist evaluation would be highly suitable given its it a theory-driven evaluation approach that works with complexity. I'm not suggesting you SHOULD do this, just asking that you don't suggest you will without backing it up.\n\nSome minor issues:\n* The first paragraph is awkward with some sentences that seem out of place. One or more of the sentence should possibly sit in the following paragraph? Please review this for coherence.\n* Line 43 - missing the word \"is\"?\n* Lines 50-51 - you say \"While IKT shows promise in bridging the gap between research and practice, there may also be challenges encountered along the way\". As you know, there WILL be challenges! Perhaps rephrase this so the understatement is less pronounced.\n* Lines 51-52, you say challenges from the perspective of decision-makers include presumptions about academic knowledge, skills and capabilities - can you clarify this please?\n* Line 67 \"as well as well-conducted evaluations\" is awkward, perhaps rephrase \"as well as high quality evaluations\"\n* You refer to \"quantitative surveys\" and \"qualitative interviews\". My guess is that the surveys will have some open ended questions so they will be analysed using mixed-methods, and aren't interviews always qualitative? So just saying surveys and interviews (or semi-structured interviews?) is preferable.\n* Referencing errors meant I was unable to follow some of your points on lines 140-141\n* Given the legendarily difficult task of identifying distal impacts in KT, I think your process evaluation is likely to produce better empirical evidence for impact than the outcome evaluation. You may want to consider the literature on a contributions approach.* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **Yes**\n* Level of interest: **An article of importance in its field**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"reviewersInvited","content":"","date":"2020-12-18T00:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"","date":"2020-11-24T00:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2020-11-24T00:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2020-11-23T23:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-11-23T23:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"health-research-policy-and-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"hrps","sideBox":"Learn more about [Health Research Policy and Systems](http://health-policy-systems.biomedcentral.com/)","snPcode":"12961","submissionUrl":"https://submission.nature.com/new-submission/12961/3","title":"Health Research Policy and Systems","twitterHandle":"@HarpsJournal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"acb1ad53-a649-4a74-80e7-c69540b8192e","owner":[],"postedDate":"December 3rd, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":1312472,"name":"Health Economics \u0026 Outcomes Research"},{"id":1312473,"name":"Health Policy"}],"tags":[],"updatedAt":"2021-01-26T15:04:10+00:00","versionOfRecord":{"articleIdentity":"rs-117198","link":"https://doi.org/10.1186/s12961-020-00675-w","journal":{"identity":"health-research-policy-and-systems","isVorOnly":false,"title":"Health Research Policy and Systems"},"publishedOn":"2021-01-18 15:02:16","publishedOnDateReadable":"January 18th, 2021"},"versionCreatedAt":"2020-12-03 17:18:22","video":"","vorDoi":"10.1186/s12961-020-00675-w","vorDoiUrl":"https://doi.org/10.1186/s12961-020-00675-w","workflowStages":[]},"version":"v1","identity":"rs-117198","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-117198","identity":"rs-117198","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}