Use of Routine Health Information Systems Data in Developing and Monitoring District and Facility Health Plans: A scoping review

Use of Routine Health Information Systems Data in Developing and Monitoring District and Facility Health Plans: A scoping review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Use of Routine Health Information Systems Data in Developing and Monitoring District and Facility Health Plans: A scoping review Elaine Byrne, Arthur Heywood This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2565795/v2 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 02 Oct, 2023 Read the published version in BMC Health Services Research → Version 2 posted 9 You are reading this latest preprint version Show more versions Abstract Background: Routine Health Information Systems data should be used in a systematic and institutionalised manner to support the making of plans, the monitoring of plans and in supportive supervision. To explore to what extent there is discussion about the linkage between planning, monitoring and supervision of sub-national programs using routine data we conducted a scoping review. The review question was: How are routine health information systems used or should be used in developing and monitoring health plans at district and facility level? Methods: From a search of Ovid Medline (all), EMBASE and Web of Science along with a review of grey literature and involving a number of key stakeholders in reviewing and identifying any missing resources a total of over 2200 documents were reviewed and data from 13 documents were extracted. Results: Overall, there are many descriptions of how to implement and strengthen systems, ways to assess and improve data availability and quality, tools to improve the data use context, training in data use and mechanisms to involve stakeholders and strengthen infrastructure. However, there are massive gaps in relation to good use cases or examples of where routine health data is used in the development, monitoring and supervision of plans at district and facility level. Conclusions: There appears to be no institutionalised obligation of planners to monitor plans, very little guidance on how to practically monitor programs and minimal discussion about how to use the routinely available data to supportively supervise the implementation of the plans. To overcome these shortcomings, we recommend that practical procedures to ensure linkage of existing district plans to regular monitoring of priority programs are institutionalised, that mechanisms for making managers institutionally accountable for monitoring and supervising these plans are put in place, and that practical guidelines for linking plans with RHIS data and regular monitoring and supportive supervision are developed. Scoping review Routine health information systems planning monitoring supervision Figures Figure 1 Figure 2 Figure 3 Figure 4 Background In planning, managing, governing, decision making and delivering health services, programs and interventions, data informs decisions at all levels of the health system ( 1 – 3 ). Routine Health Information System (RHIS) data should be used in a systematic and institutionalised manner to support the making of plans, the monitoring of plans and in supportive supervision. There are many tools and assessments to monitor and evaluate HIS strengthening interventions which rely on assessing data quality and data use. Many definitions and methods for assessing data quality exist (i.e., accuracy, reliability, precision, completeness, timeliness, integrity, and confidentiality ( 4 ). However, there is less consensus and fewer monitoring tools available for routine data use and very little discussion about the linkage between planning, monitoring and supervision of subnational programs using routine data. In a recent scoping review ( 5 ) there were many examples of the use of DHIS2 (a RHIS platform used in over 70 Lower and Middle Income Countries (LMICs)) data in terms of program review and planning. The most common areas were in terms of use of data in developing periodic plans, for the monitoring and comparison of performance, review meetings, and use of reports. However in terms of planning, very little detail was given on how the DHIS2 informed the plans - in most documents there were simple statements made about DHIS2 data informing plans and in one case excerpts from the plan were presented ( 6 ). However, no further detail on how action plans were previously used or not used or how they planned to be implemented were included. It is not surprising then that a review of data use work practices in Chinsali, Zambia ( 7 ) found that data from DHIS2 (the routine health information system in Zambia) is not being used to develop, monitor or evaluate the district and sub-district health performance plans. The annual plans for Reaching Every District (RED) were found to be comprehensive and reflect priorities of the health sector, but data is collected in a variety of formats and once developed, were not used, no monitoring took place and supervision did not use data. Additionally, in Chinsali, the key indicators in the RED plans were not matched to routine data collected at the facility level. This is not an exception. Boerma et al ( 8 ) note that despite a number of international and national monitoring and evaluation frameworks and guidelines, routine health data is not being used to monitor or evaluate performance plans. They also note that many LMICs face challenges in producing data that would be of sufficient quality to permit the regular tracking of progress in health interventions/services and strengthening health systems ( 8 ). However, we have found no systematic review of the literature on whether and how routine health data are used in the development, monitoring and supervision of facility or subnational health plans. This scoping review will address this gap. Defining data use As Byrne & Sæbø ( 5 ) note data use is not easy to define, as both ‘data’ and ‘use’ can be conceptualised in many different ways. Jones ( 9 ) suggests distinguishing between “data in principle” (as they are recorded) and the “data in practice” (as they are used). There are also different conceptualisations of ‘use’ and consequently many different definitions of data use. In the DHIS2 information cycle ( https://docs.dhis2.org/en/use/what-is-dhis2.html ) data use is understood as the central component of a cyclical process that starts with decisions, goes through data collection, visualisation, dissemination, discussion and interpretation and then back to decisions and actions. Similarly, Nutley interprets data use in decision-making “ ... as the analysis, synthesis, interpretation, and review of data for data-informed decision-making processes, regardless of the source of data” and therefore use “... goes beyond data reporting and passive dissemination of reports.” (10, p2). Nutley ( 10 ) goes on to categorise data use in terms of data and information regularly demanded, analysed, synthesised, reviewed and used in: (i) program review and planning, (ii) advocacy and policy development, and (iii) decision-making processes. Nutley doesn’t define each of these categories but classifies all three as the long-term outcomes of the use of data. We are particularly concerned in this review with data being used in program review and planning - informing the development of plans, in monitoring and supervision and in evaluations or review of the plans. Given the divergence in defining data use, it is not surprising that definitions and methods for monitoring and measuring data use have posed challenges. Nutley and Li ( 4 ) note, data sharing, visualisation, dissemination, and review are often considered cases of data use. As a result, there are many different dimensions of data use that get measured, for example transparency, timeliness, visibility, accessibility, dissemination of information, calculation of key indicators, preparation of information products, and presentation of the achievement of targets (Nutley and Li, 2018). In this review we focus on the use of routine data and information specifically for the making of plans and their monitoring and supervision at sub-national level in the health system. This emphasises continuity of data use in quantifying the key performance indicators in the initial plan, collecting quality data at a local level and using the same indicators for periodic monitoring by program managers and their supervisors doing performance assessment. This does not guarantee action, but ensures that there is high quality data available, visible and shared throughout the planning and implementation cycle. Methods & analysis The Joanna Briggs Institute Guidelines approach of Peters et al. ( 11 ) was followed in this review and included the following steps: defining and aligning the objective/s and question/s; developing and aligning the inclusion criteria with the objective/s and question/s; describing the planned approach to evidence searching, selecting, charting and summarising the evidence. The review question for this literature review was: How are routine health information systems used or should be used in developing and monitoring health plans at district and facility level? Databases searched included: Ovid Medline (all), EMBASE and Web of Science along with a review of grey literature such as documents from WHO, MEASURE, UNICEF, DHIS2 resources, as well as involving a number of key stakeholders in reviewing and identifying any missing resources. Inclusion criteria were that the documents describe how RHISs are used or should be used in developing and monitoring their annual health plans. Exclusion criteria were: Articles that mention challenges/ concerns with the development or monitoring of health plans only. Articles that do not use routine data to develop or monitor health plans, but conduct a particular survey or research to monitor plans. Documents that are theoretical/conceptual only with no examples of use in practice. Non-English language studies. All types of studies were included, as well as original research and reviews. No quality review was conducted as within a scoping review the complete landscape of publications is to be included regardless of quality. However, the article needed to describe the methodology and findings in sufficient detail to be informative in some way, such as in terms of process, content, or findings. Search terms included: ‘routine health information system(s)’ AND ‘plan’ OR ‘micro-plans’ AND ‘monitoring’ (see annex 1 for search strings). As the RED strategy is particularly relevant to the review question the same databases were also searched independently for any article that included ‘Reaching Every District’, ‘RED’ or ‘Reaching Every Child’. All results were imported into Covidence and duplicates removed. This search was updated periodically after the project start date and included the articles or documents retrieved through snowballing or from stakeholders. Included documents were validated by consulting with expert stakeholders to check for any missing relevant documentation. Identified sources of evidence underwent a two-level review process: a title and abstract review, and a full-text review. Data was charted against a number of criteria: Title of article/publication, Lead author, Year, Journal/publication outlet and Country of study. Further details included: practical examples of M&E of action plan, how M&E was implemented, at what level M&E took place, whether RHIS was highlighted as main source of data, whether a framework/model of M&E was presented, description of framework/mode, how M&E is related to a plan, what plan and what the main purpose of M&E. A total of 2442 articles were imported, 153 duplicates removed and 2289 articles’ title and abstract reviewed. Forty-five full texts were reviewed and 32 excluded with the main reasons being the lack of focus of planning. Data from thirteen articles was extracted (Fig. 1 ). Both authors screened, reviewed and extracted at all stages. Both authors have decades of experience in RHIS in developing countries at district level and below, as well as in researching and writing about data and information use. Conflicts were resolved between the two authors, though the department research group was available to arbitrate if it had been needed. The list of articles included and preliminary findings were shared with stakeholders from global health institutions such as UNICEF, WHO and RHINO, the health information systems research group at University of Oslo, and the Global Health Information Systems Programme network. Description of documents included A total of 13 documents were included at full extraction stage (Table 1). Most of the documents were published between 2013 and 2019, with 3 published in 2013 and 2016 (Fig. 2). The documents were published in health service, health policy, systems and planning journals (9) with others in information systems (1), East Mediterranean Health (1), rural health (1) and public health (1) (Fig. 3). Most of the articles were based on studies in Africa and Asia, with one in South America, Middle East and one in general on LMICs (Fig. 4). Content of documents included: The documents are described in two subsections based on the two overarching categories that emerged from the included articles: results from country assessments and reviews of planning processes at sub-national level. In terms of the exclusion criteria we excluded theoretical/conceptual articles/reports that had no practical base. Examples of those excluded are: a logic model for improving the use of health data for health system strengthening with recommendations that affect the use of data in decision making ( 12 ) a correspondence article on planning, implementation, monitoring, and evaluation of integrated health services which emphasises the importance of strong M&E systems ( 13 ) a country-led platform for information and accountability that provides guidance to countries and partners for strengthening monitoring, evaluation and review (M&E) of national health plans and strategies (NHS), but no specific plan is being monitored ( 14 ) Use of RHIS to evaluate HSS interventions, with a good description of indicator selection, but not linking use of RHIS to evaluate plans ( 15 ) Organizing Framework for a Functional National HIV Monitoring and Evaluation System but does not provide detailed guidance on how to operationalize the system ( 16 ) However, in the discussion section of this article we look at how some of these conceptual models could assist in informing guidance on local level planning, monitoring and supervision of action plans. 6.3. Evaluations/Assessments of M&E systems and processes In a study in South Africa on perceptions about data-informed decisions for HIV, Nicol et al ( 17 ) explore the challenges in relation to data use and note that organisational and capacity issues need to be addressed before information will be used. This includes the development of a culture of information use, trust in the data, capacity to analyse, interpret and use information. They suggest that "Facility and program managers should be provided with opportunities for capacity development as well as practice-based, in-service training, and be supported to use information for planning, management and decision-making" (17, p.765). However, there are no practical guidelines or suggestions on how this is to be achieved. They observe that there are mechanisms and processes in place to promote use of information (performance meetings, access to routine reports, directives/ SOPs, monthly targets, existence of information), but these are not being used (or are 'selectively used'). In a similar fashion Nabyonga-Orem, J. et al ( 18 ) look at the harmonisation and standardisation of the health sector and programme reviews in the WHO African Region. They highlight the main challenges in terms of performance assessment (weak institutional capacity for M&E, desynchronised planning timeframes, inadequate time allocated for comprehensive performance assessments, weak follow-up mechanisms, lack of stakeholder engagement and divergent political agendas). They call for the standardisation and institutionalisation of performance assessments, but give no detail on how this can be done and do not link it with planning and a comprehensive M&E framework. In Malawi, Chaulagai, C.N., et al ( 19 ) reviewed the District Implementation Plan (DIP) process and described the stages of getting baseline data to set priorities and targets in the DIP. They conclude that most DIPs are vague and that it is hard to track implementation status and results. There were some attempts to computerise the DIP process so that there are links with the routine data collected for monitoring the plan, but this resulted in only the person entering the data being involved in the planning and monitoring process. The purpose of the DIP process was to enable the allocation of resources based on performance, so that facilities and districts could compare or rank their performance in relation to other facilities of districts. The DIPs were also accompanied by a system of quarterly feedback, supportive supervision visits and annual reviews. However, there is no detailed description of a M&E plan or process on how the DIP should be monitored or supervised. However, some good practices were identified – gaining consensus on indicators and tools, skills training on utilising existing data to calculate indicators and management of health services, establishing regular meetings and reporting (quarterly management and annual performance reviews at all levels) and the development of routine monitoring and guidelines for an annual health sector joint review. They conclude that there was overall little improvement in the use of data in rationalising decisions and that "no matter how good the design of an information system, it will not be effective unless there is internal desire, dedication and commitment of leadership to have an effective and efficient health service management system” (19, p.375). So overall the published evaluations/assessments of M&E systems and processes do not reveal a practice of using RHIS routine data in the planning, monitoring and supervision process. However, some of the key findings could inform the implementation of such a system, namely, the need to develop organisational and institutional capacity ( 17 ), the standardisation and institutionalising of performance assessment ( 18 ) and the commitment of leadership to effective and efficient service management ( 19 ). Though not showing that RHIS is being used some of the other articles reviewed indicate that RHIS data could be used in monitoring programmes, even if not all RHIS data is not of the same quality, by Increasing use, data quality monitoring, and management and information technology of RHIS ( 21 ). Cibulskis, R.E. et al ( 22 ) argue that even imperfect routine health data can be used whilst simultaneously working on the quality and completeness of it to encourage ‘a more methodical approach to planning and monitoring services’ and with improved quality of data be used on a routine basis to monitor performance. In the remaining articles there is mention of data being used in planning and monitoring, but findings are not particularly informative in terms of practical examples or lessons learned. For example, a theoretically informed review of the Kenyan HIS and accountability, includes some mention of use of data in monitoring plans, but it is rather vague ( 30 ); a M&E system at national level in Brazil highlights the importance of integrating all the M&E tools and plans across departments and units, but not focusing on how this was done or how the integrated plans were monitored ( 23 ), and; the development of a framework for the M&E of a National plan in Iran indicates that numerous surveys and routine information systems would be needed to monitor the plan and even using these sources there still would be gaps in data needed to monitor the plan ( 24 ). Similarly, a review of 5 health system strengthening changes made note that a common evaluation framework of HIS strengthening was used, but it is not described ( 25 ). The authors note that there is hope that this will assist in “… linking HIS with decision making, and its impact on measures of health system outputs and impact” (25, p.1), but no examples are given of these decisions or processes involved in linking HIS, decision making and impact. Exploring other changes made to HIS for improving use in decision-making Nutley, T. et al ( 26 ) look at the impact of the District Health Plan (DHP) decision-support tool on decision making at the district level in Kenya, but there is little detail given on the tool and there is no link to a plan or monitoring of the plan. In fact, the 11 review questions of the HIV programme included in the tool works in parallel rather than as part of an integrated M&E system. Evaluations of plans One example of monitoring facility level plans was in the study by Enkhtuya et al ( 27 ) when reviewing the national RED planning in one district. The district and family practitioners were able to map areas of low coverage, undertake barrier analysis, and provide detailed costed activities required to reach these populations. The RED M&E plan includes supportive supervision (such as local problem solving, involvement of community), micro-planning activities and budget requests through normal annual operational planning and budgeting processes. Overall though the reviewers concluded that there is a need to change the RED approach to management as local area knowledge is currently absent in plans and supportive supervision. Changing the style and approach to planning requires a change in the style of and approach to management - moving from supervisor as inspector to supporter. Though the RED strategy is intended as an immunisation-specific intervention, the process in Mongolia indicates its potential for wider health system applications. In fact Chan Soeung, S. et al ( 28 ) draw a similar conclusion on community level involvement in terms of planning, recognising that “a shift in planning focus is needed from a district-wide perspective down to a facility- and community-level system of analysis and operations”(28, p.532). However, the Chan Soeung, S. et al ( 28 ) study was on equity and not planning and so provides little insight into using RHIS for planning, monitoring and supervision. The only other article that looked specifically at decision-making and planning was a literature review of decision making for health in LMICs by Wickremasinghe et al ( 29 ). They found twelve examples of tools to assist district-level decision-making. The use in decision-making comprised generally the use of data to identify priorities, and in developing an action plan to address those priorities. Four of the studies included steps for reviewing or monitoring the plans with HIS data, but “… there was limited evidence about their sustained impact on district level decision-making and whether they have led to changes in resource allocation patterns” (29, pii23). However, in terms of lessons around data use that could be used in terms of planning, monitoring and supervision, there were three features that kept recurring: relevance depended on data quality, for consensus a structured decision-making process is needed and that communities need to be included in the decision making process. So overall the articles indicate the need for local level planning that involves communities and the review article additionally indicates the need for data quality and a structured process for decision making. Discussion This study looked at the link between planning and monitoring at subnational level using routine information and found that there were surprisingly few articles that looked at the concept and even fewer that described the practicalities of implementing the links. While there are a number of high level descriptions of what should be done to develop a culture of information use, most were very vague and there were very few details of how this process can be institutionalised, how staff are held accountable for implementing their plans or how to actually monitor the plans made. What we found missing was how to institutionalise regular, structured data use for monitoring program performance at service delivery level and ensure its discussion and interpretation by lower level stakeholders. However, there are some existing frameworks on the use of routine data in monitoring as well as global strategies that support the development and review of lower level/micro plans. We didn’t find any articles using these in the literature we reviewed but they could be used in conjunction with the findings from our review to inform guidelines for planning, monitoring and supervision at local level. Nutley and Lee ( 4 ) reviewed the main assessments and tools used for data use. They include a review of data use in the health information system strengthening model (HISSM) and the Measure Evaluation Logic Model for improving data use. The Measure Evaluation Logic Model describes specific activities (8 domains) and interventions needed to improve the use of health data for improved health programs and policies. This model builds on the HISSM by providing specific and detailed ways to support the use of HIS data. There is no specific reference to use of data in developing, monitoring and supervising plans made in any of the models, but Nutley and Li ( 4 ) look at various assessment tools and map them according to the dimensions of data use (data quality, health statistics, information products, data review, advocacy, decision and action). None of these tools are specific to data use and action planning. Nutley and Li conclude that “Few tools that measure the outcome of data use for improved health program performance exist. ..... Better measures of the outcome of data use are needed, along with ways to easily track the health program and health system outcomes associated with decisions that are implemented." (4, p.32). Our review confirms this conclusion. To consolidate the contributions of the included articles in this review towards this gap we map the content of the articles extracted against the 8 domains of the Measure Evaluation Logic Model (Table 2): Assessing and improving the data use context Engaging data users and data producers Improving data quality Improving data availability Identifying information needs Building capacity in data use core competencies Strengthening the organisation’s infrastructure Monitoring, evaluating, & communicating successes There are a couple of the domains that could be expanded based on the articles we extracted, namely the assessment and improving the data use context and strengthening the organisations infrastructure. In relation to the domain of assessing and improving the data use context, this is often referred to as developing a culture of information use, we are reminded by Reynolds, H. W. et al ( 13 ) and Bernardi ( 30 ), that such a culture requires an institution to take responsibility and accountability. More detail on how to achieve improvements in the context include: stakeholders linking data collection and strategic plans, operational plans and program plans ( 13 ); stakeholders placing value on the data (Nutley et al 2013) and showing commitment ( 19 ); gaining consensus on a structured decision-making process ( 29 ). All of this requires standardisation, institutionalisation and coordination of planning, M&E, assessments and follow up mechanisms ( 18 ). As Mutale et al ( 25 ) note, data use in planning, monitoring and supervision needs to be as institutionalised as stocking a pharmacy or immunising a child. In a commentary on the historical evolution of monitoring and evaluation Thomas, J.C. et al ( 31 ) note that monitoring and evaluation for health in LMICs has advanced from an emerging discipline to one that adheres to standards, is systematised and mainstreamed across programmes. They also note that significant capacity has been developed and resources allocated to collecting and using data, enabling a move away from paper-based to electronic systems. What is interesting are the tensions that exist with these changes: single, unified country system versus accountability and control by particular disease/ programmes and particular donors; desire for a shared integrated system versus the desire for specific outcomes country autonomy versus donor control. If we are to see more examples of RHIS being used in planning, monitoring and evaluation these tensions need to be addressed in creating the context for effective data use. In terms of engaging data users and data producers Wickremasinghe et al, ( 29 ) remind us of the importance of including the community in the decision making process as key data users and producers. Shifting planning emphasis to the health centre and community is also raised ( 27 , 28 ). In relation to data quality, most articles highlighted how important data quality was if it was to be used. Many tools have been developed to improve data quality including minimal data entry and automated graphs ( 26 ). Equally important is that the data is trusted by users ( 17 ) and this is often subjective and goes beyond data quality. Cibulskis, R.E. et al ( 22 ), as do Wagenaar et al ( 21 ), remind us that we should not wait until data quality is ‘perfect’ before using the data as improving the existing system can continue while data is being used and using data and improving data quality is a duality with one feeding in the other. Counter arguing the poor quality of RHIS, Wagenaar et al ( 21 ) note that RHIS data are superior to intermittent community sample surveys which can have data delays far works than RHIS and avoid the high costs of conducting surveys. They note that there are still existing challenges with RHIS such as population data, including the excluded and programmes or policies not provided by the facility or their outreach services. There are many examples given of mechanisms or activities to strengthen the organisational infrastructure. Chaulagai, C.N., et al ( 19 ) give a comprehensive list, and Nicol et al ( 17 ) and others give examples of key issues they found to facilitate use in their reviews. These include, but are not limited to: gaining consensus on indicators and tools, skills training on utilising existing data to calculate indicators and management of health services, establishing regular meetings and reporting, the development of routine monitoring and guidelines for periodic reviews, and standardising and institutionalising the planning and monitoring process through agreed strategies, SoPs and guidelines. However, there is no mention of the need to institutionalise monitoring of annual plans at sub-national level and using this information to supervise managers and hold them accountable for implementation of the plans. The RED approach, with its five operational strategies developed through a facility/ district micro planning tool using routine data, comes the closest to describing the expected links between planning, monitoring and supervision. The strategy has been implemented since 2003 in 53 LMICs, ( 27 ) and adapted to country realities, but multiple studies show that very few districts or facilities had updated micro plans ( 7 , 32 , 33 ). An excellent tool with the support of all major immunisation donors has been squandered because it does not clearly define implementation modalities and as a result has not been followed through at subnational level. Fundamentally what emerges from this review is that the main obstacle to using routine data in monitoring and supervision of annual plans is the weak institutionalisation of the planning and monitoring processes and lack of clarity on the organisational procedures needed for this to occur at sub-national level. This requires a data use vision by national leaders that links planning to monitoring and promotes commitment and leadership at all levels. Conclusion The basic premise of decentralised planning is that district level plans should be regularly monitored by the planners themselves using routine data and their implementation supervised by immediate managers ( 34 , 35 ). Our review has found that this linkage does not exist in the published literature. There appears to be no institutionalised obligation of planners to monitor (micro) plans, very little guidance on how to practically monitor programs and minimal discussion about how to use the routinely available data to supportively supervise the implementation of the plans. Overall, there are many descriptions of how to implement and strengthen systems, ways to assess and improve data availability and quality, tools to improve the data use context, training in data use and mechanisms to involve stakeholders and strengthen infrastructure. However, there are massive gaps in the literature in relation to good use cases or examples of where routine health data is used in the development, monitoring and supervision of plans at district and facility level. Likewise, there are gaps in terms of guidelines on how this can happen. The RED approach appears to be the best available framework as it clearly links facility micro-planning, local monitoring of implementation and supervision ( 27 , 28 ) and provides some instruction on how to implement the various steps. However, global RED guidelines lack practical details and it seems they have not been adapted to ensure local implementation, particularly around data collection and use. Consolidating the information on M&E from the reviewed documents and mapping to the Measure Evaluation logic model domains could be a useful starting point in developing such guidelines. It is possible that some relevant documents may be published in journals not indexed on the databases searched or the key experts consulted were not aware of guidelines available. We are therefore not concluding that there are no other examples or guidelines that exist, but that these examples or guidelines are not readily available in the published literature. The existing mechanisms that could be built upon include performance meetings, access to routine reports, directives/ SOPs, monthly targets, existence of information, but these need practical guidelines on how and when to use. The main gaps that need to be addressed are, Practical procedures to ensure linkage of existing district plans to regular institutionalised monitoring of priority programs Mechanisms for making managers institutionally accountable for monitoring and supervising these plans, and Developing practical guidelines for linking plans with RHIS data and linking plans with regular monitoring and supportive supervision. Abbreviations DHIS2 District Health Information Software 2 DHP District Health Plan DIP District Implementation Plan HIS health Information System HISSM Health Information System Strengthening LMICs Lower and Middle Income Countries M&E Monitoring and Evaluation PRISM Performance of Routine Information System Management RED Reaching Every District RHIS Routine Health Information Systems UNICEF United Nations Children’s Fund WHO World Health Organisation Declarations Ethics approval and consent to participate Not applicable as a desk review of material in the public domain Consent for publication Not applicable as no personal or individual data collected or reported on Availability of data and materials All data generated or analysed during this study are included in this published article. Competing interests No competing interests. Funding Funding from GAVI Global (A5) support to HISP Centre UiO 2021-2023 supported AH time in conducting this review. Authors' contributions Both authors equally contributed to the conception of the work. The search terms and inclusion and exclusion criteria were agreed upon by both authors. A librarian, Paul Murphy, at the RCSI conducted the search in consultation with EB. EB imported the combined documents returned from the databases into covidence. Both authors equally contributed to the data analysis and interpretation, drafting, revising and final approval of the article. Acknowledgements We appreciate the funding from GAVI for this work which is part of their continued support to the Global HISP network. Thanks to Paul Murphy Information Specialist at RCSI Library for his expertise, advice and support in conducting the search. We are also grateful for all the experts who gave of their time to review the different versions of our draft manuscript and forwarded documents/links of potential sources of further information for our review. We appreciate the feedback from the Health Information Systems Programme Designing for Data Use Lab group at the HISP Centre, University of Oslo . Authors' information Elaine Byrne (PhD) worked with HISP South Africa (1997-2008) when living and working in South Africa and whilst doing her PhD. In Oct 2021 she joined the Department of Informatics at the University of Oslo on a leave of absence from the University of Medicine and Health Sciences, Royal College of Surgeons in Ireland (RCSI) to work on DHIS2 data use practice. Her general research interests are around research that supports practice and focuses on healthy people in healthy societies. Arthur Heywood (AH) is a public health veteran activist., formerly a university lecturer and professor in South Africa, and currently consultant with Global Health Institutions, such as GAVI, UNICEF, WHO and HISP largely on the implementation and use of RHIS. Arthur set up a Masters programs in Public Health and Information Systems in South Africa, Mozambique, Malawi and Tanzania, all of them with a focus on using information to improve PHC service provision and to promote equity. He is also a founding member of the Health Information Systems Program in post-apartheid South Africa. References AbouZahr C, Boerma T. Health information systems: the foundations of public health. . Bull World Health Organ 2005;83(8):578-83. MEASURE Evaluation. Strengthening Health Information Systems in Low- and Middle-Income Countries—A Model to Frame What We Know and What We Need to Learn. 2017. World Health Organization. Everybody’s business: strengthening health systems to improve health out comes: WHO’s framework for action. . Geneva: World Health Organization; 2007. Nutley T, Li M. Conceptualizing and Measuring Data Use: A Review of Assessments and Tools. University of North Carolina: MEASURE Evaluation; 2018. Byrne E, Sæbø JI. Use of DHIS2 Data: A Scoping Review. BMC Health Services Research. 2022;1234( https://doi.org/10.1186/s12913-022-08598-8). Asah FN, Nielsen P, Sæbø JI, editors. Challenges for Health Indicators in Developing Countries: Misconceptions and Lack of Population Data. 14th International Conference on Social Implications of Computers in Developing Countries IFIP 94 WG; ICTs for promoting social harmony: Towards a sustainable information society; 2017; Indonesia: Cham: Springer International Publishing. Heywood A. Data Use in Chinsali, Zambia. 2021. Boerma T, AbouZahr C, Bos E, Hansen P, Addai E, Low-Beer D. Monitoring and Evaluation of health systems strengthening: an operational framework. WHO, World Bank, GAVI and Global Fund; 2009. Jones M. What we talk about when we talk about (big) data. The Journal of Strategic Information Systems. 2019;28(1):3-16. Nutley T. Improving Data Use in Decision Making: An Intervention to Strengthen Health Systems. MEASURE Evaluation; 2012. Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13(3):141-6. Nutley T, Reynolds HW. Improving the use of health data for health system strengthening. Global Health Action. 2013;6(1):20001. Reynolds HW, Sutherland EG. A systematic approach to the planning, implementation, monitoring, and evaluation of integrated health services. BMC Health Serv Res. 2013;13:168. World Health Organization, Internationl Health Partnerships. Monitoring, evaluation and review of national health strategies: a country-led platform for information and accountability.; 2011. Law MR, Hotchkiss DR, Grépin KA. Evaluating Health Systems. Strengthening Interventions Using Routinely Collected Data. 2022. UNAIDS MERG. Organizing Framework for a Functional National HIV Monitoring and Evaluation System. 2008. Nicol E, Bradshaw D, Uwimana-Nicol J, Dudley L. Perceptions about data-informed decisions: an assessment of information-use in high HIV-prevalence settings in South Africa. BMC Health Services Research. 2017;17(2):765. Nabyonga-Orem J, Tumusiime P, Nyoni J, Kwamie A. Harmonisation and standardisation of health sector and programme reviews and evaluations – how can they better inform health policy dialogue? Health Res Policy Sys 2016;14(87). Chaulagai CN, Moyo CM, Koot J, Moyo HB, Sambakunsi TC, Khunga FM, et al. Design and implementation of a health management information system in Malawi: issues, innovations and results. Health Policy and Planning. 2005;20(6):375-84. Chaulagai CN, Moyo CM, Koot J, Moyo HBM, Sambakunsi TC, Khunga FM, et al. Design and implementation of a health management information system in Malawi: issues, innovations and results. Health Policy and Planning. 2005;20(6):375-84. Wagenaar BH, Sherr K, Fernandes Q, Wagenaar AC. Using routine health information systems for well-designed health evaluations in low- and middle-income countries. HEALTH POLICY AND PLANNING. 2016;31(1):129-35. Cibulskis RE, Hiawalyer G. Information systems for health sector monitoring in Papua New Guinea. Bull World Health Organ. 2002;80(9):752-8. Sellera PEG, Brito CBM, Jovanovic MB, Rodrigues SO, Oliveira C, Santos SOD, et al. The Implementation of the Monitoring and Evaluation System of the State Health Secretariat of the Brazilian Federal District (SHS/DF). Cien Saude Colet. 2019;24(6):2085-94. Abdi Z, Majdzadeh R, Ahmadnezhad E. Developing a framework for the monitoring and evaluation of the Health Transformation Plan in the Islamic Republic of Iran: lessons learned. East Mediterr Health J. 2019;25(6):394-405. Mutale W, Chintu N, Amoroso C, Awoonor-Williams K, Phillips J, Baynes C, et al. Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative. BMC Health Serv Res. 2013;13 Suppl 2(Suppl 2):S9. Nutley T, McNabb S, Salentine S. Impact of a decision-support tool on decision making at the district level in Kenya. Health Res Policy Syst. 2013;11:34. Enkhtuya B, Badamusuren T, Dondog N, Khandsuren L, Elbegtuya N, Jargal G, et al. Reaching every district - development and testing of a health micro-planning strategy for reaching difficult to reach populations in Mongolia. Rural Remote Health. 2009;9(2):1045. Chan Soeung S, Grundy J, Duncan R, Thor R, Bilous JB. From reaching every district to reaching every community: analysis and response to the challenge of equity in immunization in Cambodia. . Health Policy Plan. 2013;28(5):526-35. Wickremasinghe D, Hashmi IE, Schellenberg J, Avan BI. District decision-making for health in low-income settings: a systematic literature review. Health Policy Plan. 2016;31 Suppl 2(Suppl 2):ii12-ii24. Bernardi R. Health Information Systems and Accountability in Kenya: A Structuration Theory Perspective. Journal of the Association for Information Systems. 2017;18(12). Thomas JC, Doherty K, Watson-Grant S, Kumar M. Advances in monitoring and evaluation in low- and middle-income countries. Evaluation and Program Planning. 2021;89:101994. Mafigiri DK, Iradukunda C, Atumanya C, Odie M, Mancuso A, Tran N, et al. A qualitative study of the development and utilization of health facility-based immunization microplans in Uganda. Health Res Policy Syst. 2021;19(Suppl 2):52. Ngomba AV, Kollo B, Bita AF, Djouma FN, Edengue JM, Elongue MJ, et al. [Immunization programme in urban areas in Cameroon: a case study of the Djoungolo Health District]. Pan Afr Med J. 2016;25:213. Lankester T, Nathan G. Monitoring and evaluating the health programme. In: Lankester T, Nathan G, editors. Setting up Community Health and Development Programmes in Low and Middle Income Settings. 4th ed: Oxford, 2019; online edn, Oxford Academic, 1 Mar. 2019; 2019. Green A. An Introduction to Health Planning in Developing Countries. 2nd ed: Oxford Medical Publications, Oxford University Press; 1999. Tables Table 1 Lead author Title Publication outlet Abdi, Z. et al (2019) Developing a framework for the monitoring and evaluation of the Health Transformation Plan in the Islamic Republic of Iran: lessons learned East Mediterr Health J Bernardi, R. (2017) Health Information Systems and Accountability in Kenya: A Structuration Theory Perspective Journal of the Association for Information Systems Chan Soeung, S. et al. (2013) From reaching every district to reaching every community: analysis and response to the challenge of equity in immunization in Cambodia Health Policy and Planning Chaulagai, C.N., et al (2005) Design and implementation of a health management information system in Malawi: issues, innovations and results Health Policy and Planning Cibulskis, R.E. et al (2002) Information systems for health sector monitoring in Papua New Guinea Policy and Practice Enkhtuya, B. et al (2009) Reaching every district - development and testing of a health micro-planning strategy for reaching difficult to reach populations in Mongolia Rural and Remote Health Mutale, W. et al (2013) Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative BMC Health Services Research Nabyonga-Orem, J. et al (2016) Harmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue? Health Research Policy & Systems Nicol, E. et al (2017) Perceptions about data-informed decisions: an assessment of information-use in high HIV-prevalence settings in South Africa BMC Health Services Research Nutley, T. et al (2013) Impact of a decision-support tool on decision making at the district level in Kenya Health Research Policy & Systems Sellera, P.E.G. et al (2019) The Implementation of the Monitoring and Evaluation System of the State Health Secretariat of the Brazilian Federal District (SHS/DF) Ciência & Saúde Coletiva (Science & Public Health) Wagenaar, B.H. et al (2016) Using routine health information systems for well-designed health evaluations in low- and middle-income countries Health Policy and Planning Wickremasinghe, D. , et al (2016) District decision-making for health in low-income settings: a systematic literature review Health Policy and Planning Table 2 Measure Evaluation logic model domains Content of included articles related to domain Assessing and improving the data use context assess and improve the data use context/data culture (Nutley & Reynolds, 2013) organisational and capacity issues such as the development of a culture of information use (Nicol et al., 2017) requires an institution taking responsibility and accountability (Bernardi, 2017; Reynolds & Sutherland, 2013) stakeholders need to make the link between data collection and strategic plans, operational plans and program plans (Reynolds & Sutherland, 2013) stakeholders and decision-makers need to place value on data they use in decision making (Nutley & Reynolds, 2013) There needs to be ‘the internal desire, dedication and commitment of leadership ’ (Chaulagai et al., 2005) for consensus a structured decision-making process is needed (Wickremasinghe et al., 2016) standardisation, Institutionalisation and coordination of planning, M&E, assessments and follow up mechanisms (Nabyonga-Orem et al., 2016) Monitoring programs needs to be “as institutionalised as stocking a pharmacy or immunising a child” (Mutale et al., 2013) Problem-solving planning methodology progressing from health mapping to barrier analysis, to activity planning and costing and finally to monitoring and evaluation (Enkhtuya et al., 2009) Engaging data users and data producers engage with other stakeholders - data users and data producers (Nutley & Reynolds, 2013) communities need to be included in the decision making process (Wickremasinghe et al., 2016) Shift of planning emphasis to the health centre and community is needed. (Chan Soeung et al., 2013; Enkhtuya et al., 2009) Improving data quality improve data quality (Law et al., 2022; Nutley & Reynolds, 2013) relevancy depends on quality (Wickremasinghe et al., 2016b. developing trust in data (Nicol et al., 2017) improve existing systems whilst using even imperfect routine health data (Cibulskis & Hiawalyer, 2002) Improving data availability improve data availability (Nutley & Reynolds, 2013) Identifying information needs identify information needs (Nutley & Reynolds, 2013; World Health Organization & Internationl Health Partnerships, 2011) Building capacity in data use core competencies build capacity in data use core competencies (Nutley & Reynolds, 2013) capacity to analyse, interpret and use information (Nicol et al., 2017) Re-orientation of management approaches from ‘inspection’ to supportive supervision (Enkhtuya et al., 2009) Strengthening the organisation’s infrastructure improve organisation’s data demand and use infrastructure (Nutley & Reynolds, 2013) mechanisms to improve practice gaining consensus on indicators and tools, skills training on using data to calculate indicators and management of health services, establishing regular meetings and reporting, the development of routine monitoring and guidelines for an annual health sector review (Chaulagai et al., 2005) Monitoring, evaluating, & communicating successes monitor, evaluate, and communicate results of data use interventions (Nutley & Reynolds, 2013) development of communication strategies and health networks that enable providers to adjust service delivery according to the needs of vulnerable population (Chan Soeung et al., 2013) Additional Declarations No competing interests reported. Supplementary Files Annex1Searchstrings.docx PRISMA2020checklistcompleted.docx Cite Share Download PDF Status: Published Journal Publication published 02 Oct, 2023 Read the published version in BMC Health Services Research → Version 2 posted Editorial decision: Major revision 19 Jun, 2023 Reviews received at journal 07 Jun, 2023 Reviewers agreed at journal 31 May, 2023 Reviewers agreed at journal 27 May, 2023 Reviewers invited by journal 27 May, 2023 Editor assigned by journal 30 Apr, 2023 Editor invited by journal 28 Apr, 2023 Submission checks completed at journal 28 Apr, 2023 First submitted to journal 04 Apr, 2023 You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2565795","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[{"code":1,"date":"2023-03-03 00:17:44","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":197282726,"identity":"ae246216-9cd2-44dd-9561-32d64227f208","order_by":0,"name":"Elaine Byrne","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYDACdgaGA2DEwMD44AOQZGNgbMCvhRmhhdlwBrFaGKBa2KR5iHEXPzPzw8MFFXcSt89uPiBtu8fOno+BufEBPi2SzWwGh2eceZY4586xBOOcZ8mJbQyMzQb4tBgcBiLetsOJMyRyDJJzDhxIAPqlTQKfFvvD7B8O8/4Dacn/cNjiwAF7oJb2H3htYeYB2tIAtoWxmeHAAUagw9rw6WCQOMxTcJjn2DPjGRJpxow9B4B+YWZsxusw/vb2zZ95au7IzpBIfv7jxwE7e/n29ocf8FoDBY4NcCYzMeqBwJ5IdaNgFIyCUTASAQAGikv1UInOEAAAAABJRU5ErkJggg==","orcid":"","institution":"Royal College of Surgeons in Ireland","correspondingAuthor":true,"prefix":"","firstName":"Elaine","middleName":"","lastName":"Byrne","suffix":""},{"id":197282727,"identity":"d2218b62-5260-429f-8d90-476702287d03","order_by":1,"name":"Arthur Heywood","email":"","orcid":"","institution":"HISP Africa","correspondingAuthor":false,"prefix":"","firstName":"Arthur","middleName":"","lastName":"Heywood","suffix":""}],"badges":[],"createdAt":"2023-02-08 17:59:17","currentVersionCode":2,"declarations":"","doi":"10.21203/rs.3.rs-2565795/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-2565795/v2","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12913-023-09914-6","type":"published","date":"2023-10-02T15:02:06+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":36584576,"identity":"fc240eb4-0c5d-4786-8d15-b17ee54b8cb1","added_by":"auto","created_at":"2023-05-03 15:09:02","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":180585,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003ePRISMA indicating 13 articles included\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/13121a1c37e7503bf91edb93.jpeg"},{"id":36584578,"identity":"43fb0eef-f823-482b-8c0f-97d185d80909","added_by":"auto","created_at":"2023-05-03 14:53:02","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":7667,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eArticles included by year of publication\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/63e31c30f5ea86173040d067.png"},{"id":36584580,"identity":"7e872c5a-ea2b-411e-b955-06e65a47c2b3","added_by":"auto","created_at":"2023-05-03 15:01:02","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":16489,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eIncluded articles by publication outlet\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/5b7424e76c49dbd0a7f878b6.png"},{"id":36584582,"identity":"43bbfc8e-5c86-44c1-87f5-ec289508dc7a","added_by":"auto","created_at":"2023-05-03 15:01:02","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":11135,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eGeographic area of study in included articles\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/e81f987074cca9542ed68e74.png"},{"id":44301757,"identity":"63640bcb-660c-4784-ab85-91507ec53309","added_by":"auto","created_at":"2023-10-09 15:08:02","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":476021,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2565795/v2/c0b7f375-ea3d-40e6-bba8-961f731b4587.pdf"},{"id":36584588,"identity":"e5f49765-23f8-4dfe-8695-ef73c9b0e12e","added_by":"auto","created_at":"2023-05-03 14:53:02","extension":"docx","order_by":8,"title":"","display":"","copyAsset":false,"role":"supplement","size":13184,"visible":true,"origin":"","legend":"","description":"","filename":"Annex1Searchstrings.docx","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/93515adbe5fd23b2ba8334cc.docx"},{"id":36584590,"identity":"ea20fc34-8291-445a-8684-cf511214d9bb","added_by":"auto","created_at":"2023-05-03 15:09:02","extension":"docx","order_by":9,"title":"","display":"","copyAsset":false,"role":"supplement","size":27561,"visible":true,"origin":"","legend":"","description":"","filename":"PRISMA2020checklistcompleted.docx","url":"https://assets-eu.researchsquare.com/files/rs-2775860/v1/7da36b6e4f71b69be578d856.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Use of Routine Health Information Systems Data in Developing and Monitoring District and Facility Health Plans: A scoping review","fulltext":[{"header":"Background","content":"\u003cp\u003eIn planning, managing, governing, decision making and delivering health services, programs and interventions, data informs decisions at all levels of the health system (\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). Routine Health Information System (RHIS) data should be used in a systematic and institutionalised manner to support the making of plans, the monitoring of plans and in supportive supervision. There are many tools and assessments to monitor and evaluate HIS strengthening interventions which rely on assessing data quality and data use. Many definitions and methods for assessing data quality exist (i.e., accuracy, reliability, precision, completeness, timeliness, integrity, and confidentiality (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e). However, there is less consensus and fewer monitoring tools available for routine data use and very little discussion about the linkage between planning, monitoring and supervision of subnational programs using routine data.\u003c/p\u003e \u003cp\u003eIn a recent scoping review (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) there were many examples of the use of DHIS2 (a RHIS platform used in over 70 Lower and Middle Income Countries (LMICs)) data in terms of program review and planning. The most common areas were in terms of use of data in developing periodic plans, for the monitoring and comparison of performance, review meetings, and use of reports. However in terms of planning, very little detail was given on how the DHIS2 informed the plans - in most documents there were simple statements made about DHIS2 data informing plans and in one case excerpts from the plan were presented (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e). However, no further detail on how action plans were previously used or not used or how they planned to be implemented were included.\u003c/p\u003e \u003cp\u003eIt is not surprising then that a review of data use work practices in Chinsali, Zambia (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e) found that data from DHIS2 (the routine health information system in Zambia) is not being used to develop, monitor or evaluate the district and sub-district health performance plans. The annual plans for Reaching Every District (RED) were found to be comprehensive and reflect priorities of the health sector, but data is collected in a variety of formats and once developed, were not used, no monitoring took place and supervision did not use data. Additionally, in Chinsali, the key indicators in the RED plans were not matched to routine data collected at the facility level. This is not an exception. Boerma et al (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e) note that despite a number of international and national monitoring and evaluation frameworks and guidelines, routine health data is not being used to monitor or evaluate performance plans. They also note that many LMICs face challenges in producing data that would be of sufficient quality to permit the regular tracking of progress in health interventions/services and strengthening health systems (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e). However, we have found no systematic review of the literature on whether and how routine health data are used in the development, monitoring and supervision of facility or subnational health plans. This scoping review will address this gap.\u003c/p\u003e"},{"header":"Defining data use","content":"\u003cp\u003eAs Byrne \u0026amp; S\u0026aelig;b\u0026oslash; (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) note data use is not easy to define, as both \u0026lsquo;data\u0026rsquo; and \u0026lsquo;use\u0026rsquo; can be conceptualised in many different ways. Jones (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e) suggests distinguishing between \u0026ldquo;data in principle\u0026rdquo; (as they are recorded) and the \u0026ldquo;data in practice\u0026rdquo; (as they are used). There are also different conceptualisations of \u0026lsquo;use\u0026rsquo; and consequently many different definitions of data use. In the DHIS2 information cycle (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://docs.dhis2.org/en/use/what-is-dhis2.html\u003c/span\u003e\u003cspan address=\"https://docs.dhis2.org/en/use/what-is-dhis2.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) data use is understood as the central component of a cyclical process that starts with decisions, goes through data collection, visualisation, dissemination, discussion and interpretation and then back to decisions and actions. Similarly, Nutley interprets data use in decision-making \u0026ldquo; ... as the analysis, synthesis, interpretation, and review of data for data-informed decision-making processes, regardless of the source of data\u0026rdquo; and therefore use \u0026ldquo;... goes beyond data reporting and passive dissemination of reports.\u0026rdquo; (10, p2). Nutley (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e) goes on to categorise data use in terms of data and information regularly demanded, analysed, synthesised, reviewed and used in: (i) program review and planning, (ii) advocacy and policy development, and (iii) decision-making processes. Nutley doesn\u0026rsquo;t define each of these categories but classifies all three as the long-term outcomes of the use of data. We are particularly concerned in this review with data being used in program review and planning - informing the development of plans, in monitoring and supervision and in evaluations or review of the plans.\u003c/p\u003e \u003cp\u003eGiven the divergence in defining data use, it is not surprising that definitions and methods for monitoring and measuring data use have posed challenges. Nutley and Li (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e) note, data sharing, visualisation, dissemination, and review are often considered cases of data use. As a result, there are many different dimensions of data use that get measured, for example transparency, timeliness, visibility, accessibility, dissemination of information, calculation of key indicators, preparation of information products, and presentation of the achievement of targets (Nutley and Li, 2018). In this review we focus on the use of routine data and information specifically for the making of plans and their monitoring and supervision at sub-national level in the health system. This emphasises continuity of data use in quantifying the key performance indicators in the initial plan, collecting quality data at a local level and using the same indicators for periodic monitoring by program managers and their supervisors doing performance assessment. This does not guarantee action, but ensures that there is high quality data available, visible and shared throughout the planning and implementation cycle.\u003c/p\u003e"},{"header":"Methods \u0026 analysis","content":"\u003cp\u003eThe Joanna Briggs Institute Guidelines approach of Peters et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e) was followed in this review and included the following steps: defining and aligning the objective/s and question/s; developing and aligning the inclusion criteria with the objective/s and question/s; describing the planned approach to evidence searching, selecting, charting and summarising the evidence. The review question for this literature review was:\u003c/p\u003e \u003cp\u003e \u003cem\u003eHow are routine health information systems used or should be used in developing and monitoring health plans at district and facility level?\u003c/em\u003e \u003c/p\u003e \u003cp\u003eDatabases searched included: Ovid Medline (all), EMBASE and Web of Science along with a review of grey literature such as documents from WHO, MEASURE, UNICEF, DHIS2 resources, as well as involving a number of key stakeholders in reviewing and identifying any missing resources.\u003c/p\u003e \u003cp\u003eInclusion criteria were that the documents describe how RHISs are used or should be used in developing and monitoring their annual health plans. Exclusion criteria were:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eArticles that mention challenges/ concerns with the development or monitoring of health plans only.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eArticles that do not use routine data to develop or monitor health plans, but conduct a particular survey or research to monitor plans.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eDocuments that are theoretical/conceptual only with no examples of use in practice.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eNon-English language studies.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eAll types of studies were included, as well as original research and reviews. No quality review was conducted as within a scoping review the complete landscape of publications is to be included regardless of quality. However, the article needed to describe the methodology and findings in sufficient detail to be informative in some way, such as in terms of process, content, or findings.\u003c/p\u003e \u003cp\u003eSearch terms included: \u0026lsquo;routine health information system(s)\u0026rsquo; AND \u0026lsquo;plan\u0026rsquo; OR \u0026lsquo;micro-plans\u0026rsquo; AND \u0026lsquo;monitoring\u0026rsquo; (see annex 1 for search strings). As the RED strategy is particularly relevant to the review question the same databases were also searched independently for any article that included \u0026lsquo;Reaching Every District\u0026rsquo;, \u0026lsquo;RED\u0026rsquo; or \u0026lsquo;Reaching Every Child\u0026rsquo;.\u003c/p\u003e \u003cp\u003eAll results were imported into Covidence and duplicates removed. This search was updated periodically after the project start date and included the articles or documents retrieved through snowballing or from stakeholders. Included documents were validated by consulting with expert stakeholders to check for any missing relevant documentation. Identified sources of evidence underwent a two-level review process: a title and abstract review, and a full-text review. Data was charted against a number of criteria: Title of article/publication, Lead author, Year, Journal/publication outlet and Country of study. Further details included: practical examples of M\u0026amp;E of action plan, how M\u0026amp;E was implemented, at what level M\u0026amp;E took place, whether RHIS was highlighted as main source of data, whether a framework/model of M\u0026amp;E was presented, description of framework/mode, how M\u0026amp;E is related to a plan, what plan and what the main purpose of M\u0026amp;E.\u003c/p\u003e \u003cp\u003eA total of 2442 articles were imported, 153 duplicates removed and 2289 articles\u0026rsquo; title and abstract reviewed. Forty-five full texts were reviewed and 32 excluded with the main reasons being the lack of focus of planning. Data from thirteen articles was extracted (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Both authors screened, reviewed and extracted at all stages. Both authors have decades of experience in RHIS in developing countries at district level and below, as well as in researching and writing about data and information use. Conflicts were resolved between the two authors, though the department research group was available to arbitrate if it had been needed. The list of articles included and preliminary findings were shared with stakeholders from global health institutions such as UNICEF, WHO and RHINO, the health information systems research group at University of Oslo, and the Global Health Information Systems Programme network.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Description of documents included","content":"\u003cp\u003eA total of 13 documents were included at full extraction stage (Table\u0026nbsp;1).\u003c/p\u003e\n\u003cp\u003eMost of the documents were published between 2013 and 2019, with 3 published in 2013 and 2016 (Fig. 2).\u003c/p\u003e\n\u003cp\u003eThe documents were published in health service, health policy, systems and planning journals (9) with others in information systems (1), East Mediterranean Health (1), rural health (1) and public health (1) (Fig. 3).\u003c/p\u003e\n\u003cp\u003eMost of the articles were based on studies in Africa and Asia, with one in South America, Middle East and one in general on LMICs (Fig. 4).\u003c/p\u003e"},{"header":"Content of documents included:","content":"\u003cp\u003eThe documents are described in two subsections based on the two overarching categories that emerged from the included articles: results from country assessments and reviews of planning processes at sub-national level. In terms of the exclusion criteria we excluded theoretical/conceptual articles/reports that had no practical base. Examples of those excluded are:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003ea logic model for improving the use of health data for health system strengthening with recommendations that affect the use of data in decision making (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ea correspondence article on planning, implementation, monitoring, and evaluation of integrated health services which emphasises the importance of strong M\u0026amp;E systems (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ea country-led platform for information and accountability that provides guidance to countries and partners for strengthening monitoring, evaluation and review (M\u0026amp;E) of national health plans and strategies (NHS), but no specific plan is being monitored (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eUse of RHIS to evaluate HSS interventions, with a good description of indicator selection, but not linking use of RHIS to evaluate plans (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eOrganizing Framework for a Functional National HIV Monitoring and Evaluation System but does not provide detailed guidance on how to operationalize the system (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eHowever, in the \u003cspan refid=\"Sec8\" class=\"InternalRef\"\u003ediscussion\u003c/span\u003e section of this article we look at how some of these conceptual models could assist in informing guidance on local level planning, monitoring and supervision of action plans.\u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e6.3. Evaluations/Assessments of M\u0026amp;E systems and processes\u003c/h2\u003e \u003cp\u003eIn a study in South Africa on perceptions about data-informed decisions for HIV, Nicol et al (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e) explore the challenges in relation to data use and note that organisational and capacity issues need to be addressed before information will be used. This includes the development of a culture of information use, trust in the data, capacity to analyse, interpret and use information. They suggest that \"Facility and program managers should be provided with opportunities for capacity development as well as practice-based, in-service training, and be supported to use information for planning, management and decision-making\" (17, p.765). However, there are no practical guidelines or suggestions on how this is to be achieved. They observe that there are mechanisms and processes in place to promote use of information (performance meetings, access to routine reports, directives/ SOPs, monthly targets, existence of information), but these are not being used (or are 'selectively used'). In a similar fashion Nabyonga-Orem, J. et al (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e) look at the harmonisation and standardisation of the health sector and programme reviews in the WHO African Region. They highlight the main challenges in terms of performance assessment (weak institutional capacity for M\u0026amp;E, desynchronised planning timeframes, inadequate time allocated for comprehensive performance assessments, weak follow-up mechanisms, lack of stakeholder engagement and divergent political agendas). They call for the standardisation and institutionalisation of performance assessments, but give no detail on how this can be done and do not link it with planning and a comprehensive M\u0026amp;E framework.\u003c/p\u003e \u003cp\u003eIn Malawi, Chaulagai, C.N., et al (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e) reviewed the District Implementation Plan (DIP) process and described the stages of getting baseline data to set priorities and targets in the DIP. They conclude that most DIPs are vague and that it is hard to track implementation status and results. There were some attempts to computerise the DIP process so that there are links with the routine data collected for monitoring the plan, but this resulted in only the person entering the data being involved in the planning and monitoring process. The purpose of the DIP process was to enable the allocation of resources based on performance, so that facilities and districts could compare or rank their performance in relation to other facilities of districts. The DIPs were also accompanied by a system of quarterly feedback, supportive supervision visits and annual reviews. However, there is no detailed description of a M\u0026amp;E plan or process on how the DIP should be monitored or supervised. However, some good practices were identified \u0026ndash; gaining consensus on indicators and tools, skills training on utilising existing data to calculate indicators and management of health services, establishing regular meetings and reporting (quarterly management and annual performance reviews at all levels) and the development of routine monitoring and guidelines for an annual health sector joint review. They conclude that there was overall little improvement in the use of data in rationalising decisions and that \"no matter how good the design of an information system, it will not be effective unless there is internal desire, dedication and commitment of leadership to have an effective and efficient health service management system\u0026rdquo; (19, p.375).\u003c/p\u003e \u003cp\u003eSo overall the published evaluations/assessments of M\u0026amp;E systems and processes do not reveal a practice of using RHIS routine data in the planning, monitoring and supervision process. However, some of the key findings could inform the implementation of such a system, namely, the need to develop organisational and institutional capacity (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e), the standardisation and institutionalising of performance assessment (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e) and the commitment of leadership to effective and efficient service management (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThough not showing that RHIS is being used some of the other articles reviewed indicate that RHIS data could be used in monitoring programmes, even if not all RHIS data is not of the same quality, by Increasing use, data quality monitoring, and management and information technology of RHIS (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e). Cibulskis, R.E. et al (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e) argue that even imperfect routine health data can be used whilst simultaneously working on the quality and completeness of it to encourage \u0026lsquo;a more methodical approach to planning and monitoring services\u0026rsquo; and with improved quality of data be used on a routine basis to monitor performance.\u003c/p\u003e \u003cp\u003eIn the remaining articles there is mention of data being used in planning and monitoring, but findings are not particularly informative in terms of practical examples or lessons learned. For example, a theoretically informed review of the Kenyan HIS and accountability, includes some mention of use of data in monitoring plans, but it is rather vague (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e); a M\u0026amp;E system at national level in Brazil highlights the importance of integrating all the M\u0026amp;E tools and plans across departments and units, but not focusing on how this was done or how the integrated plans were monitored (\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e), and; the development of a framework for the M\u0026amp;E of a National plan in Iran indicates that numerous surveys and routine information systems would be needed to monitor the plan and even using these sources there still would be gaps in data needed to monitor the plan (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e). Similarly, a review of 5 health system strengthening changes made note that a common evaluation framework of HIS strengthening was used, but it is not described (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). The authors note that there is hope that this will assist in \u0026ldquo;\u0026hellip; linking HIS with decision making, and its impact on measures of health system outputs and impact\u0026rdquo; (25, p.1), but no examples are given of these decisions or processes involved in linking HIS, decision making and impact. Exploring other changes made to HIS for improving use in decision-making Nutley, T. et al (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e) look at the impact of the District Health Plan (DHP) decision-support tool on decision making at the district level in Kenya, but there is little detail given on the tool and there is no link to a plan or monitoring of the plan. In fact, the 11 review questions of the HIV programme included in the tool works in parallel rather than as part of an integrated M\u0026amp;E system.\u003c/p\u003e \u003c/div\u003e"},{"header":"Evaluations of plans","content":"\u003cp\u003eOne example of monitoring facility level plans was in the study by Enkhtuya et al (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e) when reviewing the national RED planning in one district. The district and family practitioners were able to map areas of low coverage, undertake barrier analysis, and provide detailed costed activities required to reach these populations. The RED M\u0026amp;E plan includes supportive supervision (such as local problem solving, involvement of community), micro-planning activities and budget requests through normal annual operational planning and budgeting processes. Overall though the reviewers concluded that there is a need to change the RED approach to management as local area knowledge is currently absent in plans and supportive supervision. Changing the style and approach to planning requires a change in the style of and approach to management - moving from supervisor as inspector to supporter. Though the RED strategy is intended as an immunisation-specific intervention, the process in Mongolia indicates its potential for wider health system applications. In fact Chan Soeung, S. et al (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e) draw a similar conclusion on community level involvement in terms of planning, recognising that \u0026ldquo;a shift in planning focus is needed from a district-wide perspective down to a facility- and community-level system of analysis and operations\u0026rdquo;(28, p.532). However, the Chan Soeung, S. et al (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e) study was on equity and not planning and so provides little insight into using RHIS for planning, monitoring and supervision.\u003c/p\u003e \u003cp\u003eThe only other article that looked specifically at decision-making and planning was a literature review of decision making for health in LMICs by Wickremasinghe et al (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e). They found twelve examples of tools to assist district-level decision-making. The use in decision-making comprised generally the use of data to identify priorities, and in developing an action plan to address those priorities. Four of the studies included steps for reviewing or monitoring the plans with HIS data, but \u0026ldquo;\u0026hellip; there was limited evidence about their sustained impact on district level decision-making and whether they have led to changes in resource allocation patterns\u0026rdquo; (29, pii23). However, in terms of lessons around data use that could be used in terms of planning, monitoring and supervision, there were three features that kept recurring: relevance depended on data quality, for consensus a structured decision-making process is needed and that communities need to be included in the decision making process.\u003c/p\u003e \u003cp\u003eSo overall the articles indicate the need for local level planning that involves communities and the review article additionally indicates the need for data quality and a structured process for decision making.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study looked at the link between planning and monitoring at subnational level using routine information and found that there were surprisingly few articles that looked at the concept and even fewer that described the practicalities of implementing the links. While there are a number of high level descriptions of what should be done to develop a culture of information use, most were very vague and there were very few details of how this process can be institutionalised, how staff are held accountable for implementing their plans or how to actually monitor the plans made. What we found missing was how to institutionalise regular, structured data use for monitoring program performance at service delivery level and ensure its discussion and interpretation by lower level stakeholders.\u003c/p\u003e\n\u003cp\u003eHowever, there are some existing frameworks on the use of routine data in monitoring as well as global strategies that support the development and review of lower level/micro plans. We didn\u0026rsquo;t find any articles using these in the literature we reviewed but they could be used in conjunction with the findings from our review to inform guidelines for planning, monitoring and supervision at local level.\u003c/p\u003e\n\u003cp\u003eNutley and Lee (\u003cspan class=\"CitationRef\"\u003e4\u003c/span\u003e) reviewed the main assessments and tools used for data use. They include a review of data use in the health information system strengthening model (HISSM) and the Measure Evaluation Logic Model for improving data use. The Measure Evaluation Logic Model describes specific activities (8 domains) and interventions needed to improve the use of health data for improved health programs and policies. This model builds on the HISSM by providing specific and detailed ways to support the use of HIS data. There is no specific reference to use of data in developing, monitoring and supervising plans made in any of the models, but Nutley and Li (\u003cspan class=\"CitationRef\"\u003e4\u003c/span\u003e) look at various assessment tools and map them according to the dimensions of data use (data quality, health statistics, information products, data review, advocacy, decision and action). None of these tools are specific to data use and action planning. Nutley and Li conclude that \u0026ldquo;Few tools that measure the outcome of data use for improved health program performance exist. ..... Better measures of the outcome of data use are needed, along with ways to easily track the health program and health system outcomes associated with decisions that are implemented.\u0026quot; (4, p.32). Our review confirms this conclusion.\u003c/p\u003e\n\u003cp\u003eTo consolidate the contributions of the included articles in this review towards this gap we map the content of the articles extracted against the 8 domains of the Measure Evaluation Logic Model (Table\u0026nbsp;2):\u003c/p\u003e\n\u003col style=\"list-style-type: lower-roman;\"\u003e\n \u003cli\u003eAssessing and improving the data use context\u003c/li\u003e\n \u003cli\u003eEngaging data users and data producers\u003c/li\u003e\n \u003cli\u003eImproving data quality\u003c/li\u003e\n \u003cli\u003eImproving data availability\u003c/li\u003e\n \u003cli\u003eIdentifying information needs\u003c/li\u003e\n \u003cli\u003eBuilding capacity in data use core competencies\u003c/li\u003e\n \u003cli\u003eStrengthening the organisation\u0026rsquo;s infrastructure\u003c/li\u003e\n \u003cli\u003eMonitoring, evaluating, \u0026amp; communicating successes\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThere are a couple of the domains that could be expanded based on the articles we extracted, namely the assessment and improving the data use context and strengthening the organisations infrastructure. In relation to the domain of assessing and improving the data use context, this is often referred to as developing a culture of information use, we are reminded by Reynolds, H. W. et al (\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e) and Bernardi (\u003cspan class=\"CitationRef\"\u003e30\u003c/span\u003e), that such a culture requires an institution to take responsibility and accountability. More detail on how to achieve improvements in the context include: stakeholders linking data collection and strategic plans, operational plans and program plans (\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e); stakeholders placing value on the data (Nutley et al 2013) and showing commitment (\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e); gaining consensus on a structured decision-making process (\u003cspan class=\"CitationRef\"\u003e29\u003c/span\u003e). All of this requires standardisation, institutionalisation and coordination of planning, M\u0026amp;E, assessments and follow up mechanisms (\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e). As Mutale et al (\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e) note, data use in planning, monitoring and supervision needs to be as institutionalised as stocking a pharmacy or immunising a child.\u003c/p\u003e\n\u003cp\u003eIn a commentary on the historical evolution of monitoring and evaluation Thomas, J.C. et al (\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e) note that monitoring and evaluation for health in LMICs has advanced from an emerging discipline to one that adheres to standards, is systematised and mainstreamed across programmes. They also note that significant capacity has been developed and resources allocated to collecting and using data, enabling a move away from paper-based to electronic systems. What is interesting are the tensions that exist with these changes:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003esingle, unified country system versus accountability and control by particular disease/ programmes and particular donors;\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003edesire for a shared integrated system versus the desire for specific outcomes\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003ecountry autonomy versus donor control.\u003c/p\u003e\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf we are to see more examples of RHIS being used in planning, monitoring and evaluation these tensions need to be addressed in creating the context for effective data use.\u003c/p\u003e\n\u003cp\u003eIn terms of engaging data users and data producers Wickremasinghe et al, (\u003cspan class=\"CitationRef\"\u003e29\u003c/span\u003e) remind us of the importance of including the community in the decision making process as key data users and producers. Shifting planning emphasis to the health centre and community is also raised (\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eIn relation to data quality, most articles highlighted how important data quality was if it was to be used. Many tools have been developed to improve data quality including minimal data entry and automated graphs (\u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e). Equally important is that the data is trusted by users (\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e) and this is often subjective and goes beyond data quality. Cibulskis, R.E. et al (\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e), as do Wagenaar et al (\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e), remind us that we should not wait until data quality is \u0026lsquo;perfect\u0026rsquo; before using the data as improving the existing system can continue while data is being used and using data and improving data quality is a duality with one feeding in the other. Counter arguing the poor quality of RHIS, Wagenaar et al (\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e) note that RHIS data are superior to intermittent community sample surveys which can have data delays far works than RHIS and avoid the high costs of conducting surveys. They note that there are still existing challenges with RHIS such as population data, including the excluded and programmes or policies not provided by the facility or their outreach services.\u003c/p\u003e\n\u003cp\u003eThere are many examples given of mechanisms or activities to strengthen the organisational infrastructure. Chaulagai, C.N., et al (\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e) give a comprehensive list, and Nicol et al (\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e) and others give examples of key issues they found to facilitate use in their reviews. These include, but are not limited to: gaining consensus on indicators and tools, skills training on utilising existing data to calculate indicators and management of health services, establishing regular meetings and reporting, the development of routine monitoring and guidelines for periodic reviews, and standardising and institutionalising the planning and monitoring process through agreed strategies, SoPs and guidelines. However, there is no mention of the need to institutionalise monitoring of annual plans at sub-national level and using this information to supervise managers and hold them accountable for implementation of the plans.\u003c/p\u003e\n\u003cp\u003eThe RED approach, with its five operational strategies developed through a facility/ district micro planning tool using routine data, comes the closest to describing the expected links between planning, monitoring and supervision. The strategy has been implemented since 2003 in 53 LMICs, (\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e) and adapted to country realities, but multiple studies show that very few districts or facilities had updated micro plans (\u003cspan class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e). An excellent tool with the support of all major immunisation donors has been squandered because it does not clearly define implementation modalities and as a result has not been followed through at subnational level.\u003c/p\u003e\n\u003cp\u003eFundamentally what emerges from this review is that the main obstacle to using routine data in monitoring and supervision of annual plans is the weak institutionalisation of the planning and monitoring processes and lack of clarity on the organisational procedures needed for this to occur at sub-national level. This requires a data use vision by national leaders that links planning to monitoring and promotes commitment and leadership at all levels.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe basic premise of decentralised planning is that district level plans should be regularly monitored by the planners themselves using routine data and their implementation supervised by immediate managers (\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e). Our review has found that this linkage does not exist in the published literature. There appears to be no institutionalised obligation of planners to monitor (micro) plans, very little guidance on how to practically monitor programs and minimal discussion about how to use the routinely available data to supportively supervise the implementation of the plans.\u003c/p\u003e \u003cp\u003eOverall, there are many descriptions of how to implement and strengthen systems, ways to assess and improve data availability and quality, tools to improve the data use context, training in data use and mechanisms to involve stakeholders and strengthen infrastructure. However, there are massive gaps in the literature in relation to good use cases or examples of where routine health data is used in the development, monitoring and supervision of plans at district and facility level.\u003c/p\u003e \u003cp\u003eLikewise, there are gaps in terms of guidelines on how this can happen. The RED approach appears to be the best available framework as it clearly links facility micro-planning, local monitoring of implementation and supervision (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e) and provides some instruction on how to implement the various steps. However, global RED guidelines lack practical details and it seems they have not been adapted to ensure local implementation, particularly around data collection and use. Consolidating the information on M\u0026amp;E from the reviewed documents and mapping to the Measure Evaluation logic model domains could be a useful starting point in developing such guidelines.\u003c/p\u003e \u003cp\u003eIt is possible that some relevant documents may be published in journals not indexed on the databases searched or the key experts consulted were not aware of guidelines available. We are therefore not concluding that there are no other examples or guidelines that exist, but that these examples or guidelines are not readily available in the published literature. The existing mechanisms that could be built upon include performance meetings, access to routine reports, directives/ SOPs, monthly targets, existence of information, but these need practical guidelines on how and when to use.\u003c/p\u003e \u003cp\u003eThe main gaps that need to be addressed are,\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003ePractical procedures to ensure linkage of existing district plans to regular institutionalised monitoring of priority programs\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eMechanisms for making managers institutionally accountable for monitoring and supervising these plans, and\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDeveloping practical guidelines for linking plans with RHIS data and linking plans with regular monitoring and supportive supervision.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eDHIS2 District Health Information Software 2\u003c/p\u003e\u003cp\u003eDHP District Health Plan\u003c/p\u003e\u003cp\u003eDIP District Implementation Plan\u003c/p\u003e\u003cp\u003eHIS health Information System\u003c/p\u003e\u003cp\u003eHISSM Health Information System Strengthening\u003c/p\u003e\u003cp\u003eLMICs Lower and Middle Income Countries\u003c/p\u003e\u003cp\u003eM\u0026amp;E Monitoring and Evaluation\u003c/p\u003e\u003cp\u003ePRISM Performance of Routine Information System Management\u003c/p\u003e\u003cp\u003eRED Reaching Every District\u003c/p\u003e\u003cp\u003eRHIS Routine Health Information Systems\u003c/p\u003e\u003cp\u003eUNICEF United Nations Children\u0026rsquo;s Fund\u003c/p\u003e\u003cp\u003eWHO World Health Organisation\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable as a desk review of material in the public domain\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable as no personal or individual data collected or reported on\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll data generated or analysed during this study are included in this published article.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFunding from GAVI Global (A5) support to HISP Centre UiO 2021-2023 supported AH time in conducting this review.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBoth authors equally contributed to the conception of the work. The search terms and inclusion and exclusion criteria were agreed upon by both authors. A librarian, Paul Murphy, at the RCSI conducted the search in consultation with EB. EB imported the combined documents returned from the databases into covidence. \u0026nbsp;Both authors equally contributed to the data analysis and interpretation, drafting, revising and final approval of the article.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe appreciate the funding from GAVI for this work which is part of their continued support to the Global HISP network.\u003c/p\u003e\n\u003cp\u003eThanks to Paul Murphy Information Specialist at RCSI Library for his expertise, advice and support in conducting the search.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe are also grateful for all the experts who gave of their time to review the different versions of our draft manuscript and forwarded documents/links of potential sources of further information for our review.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe appreciate the feedback from the Health Information Systems Programme Designing for Data Use Lab group at the HISP Centre, University of Oslo . \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; information\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eElaine Byrne (PhD) worked with HISP South Africa (1997-2008) when living and working in South Africa and whilst doing her PhD. \u0026nbsp;In Oct 2021 she joined the Department of Informatics at the University of Oslo on a leave of absence from the University of Medicine and Health Sciences, Royal College of Surgeons in Ireland (RCSI) to work on DHIS2 data use practice. \u0026nbsp;Her general research interests are around research that supports practice and focuses on healthy people in healthy societies. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eArthur Heywood (AH) is a public health veteran activist., formerly a university lecturer and professor in South Africa, and currently consultant with Global Health Institutions, such as GAVI, UNICEF, WHO and HISP largely on the implementation and use of RHIS. \u0026nbsp;Arthur set up a Masters programs in Public Health and Information Systems in South Africa, Mozambique, Malawi and Tanzania, all of them with a focus on using information to improve PHC service provision and to promote equity. \u0026nbsp;He is also a founding member of the Health Information Systems Program in post-apartheid South Africa.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAbouZahr C, Boerma T. Health information systems: the foundations of public health. . Bull World Health Organ 2005;83(8):578-83.\u003c/li\u003e\n\u003cli\u003eMEASURE Evaluation. Strengthening Health Information Systems in Low- and Middle-Income Countries\u0026mdash;A Model to Frame What We Know and What We Need to Learn. 2017.\u003c/li\u003e\n\u003cli\u003eWorld Health Organization. Everybody\u0026rsquo;s business: strengthening health systems to improve health out comes: WHO\u0026rsquo;s framework for action. . Geneva: World Health Organization; 2007.\u003c/li\u003e\n\u003cli\u003eNutley T, Li M. Conceptualizing and Measuring Data Use: A Review of Assessments and Tools. University of North Carolina: MEASURE Evaluation; 2018.\u003c/li\u003e\n\u003cli\u003eByrne E, S\u0026aelig;b\u0026oslash; JI. Use of DHIS2 Data: A Scoping Review. BMC Health Services Research. 2022;1234( https://doi.org/10.1186/s12913-022-08598-8).\u003c/li\u003e\n\u003cli\u003eAsah FN, Nielsen P, S\u0026aelig;b\u0026oslash; JI, editors. Challenges for Health Indicators in Developing Countries: Misconceptions and Lack of Population Data. 14th International Conference on Social Implications of Computers in Developing Countries IFIP 94 WG; ICTs for promoting social harmony: Towards a sustainable information society; 2017; Indonesia: Cham: Springer International Publishing.\u003c/li\u003e\n\u003cli\u003eHeywood A. Data Use in Chinsali, Zambia. 2021.\u003c/li\u003e\n\u003cli\u003eBoerma T, AbouZahr C, Bos E, Hansen P, Addai E, Low-Beer D. Monitoring and Evaluation of health systems strengthening: an operational framework. WHO, World Bank, GAVI and Global Fund; 2009.\u003c/li\u003e\n\u003cli\u003eJones M. What we talk about when we talk about (big) data. The Journal of Strategic Information Systems. 2019;28(1):3-16.\u003c/li\u003e\n\u003cli\u003eNutley T. Improving Data Use in Decision Making: An Intervention to Strengthen Health Systems. MEASURE Evaluation; 2012.\u003c/li\u003e\n\u003cli\u003ePeters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13(3):141-6.\u003c/li\u003e\n\u003cli\u003eNutley T, Reynolds HW. Improving the use of health data for health system strengthening. Global Health Action. 2013;6(1):20001.\u003c/li\u003e\n\u003cli\u003eReynolds HW, Sutherland EG. A systematic approach to the planning, implementation, monitoring, and evaluation of integrated health services. BMC Health Serv Res. 2013;13:168.\u003c/li\u003e\n\u003cli\u003eWorld Health Organization, Internationl Health Partnerships. Monitoring, evaluation and review of national health strategies: a country-led platform for information and accountability.; 2011.\u003c/li\u003e\n\u003cli\u003eLaw MR, Hotchkiss DR, Gr\u0026eacute;pin KA. Evaluating Health Systems. Strengthening Interventions Using Routinely Collected Data. 2022.\u003c/li\u003e\n\u003cli\u003eUNAIDS MERG. Organizing Framework for a Functional National HIV Monitoring and Evaluation System. 2008.\u003c/li\u003e\n\u003cli\u003eNicol E, Bradshaw D, Uwimana-Nicol J, Dudley L. Perceptions about data-informed decisions: an assessment of information-use in high HIV-prevalence settings in South Africa. BMC Health Services Research. 2017;17(2):765.\u003c/li\u003e\n\u003cli\u003eNabyonga-Orem J, Tumusiime P, Nyoni J, Kwamie A. Harmonisation and standardisation of health sector and programme reviews and evaluations \u0026ndash; how can they better inform health policy dialogue? Health Res Policy Sys 2016;14(87).\u003c/li\u003e\n\u003cli\u003eChaulagai CN, Moyo CM, Koot J, Moyo HB, Sambakunsi TC, Khunga FM, et al. Design and implementation of a health management information system in Malawi: issues, innovations and results. Health Policy and Planning. 2005;20(6):375-84.\u003c/li\u003e\n\u003cli\u003eChaulagai CN, Moyo CM, Koot J, Moyo HBM, Sambakunsi TC, Khunga FM, et al. Design and implementation of a health management information system in Malawi: issues, innovations and results. Health Policy and Planning. 2005;20(6):375-84.\u003c/li\u003e\n\u003cli\u003eWagenaar BH, Sherr K, Fernandes Q, Wagenaar AC. Using routine health information systems for well-designed health evaluations in low- and middle-income countries. HEALTH POLICY AND PLANNING. 2016;31(1):129-35.\u003c/li\u003e\n\u003cli\u003eCibulskis RE, Hiawalyer G. Information systems for health sector monitoring in Papua New Guinea. Bull World Health Organ. 2002;80(9):752-8.\u003c/li\u003e\n\u003cli\u003eSellera PEG, Brito CBM, Jovanovic MB, Rodrigues SO, Oliveira C, Santos SOD, et al. The Implementation of the Monitoring and Evaluation System of the State Health Secretariat of the Brazilian Federal District (SHS/DF). Cien Saude Colet. 2019;24(6):2085-94.\u003c/li\u003e\n\u003cli\u003eAbdi Z, Majdzadeh R, Ahmadnezhad E. Developing a framework for the monitoring and evaluation of the Health Transformation Plan in the Islamic Republic of Iran: lessons learned. East Mediterr Health J. 2019;25(6):394-405.\u003c/li\u003e\n\u003cli\u003eMutale W, Chintu N, Amoroso C, Awoonor-Williams K, Phillips J, Baynes C, et al. Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative. BMC Health Serv Res. 2013;13 Suppl 2(Suppl 2):S9.\u003c/li\u003e\n\u003cli\u003eNutley T, McNabb S, Salentine S. Impact of a decision-support tool on decision making at the district level in Kenya. Health Res Policy Syst. 2013;11:34.\u003c/li\u003e\n\u003cli\u003eEnkhtuya B, Badamusuren T, Dondog N, Khandsuren L, Elbegtuya N, Jargal G, et al. Reaching every district - development and testing of a health micro-planning strategy for reaching difficult to reach populations in Mongolia. Rural Remote Health. 2009;9(2):1045.\u003c/li\u003e\n\u003cli\u003eChan Soeung S, Grundy J, Duncan R, Thor R, Bilous JB. From reaching every district to reaching every community: analysis and response to the challenge of equity in immunization in Cambodia. . Health Policy Plan. 2013;28(5):526-35.\u003c/li\u003e\n\u003cli\u003eWickremasinghe D, Hashmi IE, Schellenberg J, Avan BI. District decision-making for health in low-income settings: a systematic literature review. Health Policy Plan. 2016;31 Suppl 2(Suppl 2):ii12-ii24.\u003c/li\u003e\n\u003cli\u003eBernardi R. Health Information Systems and Accountability in Kenya: A Structuration Theory Perspective. Journal of the Association for Information Systems. 2017;18(12).\u003c/li\u003e\n\u003cli\u003eThomas JC, Doherty K, Watson-Grant S, Kumar M. Advances in monitoring and evaluation in low- and middle-income countries. Evaluation and Program Planning. 2021;89:101994.\u003c/li\u003e\n\u003cli\u003eMafigiri DK, Iradukunda C, Atumanya C, Odie M, Mancuso A, Tran N, et al. A qualitative study of the development and utilization of health facility-based immunization microplans in Uganda. Health Res Policy Syst. 2021;19(Suppl 2):52.\u003c/li\u003e\n\u003cli\u003eNgomba AV, Kollo B, Bita AF, Djouma FN, Edengue JM, Elongue MJ, et al. [Immunization programme in urban areas in Cameroon: a case study of the Djoungolo Health District]. Pan Afr Med J. 2016;25:213.\u003c/li\u003e\n\u003cli\u003eLankester T, Nathan G. Monitoring and evaluating the health programme. In: Lankester T, Nathan G, editors. Setting up Community Health and Development Programmes in Low and Middle Income Settings. 4th ed: Oxford, 2019; online edn, Oxford Academic, 1 Mar. 2019; 2019.\u003c/li\u003e\n\u003cli\u003eGreen A. An Introduction to Health Planning in Developing Countries. 2nd ed: Oxford Medical Publications, Oxford University Press; 1999.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable 1\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"614\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eLead author\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eTitle\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003ePublication outlet\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eAbdi, Z. et al\u0026nbsp;(2019)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eDeveloping a framework for the monitoring and evaluation of the Health Transformation Plan in the Islamic Republic of Iran: lessons learned\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eEast Mediterr Health J\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eBernardi, R.\u0026nbsp;(2017)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Information Systems and Accountability in Kenya: A Structuration Theory Perspective\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eJournal of the Association for Information Systems\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eChan Soeung, S. et al. (2013)\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eFrom reaching every district to reaching every community: analysis and response to the challenge of equity in immunization in Cambodia\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Policy and Planning\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eChaulagai, C.N., et al\u0026nbsp;(2005)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eDesign and implementation of a health management information system in Malawi: issues, innovations and results\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Policy and Planning\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eCibulskis, R.E. et al\u003c/p\u003e\n \u003cp\u003e(2002)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eInformation systems for health sector monitoring in Papua New Guinea\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003ePolicy and Practice\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eEnkhtuya, B. et al\u003c/p\u003e\n \u003cp\u003e(2009)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eReaching every district - development and testing of a health micro-planning strategy for reaching difficult to reach populations in Mongolia\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eRural and Remote Health\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eMutale, W. et al\u003c/p\u003e\n \u003cp\u003e(2013)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eImproving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eBMC Health Services Research\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eNabyonga-Orem, J. et al\u0026nbsp;(2016)\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eHarmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Research Policy \u0026amp; Systems\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eNicol, E. et al\u003c/p\u003e\n \u003cp\u003e(2017)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003ePerceptions about data-informed decisions: an assessment of information-use in high HIV-prevalence settings in South Africa\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eBMC Health Services Research\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eNutley, T. et al\u003c/p\u003e\n \u003cp\u003e(2013)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eImpact of a decision-support tool on decision making at the district level in Kenya\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Research Policy \u0026amp; Systems\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eSellera, P.E.G. et al\u003c/p\u003e\n \u003cp\u003e(2019)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eThe Implementation of the Monitoring and Evaluation System of the State Health Secretariat of the Brazilian Federal District (SHS/DF)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eCi\u0026ecirc;ncia \u0026amp; Sa\u0026uacute;de Coletiva\u003c/em\u003e (Science \u0026amp; Public Health)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eWagenaar, B.H. et al\u0026nbsp;(2016)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eUsing routine health information systems for well-designed health evaluations in low- and middle-income countries\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Policy and Planning\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.403908794788272%\" valign=\"top\"\u003e\n \u003cp\u003eWickremasinghe, D. , et al\u0026nbsp;(2016)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"57.00325732899023%\" valign=\"top\"\u003e\n \u003cp\u003eDistrict decision-making for health in low-income settings: a systematic literature review\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"24.592833876221498%\" valign=\"top\"\u003e\n \u003cp\u003eHealth Policy and Planning\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eTable 2\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"1002\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eMeasure Evaluation logic model domains\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eContent of included articles related to domain\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eAssessing and improving the data use context\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003eassess and improve the data use context/data culture\u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eorganisational and capacity issues such as the development of a culture of information use\u0026nbsp;(Nicol et al., 2017)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003erequires an institution taking responsibility and accountability\u0026nbsp;(Bernardi, 2017; Reynolds \u0026amp; Sutherland, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003estakeholders need to make the link between data collection and strategic plans, operational plans and program plans\u0026nbsp;(Reynolds \u0026amp; Sutherland, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003estakeholders and decision-makers need to place value on data they use in decision making\u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u003c/li\u003e\n \u003cli\u003eThere needs to be \u0026nbsp;\u0026lsquo;the internal desire, dedication and commitment of leadership \u0026rsquo;\u0026nbsp;(Chaulagai et al., 2005)\u0026nbsp;\u0026nbsp;\u003c/li\u003e\n \u003cli\u003efor consensus a structured decision-making process is needed\u0026nbsp;(Wickremasinghe et al., 2016)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003estandardisation, Institutionalisation and coordination of planning, M\u0026amp;E, assessments and follow up mechanisms\u0026nbsp;(Nabyonga-Orem et al., 2016)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eMonitoring programs needs to be \u0026ldquo;as institutionalised as stocking a pharmacy or immunising a child\u0026rdquo; \u0026nbsp;(Mutale et al., 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eProblem-solving planning methodology progressing \u0026nbsp; \u0026nbsp; from \u0026nbsp;health \u0026nbsp;mapping \u0026nbsp; \u0026nbsp; to \u0026nbsp;barrier \u0026nbsp;analysis, \u0026nbsp; \u0026nbsp; to \u0026nbsp;activity \u0026nbsp;planning \u0026nbsp; \u0026nbsp; and \u0026nbsp;costing \u0026nbsp;and \u0026nbsp; \u0026nbsp; finally \u0026nbsp;to \u0026nbsp;monitoring and evaluation\u0026nbsp;(Enkhtuya et al., 2009)\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eEngaging data users and data producers\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003eengage with other stakeholders - data users and data producers \u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u003c/li\u003e\n \u003cli\u003ecommunities need to be included in the decision making process\u0026nbsp;(Wickremasinghe et al., 2016)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eShift \u0026nbsp; of planning emphasis to the health centre and community \u0026nbsp; is needed. \u0026nbsp;(Chan Soeung et al., 2013; Enkhtuya et al., 2009)\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eImproving data quality\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003eimprove data quality\u0026nbsp;(Law et al., 2022; Nutley \u0026amp; Reynolds, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003erelevancy depends on quality (Wickremasinghe et al., 2016b.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003edeveloping trust in data\u0026nbsp;(Nicol et al., 2017)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eimprove existing systems whilst using even imperfect routine health data \u0026nbsp;(Cibulskis \u0026amp; Hiawalyer, 2002)\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eImproving data availability\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003eimprove data availability \u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eIdentifying information needs\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003eidentify information needs \u0026nbsp;(Nutley \u0026amp; Reynolds, 2013; World Health Organization \u0026amp; Internationl Health Partnerships, 2011)\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eBuilding capacity in data use core competencies\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003ebuild capacity in data use core competencies\u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003ecapacity to analyse, interpret and use information\u0026nbsp;(Nicol et al., 2017)\u003c/li\u003e\n \u003cli\u003eRe-orientation \u0026nbsp; \u0026nbsp; of \u0026nbsp;management \u0026nbsp;approaches \u0026nbsp; \u0026nbsp; from \u0026lsquo;inspection\u0026rsquo; to supportive supervision\u0026nbsp;(Enkhtuya et al., 2009)\u0026nbsp;\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eStrengthening the organisation\u0026rsquo;s infrastructure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003e\u0026nbsp;improve organisation\u0026rsquo;s data demand and use infrastructure \u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u0026nbsp;\u003c/li\u003e\n \u003cli\u003emechanisms to improve practice gaining consensus on indicators and tools, skills training on using data to calculate indicators and management of health services, establishing regular meetings and reporting, the development of routine monitoring and guidelines for an annual health sector review\u0026nbsp;(Chaulagai et al., 2005)\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"24.550898203592816%\" valign=\"top\"\u003e\n \u003cp\u003eMonitoring, evaluating, \u0026amp; communicating successes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"75.44910179640719%\" valign=\"top\"\u003e\n \u003cul\u003e\n \u003cli\u003e\u0026nbsp;monitor, evaluate, and communicate results of data use interventions\u0026nbsp;(Nutley \u0026amp; Reynolds, 2013)\u003c/li\u003e\n \u003cli\u003edevelopment \u0026nbsp;of communication \u0026nbsp;strategies \u0026nbsp; \u0026nbsp; and \u0026nbsp;health \u0026nbsp;networks \u0026nbsp; \u0026nbsp; that \u0026nbsp;enable \u0026nbsp;providers \u0026nbsp; \u0026nbsp; to \u0026nbsp;adjust service \u0026nbsp; delivery \u0026nbsp; \u0026nbsp; according \u0026nbsp;to \u0026nbsp;the \u0026nbsp; \u0026nbsp; needs \u0026nbsp;of \u0026nbsp;vulnerable \u0026nbsp; \u0026nbsp; population\u0026nbsp;(Chan Soeung et al., 2013)\u0026nbsp;\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-health-services-research","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bhsr","sideBox":"Learn more about [BMC Health Services Research](http://bmchealthservres.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/BHSR/default.aspx","title":"BMC Health Services Research","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Scoping review, Routine health information systems, planning, monitoring, supervision","lastPublishedDoi":"10.21203/rs.3.rs-2565795/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2565795/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRoutine Health Information Systems data should be used in a systematic and institutionalised manner to support the making of plans, the monitoring of plans and in supportive supervision. To explore to what extent there is discussion about the linkage between planning, monitoring and supervision of sub-national programs using routine data we conducted a scoping review. The review question was: \u003cem\u003eHow are routine health information systems used or should be used in developing and monitoring health plans at district and facility level?\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFrom a search of Ovid Medline (all), EMBASE and Web of Science along with a review of grey literature and involving a number of key stakeholders in reviewing and identifying any missing resources a total of over 2200 documents were reviewed and data from 13 documents were extracted.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOverall, there are many descriptions of how to implement and strengthen systems, ways to assess and improve data availability and quality, tools to improve the data use context, training in data use and mechanisms to involve stakeholders and strengthen infrastructure. However, there are massive gaps in relation to good use cases or examples of where routine health data is used in the development, monitoring and supervision of plans at district and facility level.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere appears to be no institutionalised obligation of planners to monitor plans, very little guidance on how to practically monitor programs and minimal discussion about how to use the routinely available data to supportively supervise the implementation of the plans.\u003c/p\u003e\n\u003cp\u003eTo overcome these shortcomings, we recommend that practical procedures to ensure linkage of existing district plans to regular monitoring of priority programs are institutionalised, that mechanisms for making managers institutionally accountable for monitoring and supervising these plans are put in place, and that practical guidelines for linking plans with RHIS data and regular monitoring and supportive supervision are developed.\u003c/p\u003e","manuscriptTitle":"Use of Routine Health Information Systems Data in Developing and Monitoring District and Facility Health Plans: A scoping review","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2023-05-03 14:52:57","doi":"10.21203/rs.3.rs-2565795/v2","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2023-06-19T04:41:21+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2023-06-07T19:12:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"22d18ae5-a67d-407f-83f5-62720a8021d2","date":"2023-05-31T11:24:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"3237e0cd-3958-4830-b8b1-71c95386ed55","date":"2023-05-27T21:42:17+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2023-05-27T14:58:37+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2023-04-30T13:30:27+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2023-04-28T18:39:05+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2023-04-28T18:36:11+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Health Services Research","date":"2023-04-04T09:55:51+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-health-services-research","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bhsr","sideBox":"Learn more about [BMC Health Services Research](http://bmchealthservres.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/BHSR/default.aspx","title":"BMC Health Services Research","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7aed668f-32aa-448e-99ee-41dd40e9e7f2","owner":[],"postedDate":"May 3rd, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2023-10-09T15:04:37+00:00","versionOfRecord":{"articleIdentity":"rs-2565795","link":"https://doi.org/10.1186/s12913-023-09914-6","journal":{"identity":"bmc-health-services-research","isVorOnly":false,"title":"BMC Health Services Research"},"publishedOn":"2023-10-02 15:02:06","publishedOnDateReadable":"October 2nd, 2023"},"versionCreatedAt":"2023-05-03 14:52:57","video":"","vorDoi":"10.1186/s12913-023-09914-6","vorDoiUrl":"https://doi.org/10.1186/s12913-023-09914-6","workflowStages":[]},"version":"v2","identity":"rs-2565795","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2565795","identity":"rs-2565795","version":["v2"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}