{"paper_id":"8eafcb28-a612-48a8-8d7d-5d1de4788e8b","body_text":"Title: Feasibility of an Adaptive E-Learning Environment to Improve Provider Proficiency in \nEssential and Sick Newborn Care in Mwanza, Tanzania \n \nShort title: Feasibility of Adaptive Essential and Sick Newborn Care  \n \nAuthors: Peter Meaney1, Adolfine Hokororo2,3, Hanston Ndosi2, Alex Dahlen1, Theopista Jacob3, \nJoseph R Mwanga2, Florence S Kalabamu3,8, Christine Joyce4, Rishi Mediratta,1 Boris Rozenfeld5, Marc \nBerg1,5, Zack Smith1, Neema Chami2,3, Namala P Mkopi3,9, Castory Mwanga3, Enock Diocles2, Ambrose \nAgweyu6,7 \n \nABSTRACT \nIntroduction. To improve healthcare provider knowledge of Tanzanian newborn care guidelines, we \ndeveloped adaptive Essential and Sick Newborn Care (aESNC), an adaptive e-learning environment \n(AEE). The objectives of this study were to 1) assess implementation success with use of in-person \nsupport and nudging strategy and 2) describe baseline provider knowledge and metacognition. \n \nMethods. 6-month observational study at 1 zonal hospital and 3 health centers in Mwanza, Tanzania. \nTo assess implementation success, we used the RE-AIM framework and to describe baseline provider \nknowledge and metacognition we used Howell’s conscious-competence model. Additionally, we \nexplored provider characteristics associated with initial learning completion or persistent activity. \n \nResults. aESNC reached 85% (195/231) of providers: 75 medical, 53 nursing, and 21 clinical officers; \n110 (56%) were at the zonal hospital and 85 (44%) at health centers. Median clinical experience was 4 \nyears [IQR 1,9] and 45 (23%) had previous in-service training for both newborn essential and sick \nnewborn care. Efficacy was 42% (SD±17%). Providers averaged 78% (SD±31%) completion of initial \nlearning and 7%(SD±11%) of refresher assignments. 130 (67%) providers had \n≥ 1 episode of inactivity \n>30 day, no episodes were due to lack of internet access. Baseline conscious-competence was 53% \n[IQR:38-63%], unconscious-incompetence 32% [IQR:23-42%], conscious-incompetence 7% [IQR:2-\n15%], and unconscious-competence 2% [IQR:0-3%]. Higher baseline conscious-competence (OR 31.6 \n[95%CI:5.8, 183.5) and being a nursing officer (aOR: 5.6 [95%CI:1.8, 18.1]), compared to medical \nofficer) were associated with initial learning completion or persistent activity. \n \nConclusion. aESNC reach was high in a population of frontline providers across diverse levels of care \nin Tanzania. Use of in-person support and nudging increased reach, initial learning, and refresher \nassignment completion, but refresher assignment completion remains low. Providers were often \nunaware of knowledge gaps, and lower baseline knowledge may decrease initial learning completion or \nactivity. Further study to identify barriers to adaptive e-learning normalization is needed. \n \nAffiliation: \n1Stanford University School of Medicine, Palo Alto, CA \n2Catholic University of Health and Allied Sciences, Mwanza, Tanzania \n3Pediatric Association of Tanzania, Dar Es Salaam, Tanzania \n4Cornell University School of Medicine, New York, New York USA \n5Area9 Lyceum, Boston, Massachusetts, USA \n6KEMRI-Wellcome Trust Research Programme, Kenya \n7London School of Hygiene and Tropical Medicine, London, UK \n8Hubert Kairuki Memorial University, Dar es Salaam, Tanzania \n9Muhimbili National Hospital, Dar es Salaam, Tanzania \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \nNOTE: This preprint reports new research that has not been certified by peer review and should not be used to guide clinical practice.\n\n \n \n \nKey questions. \n1) What is already known on this topic.  \nSummarize the state of scientific knowledge on this subject before you did your study and why this study needed to be done.  \n- In sub-Saharan Africa, gaps in care quality may contribute to its high neonatal mortality.  \n- Provider knowledge is a main driver of care quality, but current conventional in-service \neducation methods are inadequate in adaptivity, reach, effectiveness, and refresher \nassignments. \n- Hard copies of national guidelines have been disseminated to health facilities expectations are \nHCPs will learn and adhere to them. \n- Adaptive eLearning, a subdomain of e-learning, holds the potential to overcome limitations to in-\nservice medical education, but the optimal implementation strategy is unknown. \n2) What this study adds.  \nSummarize what we now know because of this study that we did not know before. \n- Baseline knowledge of essential and sick newborn care was low, mostly due to unconscious \nincompetence (providers thinking they were correct when they were incorrect). \n- Initial learning completion increased significantly with the use of an in-person program manager \nand an escalating nudging strategy, and technical issues were not identified as a significant \nlimitation to participation. \n \n3) How this study might affect research, practice, or policy.  \nSummarize the implications of this study. \n- Provider self-reporting may underestimate knowledge gaps as most gaps are not known by \nproviders.  \n- Adaptive e-learning may be a feasible and acceptable way to disseminate guideline and \nimprove quality of care if an implementation strategy can be identified to increase refresher \nassignment completion. \n- Once the ideal implementation strategy is identified, effectiveness of adaptive e-learning at \nscale can be evaluated. \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nINTRODUCTION \nThe Government of Tanzania is committed to reducing the neonatal mortality rate from 20 per 1,000 \nlive births to the SDG target of 12 per 1,000 live births by 2030.1 Addressing gaps in quality of essential \nand emergency care is a key strategy for achieving this ambitious target.2-5 In Mwanza, Tanzania, \ncorrect diagnosis estimates at health centers and district hospitals range from 61-87%, while correct \ntreatments are administered only 21-86% of the time.6,7 \n \nProvider knowledge is a main driver of care quality and several conventional (in-person) in-service \ntrainings for essential and sick newborn care have been attempted in Tanzania. These include 2\nnd \neditions of WHO’s Essential Newborn Care, and the American Academy of Pediatrics’ Helping Babies \nBreathe.8,9 They range in content scope (care of the newborn in 1st hour vs 1st 28 days of life), levels \nand mix of cognitive learning targets (i.e. Bloom’s Taxonomy of remembering, understanding, applying, \nanalyzing, evaluating, and creating)10 as well as duration of training (2-8 days). Both have \ndemonstrated significant effectiveness when they can be sustained.11-13 \n \nUnfortunately, conventional in-service education methods are often inadequate in coverage and difficult \nto sustain.\n2,14,15 Conventional education methods do not systematically adapt to individual providers’ \nknowledge or convenience,16-19 have time-limited education and target minimal competency, which \nlimits education effectiveness.18,20-24 Our systematic review highlighted that current educational content \nand educational design often have limited adaptability to facility needs, which also decreases education \neffectiveness.23 The limited effectiveness and accuracy of current educational methods widens the \n“know-do” gap, and this gap is greater in rural, under-resourced areas where in-service education is \nlimited.25 A key research gap of the World Health Organization is to identify effective provider education \nthat extends across health systems.26,27 \n \nAdaptive eLearning, a subdomain of e-learning, pulls from computer science and artificial intelligence \nprinciples to create a cognitive model to adapt education to each provider.\n28 Usage data includes \nmetacognition categorized using Howell’s conscious-competence model: 1) conscious competence \n(correct and confident), 2) unconscious incompetence (confident in knowledge but incorrect), 3) \nconscious incompetence (not confident in knowledge), and 4) unconscious competence (not confident \nin knowledge but correct).\n29 Usage data is processed to create a cognitive model for each student and \nadjusts the sequencing of content and ratio of learning resources based on the formative assessment. \nKnowledge acquisition during initial learning has been both higher and faster compared to conventional \neducation. \n28,30 \n \nIn addition to optimizing initial learning, adaptive eLearning can address forgetfulness through \ngenerating refresher learning assignments. Forgetfulness, first described by Ebbinghaus, is an \nexponential decay of knowledge with knowledge returning to baseline days or weeks after initial \nlearning.\n31,32 We have seen this decay in our previous work in LMICs with pediatric acute care \nknowledge and CPR skills.18,21 Forgetfulness can be addressed with refresher assignments, spacing \nlearning over time.33-35 Subsequent refresher assignments start at a higher baseline than previous, \ntaking less time to achieve mastery. Over time, yields a more significant percentage of data \nremembered. In high-income settings, we have demonstrated that refresher assignment completion \nimproves pediatric resuscitation skills and patient outcomes.\n36-38 In Tanzania, the use of refresher \nassignments over time is a key implementation strategy of Helping Babies Breathe, which has \nsignificantly increased adherence to guidelines and reduced neonatal deaths by almost 50%.13  \n \nAdaptive e-learning has the potential to be rapidly scalable with existing infrastructure: it does not \nrequire significant dedicated resources to implement and maintain and may close the training gap that \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nexists for rural, under-resourced areas and allow increased dissemination of up-to date evidence based \nguidelines and reduce the knowledge deficit, decreasing the need for face-to-face education when \ninstructors are limited.39-41 While adaptive e-learning is as effective as conventional education when \nexamining patient outcomes for dyslipidemia screening, monitoring of diabetes, drug dose calculation, \nand pressure ulcer classification,42-45 there is limited evidence examining newborn and pediatric acute \ncare.46,47 Further, the optimal implementation strategy of adaptive e-learning for in-service provider \neducation in low- and middle-income countries (LMICs) is unknown.45,48 \n \nOur program, Pediatric Acute Care Education (PACE), is an adaptive e-learning environment co-\ndeveloped with the Pediatric Association of Tanzania (PAT), Catholic University of Health and Allied \nSciences, Stanford University and Area9 Lyceum to improve facility-based adherence to national \nnewborn and pediatric guidelines.(Meaney, digital health 2023) PACE is designed for all cadres \n(provider types) that may be responsible for caring for newborns and sick children. Initially piloted in \n2019, PACE has expanded in size to meet the training needs identified by PAT and PACE providers. \n \nOur initial pilot demonstrated a 30% change in conscious competence during initial learning from 66 \n(57-75%) to 94% (92-98%).\n49 The average initial learning completion was only 37% and refresher \nassignment completion was not assessed. There were 3 barriers identified to initial learning completion: \n1) use of pre-post assessments for efficacy, 2) lack of in-person technical support, and 3) ineffective \nemail nudging strategy.\n49 Based on these results we revised our implementation strategy in 3 ways: 1) \nuse of change in conscious competence from baseline for efficacy, 2) a full time in-person program \nmanager to provide support PACE providers, and 3) incorporation of an escalating nudge strategy that \nincluded emails, WhatsApp messaging and in-person support. \n \nWe applied our content development methodology to Tanzania’s essential newborn and sick newborn \ncare guidelines to develop our adaptive Essential Newborn and Sick Care (aESNC) and deployed \nmodules within PACE in May 2022.\n50 aESNC currently contains 2 assignments with total of 9 modules: \nPreparing for delivery, 1st hour of life, neonatal resuscitation, introduction to sick newborn care, birth \nasphyxia + pain, convulsions and meningitis, glucose and electrolytes, hemorrhage and jaundice, \npneumonia, sepsis, and shock. Learning objectives are restricted to Bloom’s Taxonomy levels 1 and 2 \n(remembering, understanding), and 100% conscious competence is required to complete the module. \nProviders completing all PACE modules are awarded continuing professional development credit \nthrough PAT toward maintaining professional certification. \n \nThe objectives of this study are to assess the implementation success of adaptive Essential and Sick \nNewborn Care using the revised implementation strategy and describe baseline provider knowledge \nand metacognition in Mwanza, Tanzania. We evaluated implementation success using the RE-AIM \n(Reach, Efficacy, Adoption, Implementation and Maintenance) framework adapted by Soicher et al for \nthe implementation of education interventions for higher education and describe provider knowledge \nand metacognition using Howell’s consciousness competence model.\n29,51,52 \n \nMETHODS  \nStudy Design. This was a prospective single-arm, multi-center pilot implementation study conducted in \nMwanza, Tanzania from May 2022 to January 2023. This manuscript was formatted in accordance with \nSTROBE guidelines.\n53 \n \nSetting. Study sites included all sites currently participating in PACE that had at least 1 provider in the \nstudy cohort. Facility characteristics are listed in Table 1. Due to limited study personnel and this being \nthe initial study, PACE was initially deployed at the zonal hospital in May 2022, and then extended to \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nhealth centers within Mwanza Region that refer to the zonal hospital in stepwise fashion. Duration of \nfacility participation at time of data extraction is listed in Table 1. \n \nParticipants. The cohort consisted of a convenience sample of consenting healthcare providers who \nhad participated in PACE >30 days. Providers were identified by the facility medical officer in-charge, \nfacility head matron/patron or head of department as well as during sensitization meetings at morning \nreport and recruited to participate by the PACE program manager. Informed electronic consent was \nobtained through REDCap from all providers who participated in PACE.54 Eligible PACE providers are \nfacility-based healthcare providers responsible for providing clinical care to newborns, infants, and/or \nchildren. In Tanzania, training duration varies by cadre: medical officers require 5 years of training and \n1 year of internship; advanced degree nursing 3 years and 1 year internship; nurses and clinical officers \n3 years of training, and assistant medical officers 2 years of training. We excluded providers who \ndeclined to consent or who withdrew from study; these numbers are given in the CONSORT diagram, \nFig. 1. \n \nImplementation Strategy. aESNC was implemented through the Pediatric Acute Care Education \n(PACE) program. PACE is an adaptive e-learning environment with locally derived content, a steering \ncommittee, a program manager to provide in-person technical support and series of motivators to \nincrease completion. It is designed to increase provider proficiency in neonatal and pediatric evidence-\nbased guidelines in Tanzania and has been previously described. (Meaney, digital health 2023)  \n \nAdaptive e-learning with locally derived content\n. aESNC, consists of 2 assignments: Essential Newborn \nCare (3 modules, 47 learning objectives) and Sick Newborn Care (6 modules, 91 learning objectives). \naESNC content was collaboratively developed with subject matter experts and content designers using \nTanzania’s national “Guidelines for Neonatal Care and Establishment of Neonatal Care Unit.\n50 Content \ncreation was supervised by subject matter experts (SME) from the PAT’s Continuing Professional \nDevelopment Committee (NM, NC, CM), the Tanzanian Society for Pediatric Nursing (ED), and Area9 \nSenior Learning Architects (BR, MB).  \n \nPACE Steering committee\n. The Steering Committee (AH, PAM, AA, HN) provides oversight and \ncoordination of PACE management, research administration, publications and data sharing, and \nintegration of all resources needed for the project. The chair of the steering committee is responsible for \ncommunication among committee members, including meeting schedules and agendas, and rotates \namong the members on a yearly basis. The PACE steering committee consisted of experts in newborn \nand pediatric care, provider education, and implementation research. \n \nThe Program Manager.\n The program manager is a medical (MD) or nursing (RN) officer with \nexperience working in the Tanzanian health system, human subjects research training, effective \ncommunication skills and, either formal or informal health education and/or IT skills. The program \nmanager conducts sensitization meetings at facilities and generates contact lists. These lists are \nreviewed and augmented by the medical officer in charge, head of department and/or chief nursing \nmedical officer as appropriate, and current PACE providers. The program manager meets with \nconsented providers in-person individually to set up PACE on their mobile device, ensure proper \nfunctioning and provide initial data bundle. \n \nMotivational strategies:\n \n• Nudges. Weekly, providers who hadn’t completed all PACE initial learning assignments were given \nreminders or “nudges” to complete their learning. Our escalating nudge strategy was as follows: No \nactivity for 2 days: auto email reminder from Rhapsode; > 5 days: WhatsApp using standardized \nstatements; > 30 days: a virtual or face-to-face meeting with program manager. \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \n• Internet support. Providers were reimbursed up to 10gb of data (~4USD) monthly on their mobile \ncarrier during the study period.  \n• Maintaining certification. Continuing Professional Development credit was awarded via Pediatric \nAssociation of Tanzania for completion of all PACE initial learning assignments. \n• Passive feedback to health system leadership. Program manager would send PACE Facility \nProgress Report (PDF) to facility stakeholders weekly via email. Stakeholders to this process may \ninclude hospital administration, regional and council health management teams, and the Pediatric \nAssociation of Tanzania leadership. Reports included aggregate activity, metacognition, and \nmedian time to proficiency at a facility level, but not does not report proficiency or metacognition at \nan individual level. \n \nHypothesis. Implementation of aESNC with addition of in-person coordinator and nudging strategy to \nour implementation strategy would increase reach and improve average completion of both initial \nlearning and refresher assignments compared to our initial PACE pilot. \n \nVariables. \nImplementation Outcomes. We used the established RE-AIM for the educational intervention \nimplementation framework to define our feasibility outcomes of Reach, Efficacy, and \nImplementation.52,55 Adoption and Maintenance were not assessed. We assessed Reach using \nindividual participating providers as a proportion of providers identified as eligible.  \n \nEfficacy was assessed using change in provider conscious competence (current – baseline). Baseline \nprovider proficiency was determined using providers initial responses to knowledge probes, and current \nprovider proficiency by last response to knowledge probes. Responses were categorized using \nHowell’s conscious-competence model: 1) conscious competence (correct and confident), 2) \nunconscious incompetence (confident in knowledge but incorrect), 3)bconscious incompetence (not \nconfident in knowledge), and 4) unconscious competence (not confident in knowledge but correct).\n29  \n \nWe used 6 metrics to assess implementation: persistent activity, average progress of initial learning \ncompletion, average progress of refresher assignment completion, time to enrollment, nudging \nutilization, and loss to follow-up. Persistent activity was defined as actively using PACE within last 2 \nweeks of the study (Jan 2023). Average progress of initial learning was percentage of achieving 100% \nconscious competence of all aESNC learning objectives. Average progress of refresher assignments \nwas percentage of achieving 100% of all content assigned by Rhapsode for refresher learning. Time to \nenrollment was days from consent to enrollment interview. Nudging utilization included % of providers \nwith at least one nudge, median number of nudges and distribution of nudge types. Loss to follow up \nwas defined as those who were inactive for > 30 days and were not able to be contacted. \n \nPredictors, Potential confounders, and Effect modifiers. Several variables were collected to describe \nthe cohort. These included facility, cadre (profession), years of clinical experience, previous newborn \ntraining, job satisfaction, motivation, and baseline knowledge and metacognition. Facilities were defined \nby government designation as a zonal hospital or health center. Cadre categories include specialists, \nmedical officers, assistant medical officers, clinical officers, assistant clinical officers, nursing officers, \nassistant nurse officers, nurse midwives, medical attendants, laboratory scientist/technologists, and \nhealth assistants. The continuous variable “years clinical experience” was collapsed to a categorical \nvariable of <= 1 year, 2-3 years, 4-9 years, and 10+ years based on quartiles rather than visual \ninspection given small numbers. Previous newborn training was defined as ever having taken either \nEssentials of Newborn Care, Helping Babies Breathe, both or neither. Job satisfaction and motivation \nwere measured using previous questionnaires validated for healthcare providers in Ghana by \nBonenberger et al.\n56  \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \n \nData sources/measurement. At enrollment, providers completed electronic surveys regarding \ndemographics, previous clinical training, job satisfaction and motivation via REDCap. Provider \nresponse data including knowledge competence, metacognition, average progress on initial learning or \nrefresher assignments was collected within the Area9 Rhapsode platform. Data from Rhapsode and \nREDCap was extracted on January 14\nth, 2023, linked with study ID numbers by name and deidentified \nprior to analysis. \n \nBias. The cohort was identified through sensitization meetings at facilities by the program manager, as \nwell as by facility administrators using staff rosters and study providers and at same points of care over \nthe same time frame for each facility. Outcome data utilized provider usage data from Rhapsode. In \naddition, enrollment interview and reminders were tracked in the study database. These modules were \nnewly developed, and no study provider had piloted these modules before the study. In addition, \nbaseline competencies were obtained as part of the exposure, allowing to account for an effect of \nprevious training. Our multivariable analyses on factors associated with new provider retention adjusted \nfor plausible prognostic factors. Providers completed all surveys and outcome data electronically using \ntheir mobile phones. Potential data entry errors were followed up by the program manager with the \nprovider for clarification where needed and logged within the database. The outcome assessments \nblinded to the study team and were linked by name and email addresses to the study database and \ndeidentified prior to analysis. \n \nStudy size. As this was a feasibility study, no power calculation was done. Study size was determined \nby number of providers enrolled during the study period. Based on an estimate of 20% of providers at \nthe zonal hospital and 40% of health center providers caring for newborns and sick children our initial \nenrollment target was 50 medical officers, 30 clinical officers, and 50 nursing officers over 6 months. \n \nStatistical methods. Descriptive, univariate, and multivariate analyses were performed. Summary \nresults are presented as means and standard deviations for normally distributed variables and medians \nwith interquartile ranges for variables that were not normally distributed. For our multivariable logistic \nregression exploring completion of initial learning or persistent activity, we selected provider external \ncharacteristics (facility, cadre, experience, previous training) a-priori and did not incorporate response \ndata (job engagement, motivation, or knowledge proficiency data) given the limited size to the cohort. \nSignificance was set at p\n≤ 0.05 and models were fit using Python version 3.8.5.  \n \nRESULTS: \nReach. All 4 facilities invited to participate adopted PACE. Of 246 providers, 90% (221) were eligible, \nwith 17 of 25 ineligible due to not completing screening survey. Consent rate of eligible providers was \n90% (n=195) with “no time to participate” being the greatest reason for declining participation (Figure \n1).  \n \nProvider Characteristics. 110 (56%) of providers in this cohort were based at the zonal hospital, \nfollowed by HC#2 at 42 (22%), HC #1 at 27 (14%) and #3 at 16 (8%). Providers were most commonly \nmedical officers (75, 39%), nursing officers (53, 27%) and clinical officers (21, 11%). Median years of \nclinical experience was 4 [IQR 1-9]. The reported prevalence of previous newborn training of Essential \nNewborn Care, Helping Babies Breathe, and both ENC and HBB were 23% was 31%, 41%, and 23%, \nrespectively. Overall job satisfaction was 3.6/5 (SD 0.8) with highest sub scores of morale 4.1/5 (SD \n0.8) and supervision 3.8/5 (SD 1.0) and lowest sub scores of renumeration 3.9 (SD 1.2), followed by in-\nservice training 3.5/5 (SD 1.2), management 3.5/5 (SD 1.0) and work environment at 3.5/5 (SD 1.0). \nOverall motivation scores were 3.7/5 (SD 0.6).  \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nEfficacy. For providers who completed initial learning (n= 110), average conscious competence \nincreased 42 ± 1 percentage points, while average unconscious incompetence decreased 31 ± 1 \npercentage points. Of the efficacy, the improvement of conscious competence, 76% was due to \nreduction in unconscious incompetence. Average conscious incompetence decreased by 10±1 \npercentage points, while provider unconscious competence was extremely low both before and after \nPACE. (Supplementary Data A1-3).  \n \nImplementation. The average progress of initial learning for all providers was 78% (SD = 31%) \n(Supplementary Data C1). 56% (110) completing all initial learning, 13% (25) incomplete but active \nwithin 2 weeks of the end of the study, and 31% (60) becoming inactive during the study period. \nProviders who completed the initial learning took on average 4.5 [IQR 2.4, 6.1] hours (Supplementary \nData B1-3). Inactive providers dropped out after on average 1.9 [IQR 0.8, 3.2] hours, and the active \nproviders who are still finishing initial learning spent 4.0 [IQR 2.3, 5.4] hours. Median duration to \ncomplete all modules was 61 days [IQR 34,111]. Reduction in probability of provider remaining \ncompleting initial learning or remaining active appeared to be evenly distributed over % completion and \ndays of PACE participation (Supplementary Data C1 and B4).  \n \nAverage progress of refresher assignments was 7%, with none completing all refresher assignments, \n27% (37) incomplete but active in PACE within last 2 weeks of the study period, and 73% (98) \nbecoming inactive during the study period. Average refresher progress appears to be heavily skewed to \nlower completion with median completion 2% [IQR:0%, 8%]. (Supplementary Data C2). Total hours \nspent using modules was 0.3±0.7 for active providers and 1.0±1.3 for inactive providers. \n \nDays from consent to enrollment interview was 2 [IQR:0, 4]. 67% (130) of providers needed at least 1 \nnudge and 30% (59) needed at least one follow-up with the program manager. A total of 1,011 nudges \nwere sent: 311 (31%) email nudges (2-5 days inactivity), 564 (56%) WhatsApp nudges (5-30 days \ninactivity), and 136 (13%) nudges by the program manager (> 30 days inactivity) (Supplementary Data \nD1-2). The program manager conducted 104 nudges by phone and 30 in-person. Most frequent \nreasons reported for >30 days inactivity was “no time” (93, 69%), followed by “forgot to use” (68, 50%). \nTechnical barriers (phone not working, can’t access PACE, not sure how to use PACE) represented \nonly 15 (11%) and no provider reported inactivity due to lack of mobile data connectivity. No providers \nwished to terminate from study when interviewed with program manager at 30 days of inactivity and \nnone were lost to follow-up. \n \nBaseline knowledge and metacognition of Essential and Sick Newborn Care. Overall median \nbaseline conscious competence for aESNC was 53% [IQR:38-63%], with unconscious incompetence \n33%[IQR:25-45%] and 32%[IQR:23-42%] respectively (Table 3). aESNC had conscious incompetence \nof 7% [IQR:2-15%], and unconscious competence 2% [IQR:0-3%].  \n \nFor individual aESNC modules, conscious competence was highest in Neonatal Resuscitation (63% \n[IQR:34-75%]), 1\nst Hour of Life (58% [IQR:33-75%]), and Glucose and Electrolytes (57% [IQR:29-\n71%]), and lowest in Birth Asphyxia and Pain (38% [IQR:0-54%]), Preparing for Delivery (38% [IQR:25-\n56%]), and Pneumonia, Sepsis and Shock (43% [IQR:14-71%]). Unconscious incompetence was \nhighest in Preparing for Delivery (38% [IQR:25-56%]), Introduction to the Sick Newborn (33% [IQR:22-\n50%]), and Convulsions and Meningitis (33% [IQR:17-47%]), and lowest in 1\nst Hour of Life (17% \n[IQR:8-33%]), Hemorrhage and Jaundice (20% [IQR:0-33%]), Glucose and Electrolytes (21% [IQR:7-\n36%], and Neonatal Resuscitation (21% [IQR:5-32%]). Conscious incompetence was highest in Birth \nAsphyxia and Pain (8% [IQR:0-19%]), Hemorrhage and Jaundice (7% [IQR:0-27%]) and Preparing for \nDelivery (6% [IQR:0-13%), and zero in all others ([IQR:0-13%]). Unconscious competence was zero in \nall modules ([IQR:0-8%]). \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \n \nFactors associated with initial learning completion or persistent activity. On univariate analysis, \nbeing a nursing officer, 2-3 years of experience, and having higher baseline conscious competence \nwas associated with initial learning completion or persistent activity. On multivariate modeling, nursing \nofficers, clinical officers and “other” personnel were significantly associated with better initial learning \ncompletion or persistent activity when compared to medical officers adjusting for facility, clinical \nexperience, and previous newborn training (Table 4). Facility, previous training with either Essential \nNewborn Care or Helping Babies Breathe were not associated with initial learning completion or \npersistent activity. \n \nDiscussion. This study demonstrated that adaptive eLearning can increase reach to providers not \nreached by conventional training, and the use of in-person support and an escalating nudging strategy \nincreased initial learning completion. Further, refresher assignment completion was low, and our \nimplementation strategy needs to be revised to solidify knowledge gained through initial learning. \nFinally, provider awareness of knowledge gaps was low at baseline and lower baseline knowledge is \nassociated with decreased completion or learning activity. \n \naESNC reached 85% of providers in this study. The critical mass needed for collective behavior change \nfor providers in facilities is unknown, but using estimates from current social behavioral science suggest \nthat 10-40% of providers are needed to be engaged to drive behavior change.\n57-59 As knowledge is only \none component that drives behavior change and we know not all providers would complete all modules, \nwe will examine thresholds for reach required to achieve sustained changes in quality of care.  \n \naESNC was able to reach providers that conventional newborn training had not reached. We found that \n30% of providers had received Essential Newborn Care training, 40% had received Helping Babies \nBreathe, but only 23% had received both. The impact of this limited reach may be underestimated if this \nreported training was years ago. A reason for conventional education’s limited reach may be due to \nhigh turnover of front-line providers at academic centers, which may be able sustain training but not \nsufficient to address its training gaps. Furthermore, community health centers may have limited support \nto hold local training or send staff for training to meet this need. This conclusion from this initial study \nshould be tempered as our observed reach, as well as our enrollment rates, were higher than \nanticipated, and don’t correlate with staffing estimates reported from health system leadership (Table \n1). This may be due to a combination of incomplete lists by facility leadership and early adopter \nphenomenon. Future studies will bear this out. \n \nOur initial efficacy of 41% was consistent with our previous conventional studies that demonstrated \nknowledge efficacy rates of 10-25% after pediatric acute care training using current training methods\n18-\n21, as well as other newborn and pediatric conventional in-service education studies.5,22 It is important to \nnote that our use of formative assessments at baseline and during the educational session, are \nrelatively new measures of educational efficacy. In our pilot, the difference between latest conscious \ncompetence and baseline conscious competence was roughly 10 points higher than between pre-post \ntesting.49 \n \nThe average progress of providers for initial learning was 78%, and 56% of providers completed all \ninitial learning assignments, higher than we observed in our initial pilot of PACE. This may be due to the \nuse of in-person support and an escalating nudging strategy increased initial learning completion. The \ncompletion rate in this study is comparable to previous studies in high income countries: much higher \nthan massive open online courses progress rates of 7-15%,\n60,61 and comparable to longitudinal e-\nlearning studies of healthcare providers (44-50%).62,63 A few shorter courses (< 3 months) with more in-\n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nperson support report completion rates as high as 75-80% without incentives,64,65 and as high at 70-\n96% completion when incentive of $200- $750 was provided.66,67 \n \nIn this study, we observed more time is needed for providers to complete initial learning than \nanticipated. We had estimated providers needing 5 hours over 1 month to complete initial learning. \nWhile median time to complete initial learning was shorter than what we targeted (4.5 hours), the \nmedian days to completion of initial learning was 61 days, twice what we had estimated. Interestingly, \nthe retention curve demonstrated no significant drop-off that had been seen previously. Inadequate \ntime for education, not technical issues or lack of internet access was the main barrier.  \n \nOther reasons, besides the in-person support and escalating nudging strategy, may have contributed to \nthese findings. We did not use a designated order determined by subject matter experts as we did in \nour initial pilot, as we observed a significant drop off with disability and exposure assessment. This \nchoice of module completion may have allowed better alignment of provider learning interest and \nincreased engagement during initial usage, and familiarity with adaptive learning may have facilitated \nusage of completion of additional content. Additionally, there may have been increased peer support for \nadaptive learning due to a higher percentage of providers participating in PACE at each facility. \nWhether this improved average completion is due to our implementation strategy components \n(escalating nudging strategy, program manager, reports to facility), changes to PACE itself (self-\nselection of topics), or environmental (unmeasured peer support) is unclear. \n \nOur implementation strategy needs further revision to increase refresher assignment completion. Less \nthan a third of participants participated in refresher assignments, and the average progress through \nrefresher assignments was <10%. Improving established provider retention and completion of refresher \nassignments is key to optimizing education effectiveness. In-service provider education is designed to \nbuild upon foundational knowledge gained in pre-professional training, as well as solidify it through \nrelevance to a provider’s clinical experience. Unfortunately, many continuing professional education \nprograms are short courses without longitudinal follow-up. The use of in-person skills training, tailored \nto provider needs using peers as “clinical champions”, may enhance refresher assignment completion. \nThese clinical champions may add a point of engagement and facilitate normalization of adaptive \neLearning use to facilitate education over time. Additionally, as our nudging strategy was triggered only \nto complete all initial learning assignments, providers may believe they have no more learning to do. \nFurther refinements with our nudging strategy and sensitization meetings with facility providers and \nleadership to emphasize importance of refresher training may also help normalize adaptive e-learning \nfor providers. \n \nAdverse events occur in 18% of inpatient admission in Africa and continuous learning is needed to \nimprove patient safety.\n68,69 Although baseline knowledge of guidelines was consistent with previous \nstudies, we identified that providers are often unaware of knowledge gaps. In addition to demonstrating \nthat providers know about half of newborn knowledge at baseline, adaptive e-learning allows us to see \nproviders’ awareness of their knowledge and misconceptions. Unconscious incompetence is a latent \nthreat to health systems, and as they are not realized, may mask workforce knowledge gaps. \nUnconscious incompetence is a particular danger clinically, as providers are confident but incorrect in \nwhat care they believe should be delivered.  \n \nWhile aESNC increased provider knowledge, our exploratory analysis identified that baseline lower \nknowledge may be associated with inactivity or non-completion of aESNC. While one of the strengths \nof adaptive learning is using formative assessments to facilitate learning, providers whose baseline \nknowledge is too low may suffer cognitive overload. Our level of unconscious incompetence surpassed \nArea9's best practice guidelines, which suggest keeping it between 20 and 30%. It was also higher than \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nwhat we observed in our initial pilot. Development of a strategy of focused support to engage with those \nat risk may be needed. \n \nLimitations. Several limitations are important to note. First, job satisfaction and motivation has not \nbeen validated for turnover intention in Tanzania. Also, our study had limited ability to monitor \nimplementation metrics. Despite having solid relationships and regular meetings, we faced significant \nchallenges in obtaining complete lists of trained providers. This issue could have non-randomly limited \nour ability to reach less motivated or disengaged providers. Finally, our univariate and multivariable \nanalyses were limited by numbers and may have failed to detect important provider characteristics that \nimpact provider retention. \n \nFuture directions. All the metrics generated need to be validated against clinically meaningful \noutcomes across various contexts. Adaptive e-learning may allow us to better understand the how, \nwhen why and where providers learn, as well as how that knowledge is translated into behavior change \nand improved patient outcomes. There is also a need for user input in defining what is essential to \noptimize effectiveness when fed back to providers and healthcare managers for quality improvement. A \nbetter understanding of how providers learn will guide the development of education implementation \nstrategies that are not only effective, but scalable and sustainable. Future studies should improve the \nidentification of all eligible providers, monitor the use of data bundles, fidelity of escalating nudge \nstrategy to individual providers, aggregate reports to facility leadership, and distribution of CPD points. \n \nThe immediate next steps are to conduct qualitative research to identify barriers and facilitators to \nadaptive eLearning and refine our implementation strategy to include tailored skills training and local \ncapacity to develop new content continuously. In addition, we will refine our clinical auditing \nmethodology to validate learning metrics on clinically meaningful outcomes.  \n \nConclusion. aESNC reached many providers without previous conventional in-service training in both \nsick and essential newborn care. Use of in-person support and motivators increased initial learning \ncompletion, but refresher assignment completion was low. Providers were often unaware of knowledge \ngaps, and lower baseline knowledge may be associated with non-completion of initial learning. Further \nstudy examining tailored skills trainings, clinical champions, continuous content development, as well \nas qualitative studies to identify barriers and facilitators to normalizing adaptive e-learning in provider \nbehavior is needed. \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nAcknowledgements. The authors would like to thank the Pediatric Association of Tanzania; the \nTanzanian Ministry Of Health, Regional and Council Health Management Teams for participating in \nstakeholder meetings, Géraldine Jossellin-Duval for her leadership and mentorship of the PACE \nLearning Engineering Team; Denis Albert and Agnes Hassan for their organizing of participants and \nsite work in Mwanza; Michael Alfonzo, Jose “Jojo” Ferrer, Segolame Setlhare, CLN mentors and \nfacilitators for their valuable feedback on the initial versions of aESNC and its refinements. \n \nSAGER guidelines for sex and gender reporting were followed. \n \nConsensus statement on measures to promote equitable authorship. \n \nContributors: Substantial contributions by author: \nAuthors Study \nConception/design\n \nData \nAcquisition Analysis Interpretation \nPeter A Meaney Yes Yes Yes Yes \nAdolfine Hokororo Yes Yes Yes Yes \nHanston Ndosi Yes Yes Yes Yes \nAlex Dahlen Yes No Yes Yes \nTheopista Jacob Yes No No Yes \nJoseph R Mwanga No No No Yes \nFlorence S Mukalaba No No No Yes \nChristine Joyce No No No Yes \nRishi Mediratta No No No Yes \nBoris Rozenfeld Yes No No Yes \nMarc Berg Yes No No Yes \nZack Smith Yes No No Yes \nNeema Chami No No No Yes \nNamala P Mkopi No No No Yes \nCastory Mwanga No No No Yes \nEnock Diocles No No No Yes \nAmbrose Agweyu Yes No Yes Yes \n     \nAll Authors drafted the work or revised it critically for important intellectual content; AND approved the \nfinal version to be published; AND agree to be accountable for all aspects of the work in ensuring that \nquestions related to the accuracy or integrity of any part of the work are appropriately investigated and \nresolved. \nCompeting Interests. BR and MB are compensated by Area 9 Lyceum as Senior Learning Architect \nand Medical Director, respectively.  \n \nREDCap Database. Study data were collected and managed using REDCap electronic data capture \ntools hosted at Stanford University.\n70,71 REDCap (Research Electronic Data Capture) is a secure, web-\nbased software platform designed to support data capture for research studies, providing 1) an intuitive \ninterface for validated data capture; 2) audit trails for tracking data manipulation and export procedures; \n3) automated export procedures for seamless data downloads to common statistical packages; and 4) \nprocedures for data integration and interoperability with external sources. The Stanford REDCap \nplatform (http://redcap.stanford.edu) is developed and operated by Stanford Medicine Research IT \nteam. The REDCap platform services at Stanford are subsidized by a) the Stanford School of Medicine \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nResearch Office, and b) the National Center for Research Resources and the National Center for \nAdvancing Translational Sciences, National Institutes of Health, through grant UL1 TR001085. Area9 \nRhapsode™ meets the requirements for full GDPR compliance including encryption, data security, and \n'forget me'. \n \nIRB language. The Institutional Review Board of the Tanzania National Institute of Medical Research \n(NIMR/HO/R.8a/Vol.IX/3990), Stanford University (60379), the ethics committee of the Catholic \nUniversity of Health and Allied Science (no ID number given), and the Mwanza Regional Medical \nOfficer (Ref. No. AG.52/290/01A/115) approved the study protocol including consent procedures.  Data \ncollection procedures were completed in compliance with the guidelines of the Health Insurance \nPortability and Accountability Act (HIPAA) to ensure subject confidentiality. Informed electronic consent \nwas obtained through REDCap from all providers who participated in PACE.\n54 All providers who \ncompleted consent were included. All surveys and questionnaires were entered directly by providers \ninto REDCap. This study is reported according to the Consolidated Standards of Reporting Trials \n(CONSORT) 2010 extension to randomized pilot and feasibility trials.  \n \nPatient and public involvement. This research was done without patient involvement. Patients were \nnot invited to comment on the study design and were not consulted to develop patient-relevant \noutcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this \ndocument for readability or accuracy. \n \nRole of the funding source \n1. This study was funded by the Laerdal Foundation for Acute Medicine, Stanford University School of \nMedicine Maternal and Child Health Research Institute, Stanford Center for Innovation in Global \nHealth, and the Stanford University School of Medicine Division of Pediatric Critical Care Medicine. \n2. Funding sources had no role in project design, data collection, analysis, or interpretation; reporting, \nor the decision to submit results for publication. \n3. Stanford CTSA award number UL1 TR001085 from NIH/NCRR. \n \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n 1 4\nReferences: \n \n1. United Republic of Tanzania (TZA) - Demographics, Health & Infant Mortality. \nhttps://data.unicef.org/country/tza/\n \n2. Di Giorgio L, Evans DK, Lindelow M, et al . Analysis of clinical knowledge, absenteeism and \navailability of resources for maternal and child health: a cross-sectional quality of care study in 10 \nAfrican countries. BMJ Glob Health. Dec 2020;5(12)doi:10.1136/bmjgh-2020-003377 \n3. Lewis TP, Roder-DeWan S, Malata A, Ndiaye Y, Kruk ME. Clinical performance among \nrecent graduates in nine low- and middle-income countries. Trop Med Int Health. May \n2019;24(5):620-635. doi:10.1111/tmi.13224 \n4. Kruk ME, Gage AD, Joseph NT, Danaei G, García-Saisó S, Salomon JA. Mortality due to \nlow-quality health systems in the universal health coverage era: a systematic analysis of amenable \ndeaths in 137 countries. The Lancet. 2018/09/05/ 2018;doi:https://doi.org/10.1016/S0140-\n6736(18)31668-4 \n5. Rowe AK, Rowe SY, Peters DH, Holloway KA, Ross-Degnan D. The effectiveness of \ntraining strategies to improve healthcare provider practices in low-income and middle-income \ncountries. BMJ Glob Health. Jan 2021;6(1)doi:10.1136/bmjgh-2020-003229 \n6. Agweyu A, Hokororo A, meaney PA, Joyce CL, Setlhare S, Ferrer J. Helping Children \nSurvive (HCS) Quarterly Report. 2019:19.  \n7. Jacob T, Hokororo A, Albert D, et al. Tanzanian Clinical Learning Network: Preliminary \nData Analysis. Powerpoint. 2021:1-26. December 9, 2021.  \n8. Organization WH. Essential Newborn Care Course. Accessed November 4, 2022. \nhttps://www.who.int/publications/i/item/essential-newborn-care-course\n \n9. Pediatrics AAo. Helping Babies Breathe. Accessed November 4, 2022. \nhttps://www.aap.org/en/aap-global/helping-babies-survive/our-programs/helping-babies-breathe/ \n10. Krathwohl DR. A Revision of Bloom's Taxonomy: An Overview. Theory Into Practice. \n2002;41(4):212-218. doi:10.1207/s15430421tip4104_2 \n11. Manasyan A, Chomba E, McClure EM, Wright LL, Krzywanski S, Carlo WA. Cost-\neffectiveness of Essential Newborn Care Training in Urban First-Level Facilities. Pediatrics. \n2011;127(5):e1176-e1181. doi:10.1542/peds.2010-2158 \n12. Chaudhury S, Arlington L, Brenan S, et al. Cost  analysis of large-scale implementation of \nthe ‘Helping Babies Breathe’ newborn resuscitation-training program in Tanzania. BMC Health \nServices Research. 2016/12/01 2016;16(1):681. doi:10.1186/s12913-016-1924-2 \n13. Msemo G, Massawe A, Mmbando D, et al. Newborn mortality and fresh stillbirth rates in \nTanzania after helping babies breathe training. Pediatrics. Feb 2013;131(2):e353-60. \ndoi:10.1542/peds.2012-1795 \n14. Organisation for Economic Co-operation and Development, World Health Organization, \nWorld Bank Group. Delivering quality health services : a global imperative for universal health \ncoverage. 2018:1 online resource (96 pages). \n15. National Academies of Sciences E, and Medicine. Crossing the Global Quality Chasm: \nImproving Health Care Worldwide. 2018. The National Academies Collection: Reports funded by \nNational Institutes of Health. \n16. Bloom BS. The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective \nas One-to-One Tutoring. Educational Researcher. 1984;13(6):4-16. doi:10.2307/1175554 \n17. Rowe SY, Peters DH, Holloway KA, Chalker J, Ross-Degnan D, Rowe AK. A systematic \nreview of the effectiveness of strategies to improve health care provider performance in low- and \nmiddle-income countries: Methods and descriptive results. PLoS One. 2019;14(5):e0217617. \ndoi:10.1371/journal.pone.0217617 \n18. Meaney PA, Joyce CL, Setlhare S, et al. Knowledge acquisition and retention following \nSaving Children's Lives course for healthcare providers in Botswana: a longitudinal cohort study. \nBMJ Open. Aug 15 2019;9(8):e029575. doi:10.1136/bmjopen-2019-029575 \n19. Cox M, Afonso N, Mazhani L, et al. Knowledge Accrual Following Participation in Pediatric \nFundamental Critical Care Support Course in Gaborone, Botswana. Pediatric critical care medicine \n: a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive \nand Critical Care Societies. Aug 2018;19(8):e417-e424. doi:10.1097/PCC.0000000000001607 \n20. Wright SW, Steenhoff AP, Elci O, et al. Impact of contextualized pediatric resuscitation \ntraining on pediatric healthcare providers in Botswana. Resuscitation. Mar 2015;88:57-62. \ndoi:10.1016/j.resuscitation.2014.12.007 \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n 1 5\n21. Meaney PA, Sutton RM, Tsima B, et al. Training hospital providers in basic CPR skills in \nBotswana: acquisition, retention and impact of novel training techniques. Resuscitation. Dec \n2012;83(12):1484-90. doi:10.1016/j.resuscitation.2012.04.014 \n22. Hategekimana C, Shoveller J, Tuyisenge L, Kenyon C, Cechetto DF, Lynd LD. Correlates \nof Performance of Healthcare Workers in Emergency, Triage, Assessment and Treatment plus \nAdmission Care (ETAT+) Course in Rwanda: Context Matters. PLoS One. 2016;11(3):e0152882. \ndoi:10.1371/journal.pone.0152882 \n23. Meaney PA, Topjian AA, Chandler HK, et al. Resuscitation training in developing countries: \na systematic review. Resuscitation. Nov 2010;81(11):1462-72. \ndoi:10.1016/j.resuscitation.2010.06.024 \n24. Costello A, Dalglish S. Towards a Grand Convergence for child survival and health: A \nstrategic review of options for the future building on lessons learnt from IMNCI. 2016. \nhttp://apps.who.int/iris/bitstream/handle/10665/251855/WHO-MCA-16.04-\neng.pdf;jsessionid=BC0F755CBBBD294750E31EDEBD009379?sequence=1 \n25. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, \nutilization, and integration in public health. Annu Rev Public Health. 2009;30:151-74. \ndoi:10.1146/annurev.publhealth.031308.100049 \n26. WHO. Health Policy and System Support to Optimize Community Health Worker \nprogrammes. 2018. Accessed 2 Feb 2020. \nhttp://apps.who.int/iris/bitstream/handle/10665/275474/9789241550369-eng.pdf  \n27. WHO. Recommendations on digital interventions for health system strengthening. 2019.  \n28. Fontaine G, Cossette S, Maheu-Cadotte MA, et al. Efficacy of adaptive e-learning for health \nprofessionals and students: a systematic review and meta-analysis. BMJ Open. Aug 28 \n2019;9(8):e025252. doi:10.1136/bmjopen-2018-025252 \n29. Howell WS. The empathic communicator. Wadsworth Pub. Co.; 1982:x, 255 p. \n30. Alwadei AH, Tekian AS, Brown BP, et al. Effectiveness of an adaptive eLearning \nintervention on dental students’ learning in comparison to traditional instruction. Journal of Dental \nEducation. 2020;84(11):1294-1302. doi:https://doi.org/10.1002/jdd.12312\n \n31. Seel NM. Ebbinghaus, Hermann (1850–1909). Springer US; 2012:1069-1070. \n32. Bjork RA, Dunlosky J, Kornell N. Self-Regulated Learning: Beliefs, Techniques, and \nIllusions. Annual Review of Psychology. 2013;64(1):417-444. doi:10.1146/annurev-psych-113011-\n143823 \n33. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed practice in verbal recall \ntasks: A review and quantitative synthesis. Psychol Bull. May 2006;132(3):354-80. \ndoi:10.1037/0033-2909.132.3.354 \n34. Hintzman DL, Block RA. Memory for the spacing of repetitions. Journal of Experimental \nPsychology. 1973;99(1):70-74. doi:10.1037/h0034761 \n35. Vlach HA. The spacing effect in children' s generalization of knowledge: Allowing children \ntime to forget promotes their ability to learn. Child Development Perspectives. 2014;8(3):163-168. \ndoi:10.1111/cdep.12079 \n36. Sutton RM, Niles D, Meaney PA, et al. \"Booster\" training: evaluation of instructor-led \nbedside cardiopulmonary resuscitation skill training and automated corrective feedback to improve \ncardiopulmonary resuscitation compliance of Pediatric Basic Life Support providers during \nsimulated cardiac arrest. Pediatric critical care medicine : a journal of the Society of Critical Care \nMedicine and the World Federation of Pediatric Intensive and Critical Care Societies. May \n2011;12(3):e116-21. doi:10.1097/PCC.0b013e3181e91271 \n37. Sutton RM, Niles D, Meaney PA, et al. Low -dose, high-frequency CPR training improves \nskill retention of in-hospital pediatric providers. Pediatrics. Jul 2011;128(1):e145-51. \ndoi:10.1542/peds.2010-2105 \n38. Niles DE, Nishisaki A, Sutton RM, et al. Improved Retention of Chest Compression \nPsychomotor Skills With Brief \"Rolling Refresher\" Training. Simulation in healthcare : journal of the \nSociety for Simulation in Healthcare. Aug 2017;12(4):213-219. doi:10.1097/sih.0000000000000228 \n39. Tudor Car L, Soong A, Kyaw BM, Chua KL, Low-Beer N, Majeed A. Health professions \ndigital education on clinical practice guidelines: a systematic review by Digital Health Education \ncollaboration. BMC Med. Jul 18 2019;17(1):139. doi:10.1186/s12916-019-1370-1 \n40. Agarwal S, Rosenblum L, Goldschmidt T, Carras M, Goel N, Labrique AB. Mobile \nTechnology in Support of Frontline Health Workers. 2016. \nhttps://www.researchgate.net/publication/305676655_Mobile_Technology_in_Support_of_Frontline\n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n 1 6\n_Health_Workers_A_comprehensive_overview_of_the_landscape_knowledge_gaps_and_future_d\nirections \n41. Tudor Car L, Riboli ‐ Sasco EF, Marcano Belisario JS, et al. Mobile learning for delivering \nhealth professional education. Cochrane Database of Systematic Reviews. \n2015;(9)doi:10.1002/14651858.CD011861 \n42. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based \nlearning in the health professions: a meta-analysis. Jama. Sep 10 2008;300(10):1181-96. \ndoi:10.1001/jama.300.10.1181 \n43. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-\nlearning on clinician behaviour and patient outcomes: A systematic review. Int J Nurs Stud. May \n2016;57:70-81. doi:10.1016/j.ijnurstu.2016.01.011 \n44. Lahti M, Hatonen H, Valimaki M. Impact of e-learning on nurses' and student nurses \nknowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. Jan \n2014;51(1):136-49. doi:10.1016/j.ijnurstu.2012.12.017 \n45. Vaona A, Banzi R, Kwag KH, et al. E-learning for health professionals. Cochrane Database \nSyst Rev. Jan 21 2018;1:CD011736. doi:10.1002/14651858.CD011736.pub2 \n46. Brusamento S, Kyaw BM, Whiting P, Li L, Tudor Car L. Digital Health Professions \nEducation in the Field of Pediatrics: Systematic Review and Meta-Analysis by the Digital Health \nEducation Collaboration. J Med Internet Res. Sep 25 2019;21(9):e14231. doi:10.2196/14231 \n47. Chandrasekaran A, Thukral A, Deorari AK. E-learning in newborn health - a paradigm shift \nfor continuing professional development for doctors and nurses. Indian J Pediatr. Dec \n2014;81(12):1376-80. doi:10.1007/s12098-014-1362-2 \n48. Frehywot S, Vovides Y, Talib Z, et al. E-learning in medical education in resource \nconstrained low- and middle-income countries. Hum Resour Health. Feb 4 2013;11:4. \ndoi:10.1186/1478-4491-11-4 \n49. Smith ZH, Rozenfeld B, Setlhare S, et al. Feasibility Trial of Adaptive Electronic Learning \nfor Pediatric Healthcare Workers in Tanzania. 2023: \n50. National Guideline for Neonatal Care and the Establishement of Neonatal Care Unit (2019). \n51. Moullin JC, Dickson KS, Stadnick NA, et al. Ten recommendations for using \nimplementation frameworks in research and practice. Implement Sci Commun. 2020;1:42. \ndoi:10.1186/s43058-020-00023-7 \n52. Soicher RN, Becker-Blease KA, Bostwick KCP. Adapting implementation science for higher \neducation research: the systematic study of implementing evidence-based practices in college \nclassrooms. Cogn Res Princ Implic. Nov 5 2020;5(1):54. doi:10.1186/s41235-020-00255-0 \n53. von Elm E, Altman DG, Egger M, et al. Strengthening the Reporting of Observational \nStudies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. \nOct 20 2007;335(7624):806-8. doi:10.1136/bmj.39335.541782.AD \n54. Lawrence CE, Dunkel L, McEver M, et al. A REDCap-based model for electronic consent \n(eConsent): Moving toward a more personalized consent. Journal of Clinical and Translational \nScience. 2020;4(4):345-353. doi:10.1017/cts.2020.30 \n55. Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: extension to \nrandomised pilot and feasibility trials. BMJ. Oct 24 2016;355:i5239. doi:10.1136/bmj.i5239 \n56. Bonenberger M, Aikins M, Akweongo P, Wyss K. The effects of health worker motivation \nand job satisfaction on turnover intention in Ghana: a cross-sectional study. Human Resources for \nHealth. 2014/08/09 2014;12(1):43. doi:10.1186/1478-4491-12-43 \n57. Xie J, Sreenivasan S, Korniss G, Zhang W, Lim C, Szymanski BK. Social consensus \nthrough the influence of committed minorities. Physical Review E. 07/22/ 2011;84(1):011130. \ndoi:10.1103/PhysRevE.84.011130 \n58. Grey S. Numbers and Beyond: The Relevance of Critical Mass in Gender Research. \nPolitics & Gender. 2006;2(4):492-502. doi:10.1017/S1743923X06221147 \n59. Centola D, Becker J, Brackbill D, Baronchelli A. Experimental evidence for tipping points in \nsocial convention. Science. Jun 8 2018;360(6393):1116-1119. doi:10.1126/science.aas8827 \n60. Fu Q, Gao Z, Zhou J, Zheng Y. CLSA: A novel deep learning model for MOOC dropout \nprediction. Computers & Electrical Engineering. 2021/09/01/ 2021;94:107315. \ndoi:https://doi.org/10.1016/j.compeleceng.2021.107315\n \n61. Duncan A, Premnazeer M, Sithamparanathan G. Massive open online course adoption \namongst newly graduated health care providers. Adv Health Sci Educ Theory Pract. Oct \n2022;27(4):919-930. doi:10.1007/s10459-022-10113-x \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n 1 7\n62. Bredesen IM, Bjoro K, Gunningberg L, Hofo ss D. Effect of e-learning program on risk \nassessment and pressure ulcer classification - A randomized study. Nurse Educ Today. May \n2016;40:191-7. doi:10.1016/j.nedt.2016.03.008 \n63. Levine DA, Funkhouser EM, Houston TK, et al. Improving care after myocardial infarction \nusing a 2-year internet-delivered intervention: the Department of Veterans Affairs myocardial \ninfarction-plus cluster-randomized trial. Arch Intern Med. Nov 28 2011;171(21):1910-7. \ndoi:10.1001/archinternmed.2011.498 \n64. Horiuchi S, Yaju Y, Koyo M, Sakyo Y, Nakayama K. Evaluation of a web-based graduate \ncontinuing nursing education program in Japan: A randomized controlled trial. Nurse Education \nToday. 2009/02/01/ 2009;29(2):140-149. doi:https://doi.org/10.1016/j.nedt.2008.08.009 \n65. Le TT, Rait MA, Jarlsberg LG, Eid NS, Cabana MD. A Randomized Controlled Trial to \nEvaluate the Effectiveness of a Distance Asthma Learning Program for Pediatricians. Journal of \nAsthma. 2010/04/01 2010;47(3):245-250. doi:10.3109/02770900903560209 \n66. Fordis M, King JE, Ballantyne CM, et al. Comparison of the Instructional Efficacy of \nInternet-Based CME With Live Interactive CME WorkshopsA Randomized Controlled Trial. Jama. \n2005;294(9):1043-1051. doi:10.1001/jama.294.9.1043 \n67. Harris JM, Jr., Elliott TE, Davis BE, Chabal C, Fulginiti JV, Fine PG. Educating Generalist \nPhysicians about Chronic Pain: Live Experts and Online Education Can Provide Durable Benefits. \nPain Medicine. 2008;9(5):555-563. doi:10.1111/j.1526-4637.2007.00399.x \n68. Weick KE, Sutcliffe KM. Managing the unexpected. vol 9. San Francisco: Jossey-Bass; \n2001. \n69. Wilson RM, Michel P, Olsen S, et al. Patient  safety in developing countries: retrospective \nestimation of scale and nature of harm to patients in hospital. Bmj. 2012;344 \n70. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international \ncommunity of software platform partners. J Biomed Inform. Jul 2019;95:103208. \ndoi:10.1016/j.jbi.2019.103208 \n71. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data \ncapture (REDCap)--a metadata-driven methodology and workflow process for providing \ntranslational research informatics support. J Biomed Inform. Apr 2009;42(2):377-81. \ndoi:10.1016/j.jbi.2008.08.010 \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n Zonal \nHosp \nHC \n #1 \nHC  \n#2 \nHC \n#3 \nDuration in Study (months) 7 5 4 2 \nProviders (n) 321 39 66 59 \n Specialist Care  +  - - - \n   Pediatrician  Y N N N \nBirths/year 7000 296 3678 6002 \n1m-5y Admissions/year 6550 0 897 577 \nServices Provided     \n Outpatient Clinics  + + + + \n Inpatient Wards  + - + + \n Pediatric Ward  + - +/- + \n NICU  + - - - \n Pediatric ICU  + - - - \n Malnutrition unit  + - - +/- \n Cesarean section  + - + + \n Transfusion Services  + - + + \n Pharmacy  + + + + \n Dialysis  + - - - \nTable 1. Facility characteristics \nHC: Health Center \n \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n Table 2. Provider Characteristics Overall \n n 195 \n Facility, n (%)    Zonal Hospital 110 (56.4%) \n  Health Center #1 27 (13.8%) \n  Health Center #2 42 (21.5%) \n  Health Center #3 16 (8.2%) \n Cadre, n (%)    Specialist Pediatrician 5 (2.6%) \n  Medical Officer 75 (38.5%) \n  Assistant Medical Officer 3 (1.5%) \n  Clinical Officer 21 (10.8%) \n  Assistant Clinical Officer 2 (1.0%) \n  Nursing Officer 53 (27.2%) \n  Medical Attendant 4 (2.1%) \n  Other 31 (15.9%) \n Clinical Experience (years), \nmedian [IQR] 4 [1-9] \n Previous Newborn Training, n \n(%)  \n  Essential Newborn Care (ENC) 61 (31.3%) \n  Helping Babies Breathe (HBB) 80 (41.0%) \n   Both ENC and HBB 45 (23%) \n Job Satisfaction (1-5), mean \n(std)  \n  Overall 3.6 (0.8) \n  Renumeration 3.0 (1.2) \n  Work Environment 3.5 (1.0) \n  Tasks 3.7 (0.9) \n  Supervision 3.8 (1.0) \n  In-service Training 3.5 (1.2) \n  Management 3.5 (1.0) \n  Career Development 3.6 (1.1) \n  Morale 4.1 (0.8) \n Motivation (1-5), mean (std) 3.7 (0.6) \nCadre other: Laboratory Scientist/technologist (11), Assistant nurse officer (5), \nNurse midwife (2), Health Assistant (1); no response (2) \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \nTable 3. Baseline Metacognition Heatmap \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint \n\n \n \nUnivariate Odds Ratio \n[95% CI) \nAdjusted Odds Ratio \n[95% CI) \n Facility     Zonal Hospital comparison comparison \n  Health Center #1 1.6 [0.6, 4.2] 0.5 [0.1, 2.4] \n  Health Center #2 0.8 [0.4, 1.7] 0.3 [0.1, 1.3] \n  Health Center #3 1.0 [0.3, 3.0] 0.5 [0.1, 2.5] \n Cadre     Specialist Pediatrician insuff. data insuff. data \n  Medical Officer comparison comparison \n  Assistant Medical Officer insuff. data insuff. data \n  Clinical Officer 1.0 [0.4, 2.6] 2.4 [0.5, 11.2] \n  Assistant Clinical Officer insuff. data insuff. data \n  Nursing Officer 3.4 [1.4, 8.1] 5.6 [1.8, 18.1] \n  Medical Attendant insuff. data insuff. data \n  Other 1.1 [0.5, 2.6] 2.4 [0.6, 9.6] \n Clinical Experience     ≤  1 year comparison comparison \n  2 - 3 years 0.3 [0.1, 0.8] 1.3 [0.4, 4.7] \n  4 - 9 years 0.7 [0.3, 1.8] 0.3 [0.1, 1.1] \n  ≥  10 years 1.1 [0.4, 2.8] 0.8 [0.3, 2.1] \n Previous Newborn Training   \n  Essential Newborn Care (ENC) 1.0 [0.5, 1.9] 1.4 [0.6, 3.4] \n  Helping Babies Breathe (HBB) 1.0 [0.5, 1.8] 0.6 [0.2, 1.5] \n Job Satisfaction (1-5)     Overall 1.0 [0.7, 1.4] - \n  Renumeration 1.1 [0.8, 1.4] - \n  Work Environment 1.0 [0.7, 1.3] - \n  Tasks 0.7 [0.5, 1.0] - \n  Supervision 1.0 [0.8, 1.4] - \n  In-service Training 1.1 [0.9, 1.5] - \n  Management 1.0 [0.8, 1.4] - \n  Career Development 1.0 [0.7, 1.3] - \n  Morale 1.1 [0.8, 1.7] - \n Motivation (1-5) 1.2 [0.7, 2.1] - \n Baseline Scores (0-1)      Conscious Competence 31.6 [5.8, 183.5] - \n  Unconscious Competence 1.4 [0.1, 14.6] - \n  Conscious Incompetence insuff. data - \n  Unconscious incompetence 0.9 [0.1, 5.8] - \nTable 4. Factors associated with initial learning completion or persistent activity. \n . CC-BY 4.0 International licenseIt is made available under a \n is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)\nThe copyright holder for this preprint this version posted July 13, 2023. ; https://doi.org/10.1101/2023.07.11.23292406doi: medRxiv preprint","source_license":"CC-BY-4.0","license_restricted":false}