Impact of a 2-week Flipped Classroom Virtual Neurology Clerkship versus a Traditional 4- week Rotation on NBME Shelf Exam Scores

Impact of a 2-week Flipped Classroom Virtual Neurology Clerkship versus a Traditional 4- week Rotation on NBME Shelf Exam Scores | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Impact of a 2-week Flipped Classroom Virtual Neurology Clerkship versus a Traditional 4- week Rotation on NBME Shelf Exam Scores Igor Rybinnik, Gian Pal, Humza Farrukh, Paulina Przydzial, Ram Mani This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4631444/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 05 Nov, 2024 Read the published version in BMC Medical Education → Version 1 posted 4 You are reading this latest preprint version Abstract Objective There is conflicting evidence whether decreased clerkship duration is associated with reduced NBME shelf examination performance. We hypothesized that scores would remain stable for students in a shortened 2-week flipped classroom-based virtual rotation as compared to the traditional 4-week Neurology clerkship. Background There is conflicting evidence whether decreased clerkship duration is associated with reduced NBME shelf examination performance. We hypothesized that Neurology shelf exam scores would remain stable for students in a shortened 2-week flipped classroom-based virtual rotation as compared to the traditional 4-week Neurology clerkship with direct patient care supplemented by didactics. Methods This is a retrospective observational cohort study. In April 2020, the neurology clerkship at Rutgers Robert Wood Johnson Medical School switched from a traditional 4-week in-person clinical rotation supplemented by didactics to a 2-week flipped classroom-based virtual-only curriculum decoupled from a supplementary 2-week in-person clinical rotation to be completed later. Students were offered the opportunity to test after the 2-week virtual curriculum. NBME shelf scores for these “exam before clinical rotation” (EBC) students over the final quarter of the academic year 2019–2020 were compared to those of the “exam after clinical” (EAC) students undergoing the 4-week traditional rotation from July 2018 to March 2020. Results 321 students completed the shelf exam between July 2018 and June 2020, of which 284 students comprised the EAC cohort and 37 students, the EBC cohort. Mean scores did not differ between the EBC and EAC cohorts (81.8 ± 6.7 versus 81.8 ± 6.5, p = 0.96). Additional analyses showed no significant differences in the performance of EBC students as compared to the EAC students that tested within the same academic year 2019–2020 or within the matched quarter of the prior academic year 2018–2019, and students completing the virtual rotation who delayed testing after the clinical experience. Conclusions Testing after a 2-week flipped classroom-based virtual curriculum in Neurology without additional clinical training did not negatively affect neurology NBME shelf scores. Decoupling clinical experiences from knowledge acquisition and the evaluation of that knowledge base by means of NBME shelf examination is not likely to disadvantage learners with respect to shelf examination performance. Clinical Trial Number: Not applicable. This is not a clinical trial. Flipped classroom curriculum evaluation neurology nervous system instructional design NBME shelf examination Introduction In US medical schools, the neurology clerkship is often completed during the third year of medical school. Approximately 80% of these clerkships employ the National Board of Medical Examiners (NBME) shelf examination as an assessment tool upon completion of the entire clerkship, and this examination carries a mean weight of 21% towards the final grade.[ 1 ] Although many medical schools have overall similar pedagogical structures, the duration of the neurology clerkship varies substantially among medical schools, varying between 2–6 weeks (most often 4 weeks).[ 2 ] The optimal clerkship duration has not been established,3 and prior studies comparing different clerkship lengths on NBME shelf exam performance have had mixed results.[ 3 – 8 ] Factors including USMLE Step 1 scores, amount of dedicated study for the shelf exam, number of prior clerkships, and practice exams have all been associated with increased neurology NBME shelf exam scores.[ 9 ] For many logistical, ethical, and practical reasons, even fewer studies have looked at the effect of a completely virtual clinical clerkship versus a traditional clerkship on medical student clinical education, although guidelines for a virtual curriculum have been published by the American Academy of Neurology.[ 10 ] No studies have examined whether the presence or absence of the clinical portion of the neurology clerkship influences Neurology NBME shelf exam scores. On March 17th, 2020, the Association of American Medical Colleges (AAMC) provided directives for medical schools to halt clinical rotations in an effort to ensure student safety.[ 11 ] Subsequent AAMC guidelines restricted medical student participation in direct in-person patient contact, demonstrating the unprecedented impact of the COVID-19 pandemic on the medical student education.[ 12 – 14 ] While many medical schools were already in the process of reevaluating and modifying their curricula[ 3 , 15 , 16 ], the COVID-19 pandemic provided a unique opportunity to assess the effect of an entirely virtual curriculum, in addition to reduced clerkship length, on medical student education. At Rutgers Robert Wood Johnson Medical School (RWJMS), prior to the COVID-19 pandemic, the neurology clerkship was a mandatory 4-week rotation to be completed during the third or fourth years. The rotation consisted predominantly of direct patient care supplemented by didactics, and NBME shelf examination was mandated on the last day of the rotation. In April 2020 due to the COVID-19 pandemic, the traditional 4-week clerkship was replaced by a 2-week entirely virtual curriculum, with additional 2 weeks of in-person clinical training to be completed at a later date. The objective of this study was to evaluate the effect of a completely virtual neurology clerkship versus a traditional in-person clerkship on medical student NBME shelf exam performance. We hypothesized that shelf scores would not be significantly different for the “exam before clinical” (EBC) cohort of students who completed the 2-week virtual clerkship as compared to the “exam after clinical” (EAC) cohort who completed the traditional 4-week rotation. Methods This is a retrospective study of consecutive third- and fourth-year medical students undergoing the mandatory core neurology clerkship between July 2019 and June 2020 at RWJMS, a public medical school. The study was approved by our local institutional review board. Immediately prior to the COVID-19 pandemic, the neurology clerkship at RWJMS consisted of a traditional four-week clinical rotation which included both direct patient care (primarily inpatient experiences with an afternoon of outpatient experience per week), as well as 3 afternoons of didactic sessions for a total of 12 hours of synchronous education (7 hours of traditional lecture, 4 hours of case-based interactive learning, and 1 hour of hybrid simulation) and 4 hours of asynchronous preparatory work per flipped classroom paradigm,[ 17 ] which involved watching topic review videos. Neurology is a core clinical clerkship at RWJMS and is required in the third medical school year. While the majority of neurology clerks at our institution are third-year medical students, a small group of fourth-year students also participate annually. In the academic year 2019–2020, there were 20 fourth-year medical students enrolled in the rotation. During the height of the COVID-19 pandemic, in April 2020, the clerkship was converted into a 2-week virtual rotation which employed the flipped classroom model with a total of 22 hours of synchronous case-based interactive learning and a recommended 8–10 hours of asynchronous preparatory work per flipped classroom paradigm. A sample schedule is outlined in Table 1 . Following the virtual rotation, an additional 2 weeks of clinical in-person experience were scheduled to be completed at a later date. Students were given the choice to take the NBME shelf exam at the completion of the virtual clerkship or to delay testing until after the additional clinical training. Neurology clerkship leadership, Student Affairs Deans, and Cognitive Skills Division were available to advise students on the timing of the exam per our institutional protocol. At the beginning and conclusion of the virtual clerkship, all EBC students were asked to complete an optional anonymous online survey, which collected their demographics and assessed with a 5-point Likert scale student perception of the effectiveness of the virtual curriculum in learning specific skills and preparedness for the in-person Neurology clerkship overall. The survey also solicited open-ended comments and a list of common learning resources used by students for shelf examination preparation. Demographics data was not available for EAC cohort. Table 1 Sample schedule of a 2-week virtual rotation WEEK 1 Monday Tuesday Wednesday Thursday Friday Asynchronous assignments Watch topic review video on approach to ischemic and hemorrhagic stroke, and complete 10-MCQ vignette-based quiz (1–2 hours) Read topic review primer on neuromuscular disorders, and complete 10-MCQ vignette-based quiz (1–2 hours) Watch topic review video on multiple sclerosis, and complete 10-MCQ vignette-based quiz (1–2 hours) Watch topic review video on multiple sclerosis, and complete 10-MCQ vignette-based quiz (1–2 hours) Synchronous sessions Orientation and review of neurological localization and examination (4 hours) Cases: Ischemic and Hemorrhagic Stroke (2 hours) Cases: Neuromuscular Disorders (2 hours) Cases: Multiple Sclerosis and Neuroimmunology (2 hours) Cases: Movement Disorders (2 hours) WEEK 2 Monday Tuesday Wednesday Thursday Friday Asynchronous assignments Watch topic review video on neuro-prognostication and complete 10-MCQ vignette-based quiz (1–2 hours) Watch topic review video on approach to seizures and epilepsies, and complete 10-MCQ vignette-based quiz (1–2 hours) Watch topic review video on differential diagnosis of and approach to coma, and complete 10-MCQ vignette-based quiz (1–2 hours) Synchronous sessions Simulation: Neuro-prognostication and delivering bad news (2 hours) Cases: Seizures and Epilepsies (2 hours) Cases: Neurological Emergencies, Coma (2 hours) Cases: Headache Disorders (2 hours) Cases: Neuro-infectious Diseases (2 hours) Statistical Analysis The mean shelf exam scores of the EBC cohort of students who completed the NBME examination immediately after the 2-week virtual rotation in April to June of 2020 were compared by t-test (independent sample) with the scores of the sequentially enrolled historical cohorts completing the shelf examination immediately following a traditional 4-week in-person neurology clerkship (EAC cohort) – from July 2018 through March 2020, during which time the clerkship structure remained relatively unchanged. Three secondary analyses were conducted using t-tests. The shelf exam scores of EBC students were compared to (1) scores of EAC students within the same academic year (between July 2019 and March 2020), (2) scores of EAC students within the same quarter of the prior academic year (April 2019 to June 2019), and (3) scores of “exam after delayed clinical” (EADC) students who completed the 2-week virtual rotation but delayed testing until after the supplementary 2-week rotation. We also compared EBC cohort results with the Neurology shelf examination national means published by NBME for the academic year 2019–2020. Results 68 students participated in the 2-week virtual clerkship from April 2020 to June 2020, 65 of which (96%) were third-year students. Demographic data was available for 51 of 68 students (75%) and is summarized in Table 2 . Median age of the students in this cohort was 24–26 years, and 74.5% identified as female. 58.8% identified as non-white, with “Asian” most frequently selected (43.1%). The majority (94.1%) were third-year medical students, and 21.6% of students indicated that they completed or were enrolled in an advanced degree program (master’s or Ph.D.). Only 9.8% agreed or strongly agreed that their neurological knowledge from preclinical years was sufficient to assess patients with neurological disorders. Educational resources most commonly used daily or at least weekly in clerkships prior to this virtual rotation were question banks (used by 98%), topic review articles in reputable journals or UpToDate.com (used by 75%), review books specifically written for clerks (used by 67.3%), internet search engine queries (used by 63.4%). PubMed queries and primary literature as well as traditional book chapters were used infrequently, by 38.5% and 15.4% respectively. The results are summarized in Table 3 . 37 of the 68 students (54%) chose to take the NBME shelf exam after completion of the 2-week virtual clerkship and prior to starting the in-person rotation. The median time from the conclusion of the virtual rotation to the shelf examination was 14 days (IQR 14–37). The mean shelf exam score of this EBC cohort was 81.8 (SD 6.7). The mean shelf exam score of the 284 students who completed the traditional neurology clerkship from July 2018 to March 2020 (EAC cohort), was 81.8 (SD 6.5). In the analysis of the primary outcome, there was no statistically significant difference between the mean shelf exam scores of the EBC and EAC groups (p value of 0.96). In the prespecified subgroup analyses, there was no statistically significant differences between the mean shelf exam scores of the EBC cohort, the subgroup of EAC cohort that tested within the same academic year (81.8 ± 6.7 versus 81.5 ± 6.5, p = 0.90), the subgroup of the EAC cohort that tested within the same quarter of the prior academic year (81.8 ± 6.7 versus 82.4 ± 6.6, p = 0.65), and the EADC cohort that delayed testing until the supplementary clinical experience (81.8 ± 6.7 versus 82.7 ± 6.5, p = 0.54). Neurology shelf examination national mean for the academic year 2019–2020 published by NBME for 13,098 students (80.5 ± 7.5) did not differ qualitatively from the EBC mean. The study was adequately powered (alpha 0.05, power 0.80, 1 tail) assuming that an exam score difference of 4 is meaningful (standard error of measurement of national shelf exam score distribution published by NBME is 4). Table 2. Demographics of the EBC and EAC cohort EBC cohort: Students that completed 2-week virtual rotation (n=68) EAC cohort: Students that completed 4-week traditional rotation (n=284) Dates of rotation April 2020 to June 2020 July 2018 to March 2020 Age ranges 29 0% 0% 51.0% 33.3% 15.7% 0% 0% 43.5% 44.5% 12% Gender Female Male Other 74.5% 25.5% 0% 54.0% 46.0% 0% Ethnicity Asian Black or African American White American Indian or Alaska Native Hispanic, Latino, or of Spanish origin Native Hawaiian or Other Pacific Islander Other, or prefer not to say 43.1% 9.8% 41.2% 0% 0% 0% 5.9% 34.5% 9.5% 48.3% 0% 5.0% 0% 2.5% Highest level of education (multiple answers are possible) 3 rd year medical student 4 th year medical student Completed or currently enrolled in a masters degree program Completed or currently enrolled in a Ph.D. program. 94.1% 3.9% 1.9% 7.8% Data not available Table 3. Mean Shelf Exam Scores EBC cohort that tested after 2-week virtual rotation only EAC cohort (Shelf exam after traditional 4-week rotation) Subgroup of EAC cohort that tested in 2019-2020 academic year Subgroup of EAC cohort that tested in 4 th quarter of 2018-2019 academic year EADC cohort that tested after 2-week virtual and 2-week clinical rotations Dates April 2020 to June 2020 July 2018 to March 2020 July 2019 to March 2020 April 2019 to June 2019 Academic year 202-2021 n 37 284 119 55 31 Mean shelf Score (SD) 81.8 (6.7) 81.8 (6.5) *p value 0.96 81.6 (6.4) * p value 0.90 82.4 (6.6) * p value 0.65 80.5 (7.5) * p value 0.54 Median delay to exam (IQR) 14 (14-37) 234 (150-290) * p-value for comparison to EBC cohort 41 of 68 students completed the survey at the conclusion of the 2-week virtual clerkship, and the majority (73%) reported daily or at least weekly use of the resources prespecified by the rotation. Most students felt that during the virtual rotation, an appropriate amount of time was devoted to assigned homework (85% of responders), live case-based didactics (81% of responders), and NBME shelf exam review (74% of responders). Discussion The unprecedented limitations the COVID-19 pandemic placed on medical student clinical learning from direct patient care, provided us the unique opportunity to investigate the effect of entirely virtual clinical instruction on NBME neurology shelf exam performance. Our results indicate that shelf exam performance did not differ between the students who completed a traditional 4-week in-person versus a 2-week virtual clerkship. This is consistent with prior studies of student performance in other virtual rotations, including virtual radiology, emergency medicine, surgery, and pediatrics.[ 18 – 21 ] The effectiveness of online teaching in medical education was validated[ 22 , 23 ], and Durfee and colleagues, in particular, utilized an online flipped classroom model as well as larger group didactic lectures to confirm effectiveness of virtual radiology education on the student performance of the standardized Alliance of Medical Student Education in Radiology (AMSER) final examination[ 18 ]. Redinger and colleagues found similar NBME shelf exam performance among the cohort who completed an entirely virtual 4-week emergency medicine course as compared to a traditional elective within the same academic year, concluding that the fund of knowledge in emergency medicine was not attenuated in the virtual rotation experience[ 21 ]. Few papers have shown a negative impact of reducing clerkship length on subject exam performance.[ 5 ] While the consensus regarding the optimal length of medical school clerkships may remain ambiguous[ 2 , 3 , 24 , 25 ], our results support the conclusion that a shortened 2-week virtual rotation does not negatively impact the acquisition of a requisite knowledge base as measured by the NBME shelf examination. There were several important limitations to this study. The single center retrospective design and voluntary participation in the EBC cohort may have introduced selection bias, the probability of which is diminished by: 1) the similar results obtained in the EBC and the national NBME cohorts, and 2) the absence of statistically significant Step 1 score disparity between the EBC and EADC cohorts. The time elapsed between knowledge acquisition and testing may have influenced the results. In a traditional 4-week clerkship, students tested on the last day of the rotation, while the median time to the shelf exam for the EBC cohort was 14 days (IQR 14–37), potentially affording more dedicated study time to this group. However, the EADC students that tested after the 2-week virtual and supplementary 2-week in-person clinical rotation had a median delay from the end of virtual rotation to the shelf exam of 259 days (IQR 196–315) and yet performed similarly. The increase of synchronous education from a total of 12 hours (58% passive learning via traditional lectures) in the EAC cohort to the 22 hours of 100% case-based interactive learning may have contributed to the stable NBME shelf exam performance in our study. However, the case-based sessions in the virtual rotation were designed to address the most salient clinical scenarios appropriate to the bedside training of Neurology clerks as outlined by the Consortium of Neurology Clerkship Directors of the American Academy of Neurology[ 10 ] and were not specifically tailored to strengthen the shelf exam performance. We must also acknowledge the possibility that shelf examination performance may depend largely on students’ individual preparation methods rather than formal instruction, although the majority (73%) of the EBC cohort self-reported frequent use of the resources prespecified by the virtual rotation and indicated a trend towards decreased reliance on other commonly used resources (question bank use declined from 98–84%, the use of topic review articles in reputable journals or UpToDate.com declined from 75–66%, and review book use declined from 67–59%). Our findings raise important questions concerning the relationship between the shelf examination and clinical learning. Multiple factors are associated with performance on shelf exams including USMLE Step 1 scores, number of practice exams, amount of dedicated study time, and number of prior clerkships completed.[ 9 ] In our study, the 2-week in-person clinical training was delayed by a median of 234 days (IQR 150–190) after the conclusion of the virtual rotation, which limited the possibility and validity of correlating shelf scores to traditional measures of clinical skills. However, additional clinical training does not seem to improve shelf examination scores. Therefore, decoupling clinical experiences from knowledge acquisition and the evaluation of that knowledge base by means of NBME shelf examination is not likely to disadvantage learners with respect to shelf examination performance in the Neurology clerkship. Finally, our study focused on the Neurology NBME shelf examination, limiting generalizability of our results to other clerkships. Conclusion The NBME Neurology shelf examination scores of students who chose to test after a 2-week virtual rotation utilizing the flipped classroom model did not differ from the scores of students who tested after a 4-week traditional in-person neurology clerkship. Furthermore, no significant differences were observed between the shelf examination performance after the virtual rotation alone versus the traditional rotation within the same academic year, and within a matched quarter of the prior academic year. This observation also held true for students who completed the virtual rotation and chose to postpone testing until after supplementary clinical experience. These findings support the conclusion that a shortened 2-week flipped-classroom virtual rotation does not adversely affect the NBME shelf examination performance among neurology clerks. Abbreviations AAMC - Association of American Medical Colleges EAC – Exam after clinical EBC – Exam before clinical EADC - Exam After Delayed Clinical NBME - National Board of Medical Examiners RWJMS - Rutgers Robert Wood Johnson Medical School USMLE - United States Medical Licensing Examination Declarations Ethics approval : The study was approved by the Rutgers University Institutional Review Board. Consent to participate: Formal consent waiver was approved by the Rutgers University Institutional Review Board since study involves deidentified anonymized collection and aggregate analysis of data collected for the purposes of quality assurance of the curriculum and presents minimal or no risk to the study participants. Consent for publication : Not applicable. The manuscript does not contain any individual person’s data. Availability of data and materials: Investigators may request access to anonymized individual patient data and redacted study documents including raw datasets, analysis-ready datasets, statistical analysis plan, study methods, and dataset specifications. When approved, all documents may be accessed for a predetermined time of 12 months. Competing interests : IR: No relevant competing interests to declare; GP: No relevant competing interests to declare; RM: No relevant competing interests to declare; HF: No relevant competing interests to declare; PP: No relevant competing interests. Funding: This study was not funded. Author’s contributions : IR wrote the main manuscript and prepared the tables. GP made substantial revisions to the manuscript and statistics. HF performed background research and contributed to the “background” section and collated the references. RM made substantial revisions to the manuscript and assisted with statistics. PP contributed to the “discussion” and “conclusion” sections. Acknowledgements : None. Author’s information : Please see below. References Safdieh, J.E., et al., A dozen years of evolution of neurology clerkships in the United States: Looking up. Neurology, 2018. 91 (15): p. e1440-e1447. Examiners, N.M.B.o. Characteristics of Clinical Clerkships: NBME Survey Results. . 2022; Available from: www.nbme.org. Monrad, S.U., et al., Does Reducing Clerkship Lengths by 25% Affect Medical Student Performance and Perceptions? Acad Med, 2018. 93 (12): p. 1833-1840. Bhatia, N.D., et al., Cutting too deep? Assessing the impact of a shorter surgery clerkship on students' clinical skills and knowledge. Am J Surg, 2014. 207 (2): p. 209-12. Bostwick, J.M. and C. Alexander, Shorter psychiatry clerkship length is associated with lower NBME psychiatry shelf exam performance. Acad Psychiatry, 2012. 36 (3): p. 174-6. Case, S.M., D.R. Ripkey, and D.B. Swanson, The effects of psychiatry clerkship timing and length on measures of performance. Acad Med, 1997. 72 (10 Suppl 1): p. S34-6. Gary, N.E. and G.C. Rosevear, Effect of reduction in length of third-year clerkships on students' academic performance. J Med Educ, 1988. 63 (5): p. 406-7. Ouyang, W., M.M. Cuddy, and D.B. Swanson, US Medical Student Performance on the NBME Subject Examination in Internal Medicine: Do Clerkship Sequence and Clerkship Length Matter? J Gen Intern Med, 2015. 30 (9): p. 1307-12. Sampat, A., et al., Neurology Clerkship: Predictors of Objective Structured Clinical Examination and Shelf Performance. J Med Educ Curric Dev, 2019. 6 : p. 2382120519862782. Safdieh, J.E., et al., Core curriculum guidelines for a required clinical neurology experience. Neurology, 2019. 92 (13): p. 619-626. Prescott, J.E. Important guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. . 2020; Available from: https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak. Whelan, A.J. Updated interim guidance for medical students' participation in patient care during the coronavirus (COVID-19) outbreak. . 2020; Available from: https://www.aamc.org/news-insights/press-releases/updated-interim-guidance-medical-students-participation-patient-care-during-coronavirus-covid-19. Whelan, A.J. Updated guidance for medical students' roles in direct patient care. . 2020; Available from: https://www.aamc.org/news-insights/press-releases/covid-19-updated-guidance-medical-students-roles-direct-patient-care. Rose, S., Medical Student Education in the Time of COVID-19. JAMA, 2020. 323 (21): p. 2131-2132. Cooke, M., D.M. Irby, and B.C. O'Brien, Educating Physicians: A Call for Reform of Medical School and Residency . 1st ed. 2010: Jossey-Bass. Skochelak, S.E. and S.J. Stack, Creating the Medical Schools of the Future. Acad Med, 2017. 92 (1): p. 16-19. Hew, K.F. and C.K. Lo, Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ, 2018. 18 (1): p. 38. Durfee, S.M., et al., Medical Student Education Roadblock Due to COVID-19: Virtual Radiology Core Clerkship to the Rescue. Acad Radiol, 2020. 27 (10): p. 1461-1466. Hanson, A.E., et al., Decline in Pediatric Shelf Examination Performance During COVID-19. Cureus, 2021. 13 (10): p. e18453. Prigoff, J., M. Hunter, and R. Nowygrod, Medical Student Assessment in the Time of COVID-19. J Surg Educ, 2021. 78 (2): p. 370-374. Redinger, K.E. and J.D. Greene, Virtual Emergency Medicine Clerkship Curriculum during the COVID-19 Pandemic: Development, Application, and Outcomes. West J Emerg Med, 2021. 22 (3): p. 792-798. Dominguez, M., et al., A Neurology Clerkship Curriculum Using Video-Based Lectures and Just-in-Time Teaching (JiTT). MedEdPORTAL, 2018. 14 : p. 10691. Villa, S., et al., An Emergency Medicine Virtual Clerkship: Made for COVID, Here to Stay. West J Emerg Med, 2021. 23 (1): p. 33-39. Ripkey, D.R., S.M. Case, and D.B. Swanson, Identifying students at risk for poor performance on the USMLE Step 2. Acad Med, 1999. 74 (10 Suppl): p. S45-8. Ripkey, D.R., S.M. Case, and D.B. Swanson, Predicting performances on the NBME Surgery Subject Test and USMLE Step 2: the effects of surgery clerkship timing and length. Acad Med, 1997. 72 (10 Suppl 1): p. S31-3. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 05 Nov, 2024 Read the published version in BMC Medical Education → Version 1 posted Editorial decision: Revision requested 03 Jul, 2024 Editor assigned by journal 02 Jul, 2024 Submission checks completed at journal 02 Jul, 2024 First submitted to journal 24 Jun, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4631444","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":322473706,"identity":"75dcbe47-acca-443d-a37f-93d813e92692","order_by":0,"name":"Igor Rybinnik","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+klEQVRIiWNgGAWjYBACNiidAGIcbKhAyDA2EKflDAMDDyEtMJAAVtXYRoQWPv7Fxz4w7jmcxyfde/DgzHn3EvcDRT7+YLCR3XAAh8MkniXPYHh2uJhN5lzCwY3bihN7gCKzeRjSjHFrOWPMwHDgcGKbRI7BwYfbEoBazhgzMzAcTiRSyxyQlvOfGX8w/Methb8HScvGBqAW/h5mYBgcwGMLWzJDwoH0YjaQlhnHEox7brAZM/MYJBvPxKFFvv/wYYYPB6zz5GfkGH/sqUmQbe8//JjxR4WdbB8OLQwSCaBIaUYTYTDAoRwE+MFm1WGIjIJRMApGwSiAAwBcGmMKkFSGtgAAAABJRU5ErkJggg==","orcid":"","institution":"Rutgers Robert Wood Johnson Medical School","correspondingAuthor":true,"prefix":"","firstName":"Igor","middleName":"","lastName":"Rybinnik","suffix":""},{"id":322473708,"identity":"4832aa13-92b1-49d1-9d1d-352fb28604eb","order_by":1,"name":"Gian Pal","email":"","orcid":"","institution":"Rutgers Robert Wood Johnson Medical School","correspondingAuthor":false,"prefix":"","firstName":"Gian","middleName":"","lastName":"Pal","suffix":""},{"id":322473710,"identity":"862e28d5-64af-4a9d-9b62-60de676c22f7","order_by":2,"name":"Humza Farrukh","email":"","orcid":"","institution":"Rutgers Robert Wood Johnson Medical School","correspondingAuthor":false,"prefix":"","firstName":"Humza","middleName":"","lastName":"Farrukh","suffix":""},{"id":322473711,"identity":"06dcdd6b-4298-40cb-a4f5-d4fb7382729e","order_by":3,"name":"Paulina Przydzial","email":"","orcid":"","institution":"Rutgers Robert Wood Johnson Medical School","correspondingAuthor":false,"prefix":"","firstName":"Paulina","middleName":"","lastName":"Przydzial","suffix":""},{"id":322473712,"identity":"39be8624-9a4e-4c0c-bc7d-7099a2fbed6d","order_by":4,"name":"Ram Mani","email":"","orcid":"","institution":"Rutgers Robert Wood Johnson Medical School","correspondingAuthor":false,"prefix":"","firstName":"Ram","middleName":"","lastName":"Mani","suffix":""}],"badges":[],"createdAt":"2024-06-24 16:16:49","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4631444/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4631444/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12909-024-06239-8","type":"published","date":"2024-11-05T15:57:07+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":68750640,"identity":"5196beba-917e-4f90-9d03-7c5f6d26342a","added_by":"auto","created_at":"2024-11-11 16:12:08","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":493880,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4631444/v1/ac4128a2-2250-4d54-a6f5-7cfe7ef81862.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Impact of a 2-week Flipped Classroom Virtual Neurology Clerkship versus a Traditional 4- week Rotation on NBME Shelf Exam Scores","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn US medical schools, the neurology clerkship is often completed during the third year of medical school. Approximately 80% of these clerkships employ the National Board of Medical Examiners (NBME) shelf examination as an assessment tool upon completion of the entire clerkship, and this examination carries a mean weight of 21% towards the final grade.[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eAlthough many medical schools have overall similar pedagogical structures, the duration of the neurology clerkship varies substantially among medical schools, varying between 2\u0026ndash;6 weeks (most often 4 weeks).[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] The optimal clerkship duration has not been established,3 and prior studies comparing different clerkship lengths on NBME shelf exam performance have had mixed results.[\u003cspan additionalcitationids=\"CR4 CR5 CR6 CR7\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] Factors including USMLE Step 1 scores, amount of dedicated study for the shelf exam, number of prior clerkships, and practice exams have all been associated with increased neurology NBME shelf exam scores.[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] For many logistical, ethical, and practical reasons, even fewer studies have looked at the effect of a completely virtual clinical clerkship versus a traditional clerkship on medical student clinical education, although guidelines for a virtual curriculum have been published by the American Academy of Neurology.[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] No studies have examined whether the presence or absence of the clinical portion of the neurology clerkship influences Neurology NBME shelf exam scores.\u003c/p\u003e \u003cp\u003eOn March 17th, 2020, the Association of American Medical Colleges (AAMC) provided directives for medical schools to halt clinical rotations in an effort to ensure student safety.[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] Subsequent AAMC guidelines restricted medical student participation in direct in-person patient contact, demonstrating the unprecedented impact of the COVID-19 pandemic on the medical student education.[\u003cspan additionalcitationids=\"CR13\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] While many medical schools were already in the process of reevaluating and modifying their curricula[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], the COVID-19 pandemic provided a unique opportunity to assess the effect of an entirely virtual curriculum, in addition to reduced clerkship length, on medical student education.\u003c/p\u003e \u003cp\u003eAt Rutgers Robert Wood Johnson Medical School (RWJMS), prior to the COVID-19 pandemic, the neurology clerkship was a mandatory 4-week rotation to be completed during the third or fourth years. The rotation consisted predominantly of direct patient care supplemented by didactics, and NBME shelf examination was mandated on the last day of the rotation. In April 2020 due to the COVID-19 pandemic, the traditional 4-week clerkship was replaced by a 2-week entirely virtual curriculum, with additional 2 weeks of in-person clinical training to be completed at a later date. The objective of this study was to evaluate the effect of a completely virtual neurology clerkship versus a traditional in-person clerkship on medical student NBME shelf exam performance. We hypothesized that shelf scores would not be significantly different for the \u0026ldquo;exam before clinical\u0026rdquo; (EBC) cohort of students who completed the 2-week virtual clerkship as compared to the \u0026ldquo;exam after clinical\u0026rdquo; (EAC) cohort who completed the traditional 4-week rotation.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eThis is a retrospective study of consecutive third- and fourth-year medical students undergoing the mandatory core neurology clerkship between July 2019 and June 2020 at RWJMS, a public medical school. The study was approved by our local institutional review board.\u003c/p\u003e \u003cp\u003eImmediately prior to the COVID-19 pandemic, the neurology clerkship at RWJMS consisted of a traditional four-week clinical rotation which included both direct patient care (primarily inpatient experiences with an afternoon of outpatient experience per week), as well as 3 afternoons of didactic sessions for a total of 12 hours of synchronous education (7 hours of traditional lecture, 4 hours of case-based interactive learning, and 1 hour of hybrid simulation) and 4 hours of asynchronous preparatory work per flipped classroom paradigm,[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] which involved watching topic review videos. Neurology is a core clinical clerkship at RWJMS and is required in the third medical school year. While the majority of neurology clerks at our institution are third-year medical students, a small group of fourth-year students also participate annually. In the academic year 2019\u0026ndash;2020, there were 20 fourth-year medical students enrolled in the rotation.\u003c/p\u003e \u003cp\u003eDuring the height of the COVID-19 pandemic, in April 2020, the clerkship was converted into a 2-week virtual rotation which employed the flipped classroom model with a total of 22 hours of synchronous case-based interactive learning and a recommended 8\u0026ndash;10 hours of asynchronous preparatory work per flipped classroom paradigm. A sample schedule is outlined in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Following the virtual rotation, an additional 2 weeks of clinical in-person experience were scheduled to be completed at a later date. Students were given the choice to take the NBME shelf exam at the completion of the virtual clerkship or to delay testing until after the additional clinical training. Neurology clerkship leadership, Student Affairs Deans, and Cognitive Skills Division were available to advise students on the timing of the exam per our institutional protocol. At the beginning and conclusion of the virtual clerkship, all EBC students were asked to complete an optional anonymous online survey, which collected their demographics and assessed with a 5-point Likert scale student perception of the effectiveness of the virtual curriculum in learning specific skills and preparedness for the in-person Neurology clerkship overall. The survey also solicited open-ended comments and a list of common learning resources used by students for shelf examination preparation. Demographics data was not available for EAC cohort.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSample schedule of a 2-week virtual rotation\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWEEK 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMonday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTuesday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eWednesday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eThursday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eFriday\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAsynchronous assignments\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eWatch topic review video on approach to ischemic and hemorrhagic stroke, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRead topic review primer on neuromuscular disorders, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eWatch topic review video on multiple sclerosis, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eWatch topic review video on multiple sclerosis, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSynchronous sessions\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOrientation and review of neurological localization and examination (4 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCases: Ischemic and Hemorrhagic Stroke (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCases: Neuromuscular Disorders (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCases: Multiple Sclerosis and Neuroimmunology (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCases: Movement Disorders (2 hours)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWEEK 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMonday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTuesday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eWednesday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eThursday\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eFriday\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAsynchronous assignments\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWatch topic review video on neuro-prognostication and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eWatch topic review video on approach to seizures and epilepsies, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eWatch topic review video on differential diagnosis of and approach to coma, and complete 10-MCQ vignette-based quiz (1\u0026ndash;2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSynchronous sessions\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSimulation: Neuro-prognostication and delivering bad news (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCases: Seizures and Epilepsies (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCases: Neurological Emergencies, Coma (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCases: Headache Disorders (2 hours)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCases: Neuro-infectious Diseases (2 hours)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eThe mean shelf exam scores of the EBC cohort of students who completed the NBME examination immediately after the 2-week virtual rotation in April to June of 2020 were compared by t-test (independent sample) with the scores of the sequentially enrolled historical cohorts completing the shelf examination immediately following a traditional 4-week in-person neurology clerkship (EAC cohort) \u0026ndash; from July 2018 through March 2020, during which time the clerkship structure remained relatively unchanged. Three secondary analyses were conducted using t-tests. The shelf exam scores of EBC students were compared to (1) scores of EAC students within the same academic year (between July 2019 and March 2020), (2) scores of EAC students within the same quarter of the prior academic year (April 2019 to June 2019), and (3) scores of \u0026ldquo;exam after delayed clinical\u0026rdquo; (EADC) students who completed the 2-week virtual rotation but delayed testing until after the supplementary 2-week rotation. We also compared EBC cohort results with the Neurology shelf examination national means published by NBME for the academic year 2019\u0026ndash;2020.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003e68 students participated in the 2-week virtual clerkship from April 2020 to June 2020, 65 of which (96%) were third-year students. Demographic data was available for 51 of 68 students (75%) and is summarized in Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e. Median age of the students in this cohort was 24\u0026ndash;26 years, and 74.5% identified as female. 58.8% identified as non-white, with \u0026ldquo;Asian\u0026rdquo; most frequently selected (43.1%). The majority (94.1%) were third-year medical students, and 21.6% of students indicated that they completed or were enrolled in an advanced degree program (master\u0026rsquo;s or Ph.D.). Only 9.8% agreed or strongly agreed that their neurological knowledge from preclinical years was sufficient to assess patients with neurological disorders. Educational resources most commonly used daily or at least weekly in clerkships prior to this virtual rotation were question banks (used by 98%), topic review articles in reputable journals or UpToDate.com (used by 75%), review books specifically written for clerks (used by 67.3%), internet search engine queries (used by 63.4%). PubMed queries and primary literature as well as traditional book chapters were used infrequently, by 38.5% and 15.4% respectively.\u003c/p\u003e\n\u003cp\u003eThe results are summarized in Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. 37 of the 68 students (54%) chose to take the NBME shelf exam after completion of the 2-week virtual clerkship and prior to starting the in-person rotation. The median time from the conclusion of the virtual rotation to the shelf examination was 14 days (IQR 14\u0026ndash;37). The mean shelf exam score of this EBC cohort was 81.8 (SD 6.7). The mean shelf exam score of the 284 students who completed the traditional neurology clerkship from July 2018 to March 2020 (EAC cohort), was 81.8 (SD 6.5). In the analysis of the primary outcome, there was no statistically significant difference between the mean shelf exam scores of the EBC and EAC groups (p value of 0.96). In the prespecified subgroup analyses, there was no statistically significant differences between the mean shelf exam scores of the EBC cohort, the subgroup of EAC cohort that tested within the same academic year (81.8\u0026thinsp;\u0026plusmn;\u0026thinsp;6.7 versus 81.5\u0026thinsp;\u0026plusmn;\u0026thinsp;6.5, p\u0026thinsp;=\u0026thinsp;0.90), the subgroup of the EAC cohort that tested within the same quarter of the prior academic year (81.8\u0026thinsp;\u0026plusmn;\u0026thinsp;6.7 versus 82.4\u0026thinsp;\u0026plusmn;\u0026thinsp;6.6, p\u0026thinsp;=\u0026thinsp;0.65), and the EADC cohort that delayed testing until the supplementary clinical experience (81.8\u0026thinsp;\u0026plusmn;\u0026thinsp;6.7 versus 82.7\u0026thinsp;\u0026plusmn;\u0026thinsp;6.5, p\u0026thinsp;=\u0026thinsp;0.54).\u003c/p\u003e\n\u003cp\u003eNeurology shelf examination national mean for the academic year 2019\u0026ndash;2020 published by NBME for 13,098 students (80.5\u0026thinsp;\u0026plusmn;\u0026thinsp;7.5) did not differ qualitatively from the EBC mean. The study was adequately powered (alpha 0.05, power 0.80, 1 tail) assuming that an exam score difference of 4 is meaningful (standard error of measurement of national shelf exam score distribution published by NBME is 4).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2. Demographics of the EBC and EAC cohort\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"593\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eEBC cohort: Students that completed 2-week virtual rotation (n=68)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eEAC cohort: Students that completed 4-week traditional rotation\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e(n=284)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eDates of rotation\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003eApril 2020 to June 2020\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003eJuly 2018 to March 2020\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eAge ranges\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u0026lt;21\u003c/p\u003e\n \u003cp\u003e21-23\u003c/p\u003e\n \u003cp\u003e24-26\u003c/p\u003e\n \u003cp\u003e27-29\u003c/p\u003e\n \u003cp\u003e\u0026gt;29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e51.0%\u003c/p\u003e\n \u003cp\u003e33.3%\u003c/p\u003e\n \u003cp\u003e15.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e43.5%\u003c/p\u003e\n \u003cp\u003e44.5%\u003c/p\u003e\n \u003cp\u003e12%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eGender\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eFemale\u003c/p\u003e\n \u003cp\u003eMale\u003c/p\u003e\n \u003cp\u003eOther\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e74.5%\u003c/p\u003e\n \u003cp\u003e25.5%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e54.0%\u003c/p\u003e\n \u003cp\u003e46.0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eEthnicity\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eAsian\u003c/p\u003e\n \u003cp\u003eBlack or African American\u003c/p\u003e\n \u003cp\u003eWhite\u003c/p\u003e\n \u003cp\u003eAmerican Indian or Alaska Native\u003c/p\u003e\n \u003cp\u003eHispanic, Latino, or of Spanish origin\u003c/p\u003e\n \u003cp\u003eNative Hawaiian or Other Pacific\u003c/p\u003e\n \u003cp\u003eIslander\u003c/p\u003e\n \u003cp\u003eOther, or prefer not to say\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e43.1%\u003c/p\u003e\n \u003cp\u003e9.8%\u003c/p\u003e\n \u003cp\u003e41.2%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e5.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e34.5%\u003c/p\u003e\n \u003cp\u003e9.5%\u003c/p\u003e\n \u003cp\u003e48.3%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e5.0%\u003c/p\u003e\n \u003cp\u003e0%\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e2.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"37.268128161888704%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eHighest level of education (multiple answers are possible)\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e3\u003csup\u003erd\u003c/sup\u003e year medical student\u003c/p\u003e\n \u003cp\u003e4\u003csup\u003eth\u003c/sup\u003e year medical student\u003c/p\u003e\n \u003cp\u003eCompleted or currently enrolled in a masters degree program\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eCompleted or currently enrolled in a Ph.D. program.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"25.29510961214165%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e94.1%\u003c/p\u003e\n \u003cp\u003e3.9%\u003c/p\u003e\n \u003cp\u003e1.9%\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e7.8%\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"37.436762225969645%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eData not available\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eTable 3. Mean Shelf Exam Scores\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"617\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.479740680713128%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003eEBC cohort that tested after 2-week virtual rotation \u003cem\u003eonly\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003eEAC cohort\u003c/p\u003e\n \u003cp\u003e(Shelf exam after traditional 4-week rotation)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003eSubgroup of EAC cohort that tested in 2019-2020 academic year\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.696920583468396%\" valign=\"top\"\u003e\n \u003cp\u003eSubgroup of EAC cohort that tested in 4\u003csup\u003eth\u003c/sup\u003e quarter of 2018-2019 academic year\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003eEADC cohort that tested after 2-week virtual and 2-week clinical rotations\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.779578606158833%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.479740680713128%\" valign=\"top\"\u003e\n \u003cp\u003eDates\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003eApril 2020 to June 2020\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003eJuly 2018 to March 2020\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003eJuly 2019 to March 2020\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.696920583468396%\" valign=\"top\"\u003e\n \u003cp\u003eApril 2019 to June 2019\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003eAcademic year 202-2021\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.779578606158833%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.479740680713128%\" valign=\"top\"\u003e\n \u003cp\u003en\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e37\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e284\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e119\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.696920583468396%\" valign=\"top\"\u003e\n \u003cp\u003e55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e31\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.779578606158833%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.479740680713128%\" valign=\"top\"\u003e\n \u003cp\u003eMean shelf Score (SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e81.8 (6.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e81.8 (6.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u003cem\u003e*p\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;value\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e81.6 (6.4)\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e*\u003cem\u003ep\u003c/em\u003e value\u003c/strong\u003e 0.90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.696920583468396%\" valign=\"top\"\u003e\n \u003cp\u003e82.4 (6.6)\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e*\u003cem\u003ep\u003c/em\u003e value\u003c/strong\u003e 0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e80.5 (7.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.779578606158833%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003e*\u003cem\u003ep\u003c/em\u003e value\u003c/strong\u003e 0.54\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.479740680713128%\" valign=\"top\"\u003e\n \u003cp\u003eMedian delay to exam (IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e14 (14-37)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.66936790923825%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.696920583468396%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.807131280388979%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"12.641815235008103%\" valign=\"top\"\u003e\n \u003cp\u003e234 (150-290)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.779578606158833%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e* p-value for comparison to EBC cohort\u003c/div\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e41 of 68 students completed the survey at the conclusion of the 2-week virtual clerkship, and the majority (73%) reported daily or at least weekly use of the resources prespecified by the rotation. Most students felt that during the virtual rotation, an appropriate amount of time was devoted to assigned homework (85% of responders), live case-based didactics (81% of responders), and NBME shelf exam review (74% of responders).\u0026nbsp;\u003c/div\u003e\n\u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe unprecedented limitations the COVID-19 pandemic placed on medical student clinical learning from direct patient care, provided us the unique opportunity to investigate the effect of entirely virtual clinical instruction on NBME neurology shelf exam performance. Our results indicate that shelf exam performance did not differ between the students who completed a traditional 4-week in-person versus a 2-week virtual clerkship. This is consistent with prior studies of student performance in other virtual rotations, including virtual radiology, emergency medicine, surgery, and pediatrics.[\u003cspan additionalcitationids=\"CR19 CR20\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] The effectiveness of online teaching in medical education was validated[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], and Durfee and colleagues, in particular, utilized an online flipped classroom model as well as larger group didactic lectures to confirm effectiveness of virtual radiology education on the student performance of the standardized Alliance of Medical Student Education in Radiology (AMSER) final examination[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Redinger and colleagues found similar NBME shelf exam performance among the cohort who completed an entirely virtual 4-week emergency medicine course as compared to a traditional elective within the same academic year, concluding that the fund of knowledge in emergency medicine was not attenuated in the virtual rotation experience[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Few papers have shown a negative impact of reducing clerkship length on subject exam performance.[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] While the consensus regarding the optimal length of medical school clerkships may remain ambiguous[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], our results support the conclusion that a shortened 2-week virtual rotation does not negatively impact the acquisition of a requisite knowledge base as measured by the NBME shelf examination.\u003c/p\u003e \u003cp\u003eThere were several important limitations to this study. The single center retrospective design and voluntary participation in the EBC cohort may have introduced selection bias, the probability of which is diminished by: 1) the similar results obtained in the EBC and the national NBME cohorts, and 2) the absence of statistically significant Step 1 score disparity between the EBC and EADC cohorts. The time elapsed between knowledge acquisition and testing may have influenced the results. In a traditional 4-week clerkship, students tested on the last day of the rotation, while the median time to the shelf exam for the EBC cohort was 14 days (IQR 14\u0026ndash;37), potentially affording more dedicated study time to this group. However, the EADC students that tested after the 2-week virtual and supplementary 2-week in-person clinical rotation had a median delay from the end of virtual rotation to the shelf exam of 259 days (IQR 196\u0026ndash;315) and yet performed similarly.\u003c/p\u003e \u003cp\u003eThe increase of synchronous education from a total of 12 hours (58% passive learning via traditional lectures) in the EAC cohort to the 22 hours of 100% case-based interactive learning may have contributed to the stable NBME shelf exam performance in our study. However, the case-based sessions in the virtual rotation were designed to address the most salient clinical scenarios appropriate to the bedside training of Neurology clerks as outlined by the Consortium of Neurology Clerkship Directors of the American Academy of Neurology[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] and were not specifically tailored to strengthen the shelf exam performance.\u003c/p\u003e \u003cp\u003eWe must also acknowledge the possibility that shelf examination performance may depend largely on students\u0026rsquo; individual preparation methods rather than formal instruction, although the majority (73%) of the EBC cohort self-reported frequent use of the resources prespecified by the virtual rotation and indicated a trend towards decreased reliance on other commonly used resources (question bank use declined from 98\u0026ndash;84%, the use of topic review articles in reputable journals or UpToDate.com declined from 75\u0026ndash;66%, and review book use declined from 67\u0026ndash;59%).\u003c/p\u003e \u003cp\u003eOur findings raise important questions concerning the relationship between the shelf examination and clinical learning. Multiple factors are associated with performance on shelf exams including USMLE Step 1 scores, number of practice exams, amount of dedicated study time, and number of prior clerkships completed.[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] In our study, the 2-week in-person clinical training was delayed by a median of 234 days (IQR 150\u0026ndash;190) after the conclusion of the virtual rotation, which limited the possibility and validity of correlating shelf scores to traditional measures of clinical skills. However, additional clinical training does not seem to improve shelf examination scores. Therefore, decoupling clinical experiences from knowledge acquisition and the evaluation of that knowledge base by means of NBME shelf examination is not likely to disadvantage learners with respect to shelf examination performance in the Neurology clerkship.\u003c/p\u003e \u003cp\u003eFinally, our study focused on the Neurology NBME shelf examination, limiting generalizability of our results to other clerkships.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe NBME Neurology shelf examination scores of students who chose to test after a 2-week virtual rotation utilizing the flipped classroom model did not differ from the scores of students who tested after a 4-week traditional in-person neurology clerkship. Furthermore, no significant differences were observed between the shelf examination performance after the virtual rotation alone versus the traditional rotation within the same academic year, and within a matched quarter of the prior academic year. This observation also held true for students who completed the virtual rotation and chose to postpone testing until after supplementary clinical experience. These findings support the conclusion that a shortened 2-week flipped-classroom virtual rotation does not adversely affect the NBME shelf examination performance among neurology clerks.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eAAMC - Association of American Medical Colleges\u003c/p\u003e\n\u003cp\u003eEAC \u0026ndash; Exam after clinical\u003c/p\u003e\n\u003cp\u003eEBC \u0026ndash; Exam before clinical\u003c/p\u003e\n\u003cp\u003eEADC - Exam After Delayed Clinical\u003c/p\u003e\n\u003cp\u003eNBME - National Board of Medical Examiners\u003c/p\u003e\n\u003cp\u003eRWJMS - Rutgers Robert Wood Johnson Medical School\u003c/p\u003e\n\u003cp\u003eUSMLE - United States Medical Licensing Examination\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cu\u003eEthics approval\u003c/u\u003e: The study was approved by the Rutgers University Institutional Review Board.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eConsent to participate:\u003c/u\u003e Formal consent waiver was approved by the Rutgers University Institutional Review Board since study involves deidentified anonymized collection and aggregate analysis of data collected for the purposes of quality assurance of the curriculum and presents minimal or no risk to the study participants.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eConsent for publication\u003c/u\u003e: Not applicable. The manuscript does not contain any individual person\u0026rsquo;s data.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eAvailability of data and materials:\u0026nbsp;\u003c/u\u003eInvestigators may request access to anonymized individual patient data and redacted study documents including raw datasets, analysis-ready datasets, statistical analysis plan, study methods, and dataset specifications. When approved, all documents may be accessed for a predetermined time of 12 months.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eCompeting interests\u003c/u\u003e: IR: No relevant competing interests to declare; GP: No relevant competing interests to declare; RM: No relevant competing interests to declare; HF: No relevant competing interests to declare; PP: No relevant competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eFunding:\u003c/u\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eThis study was not funded.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eAuthor\u0026rsquo;s contributions\u003c/u\u003e: IR wrote the main manuscript and prepared the tables. GP made substantial revisions to the manuscript and statistics. HF performed background research and contributed to the \u0026ldquo;background\u0026rdquo; section and collated the references. RM made substantial revisions to the manuscript and assisted with statistics. PP contributed to the \u0026ldquo;discussion\u0026rdquo; and \u0026ldquo;conclusion\u0026rdquo; sections.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eAcknowledgements\u003c/u\u003e: None.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eAuthor\u0026rsquo;s information\u003c/u\u003e: Please see below.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eSafdieh, J.E., et al., \u003cem\u003eA dozen years of evolution of neurology clerkships in the United States: Looking up.\u003c/em\u003e Neurology, 2018. \u003cstrong\u003e91\u003c/strong\u003e(15): p. e1440-e1447.\u003c/li\u003e\n\u003cli\u003eExaminers, N.M.B.o. \u003cem\u003eCharacteristics of Clinical Clerkships: NBME Survey Results. \u003c/em\u003e. 2022; Available from: www.nbme.org.\u003c/li\u003e\n\u003cli\u003eMonrad, S.U., et al., \u003cem\u003eDoes Reducing Clerkship Lengths by 25% Affect Medical Student Performance and Perceptions?\u003c/em\u003e Acad Med, 2018. \u003cstrong\u003e93\u003c/strong\u003e(12): p. 1833-1840.\u003c/li\u003e\n\u003cli\u003eBhatia, N.D., et al., \u003cem\u003eCutting too deep? Assessing the impact of a shorter surgery clerkship on students\u0026apos; clinical skills and knowledge.\u003c/em\u003e Am J Surg, 2014. \u003cstrong\u003e207\u003c/strong\u003e(2): p. 209-12.\u003c/li\u003e\n\u003cli\u003eBostwick, J.M. and C. Alexander, \u003cem\u003eShorter psychiatry clerkship length is associated with lower NBME psychiatry shelf exam performance.\u003c/em\u003e Acad Psychiatry, 2012. \u003cstrong\u003e36\u003c/strong\u003e(3): p. 174-6.\u003c/li\u003e\n\u003cli\u003eCase, S.M., D.R. Ripkey, and D.B. Swanson, \u003cem\u003eThe effects of psychiatry clerkship timing and length on measures of performance.\u003c/em\u003e Acad Med, 1997. \u003cstrong\u003e72\u003c/strong\u003e(10 Suppl 1): p. S34-6.\u003c/li\u003e\n\u003cli\u003eGary, N.E. and G.C. Rosevear, \u003cem\u003eEffect of reduction in length of third-year clerkships on students\u0026apos; academic performance.\u003c/em\u003e J Med Educ, 1988. \u003cstrong\u003e63\u003c/strong\u003e(5): p. 406-7.\u003c/li\u003e\n\u003cli\u003eOuyang, W., M.M. Cuddy, and D.B. Swanson, \u003cem\u003eUS Medical Student Performance on the NBME Subject Examination in Internal Medicine: Do Clerkship Sequence and Clerkship Length Matter?\u003c/em\u003e J Gen Intern Med, 2015. \u003cstrong\u003e30\u003c/strong\u003e(9): p. 1307-12.\u003c/li\u003e\n\u003cli\u003eSampat, A., et al., \u003cem\u003eNeurology Clerkship: Predictors of Objective Structured Clinical Examination and Shelf Performance.\u003c/em\u003e J Med Educ Curric Dev, 2019. \u003cstrong\u003e6\u003c/strong\u003e: p. 2382120519862782.\u003c/li\u003e\n\u003cli\u003eSafdieh, J.E., et al., \u003cem\u003eCore curriculum guidelines for a required clinical neurology experience.\u003c/em\u003e Neurology, 2019. \u003cstrong\u003e92\u003c/strong\u003e(13): p. 619-626.\u003c/li\u003e\n\u003cli\u003ePrescott, J.E. \u003cem\u003eImportant guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. \u003c/em\u003e. 2020; Available from: https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak.\u003c/li\u003e\n\u003cli\u003eWhelan, A.J. \u003cem\u003eUpdated interim guidance for medical students\u0026apos; participation in patient care during the coronavirus (COVID-19) outbreak. \u003c/em\u003e. 2020; Available from: https://www.aamc.org/news-insights/press-releases/updated-interim-guidance-medical-students-participation-patient-care-during-coronavirus-covid-19.\u003c/li\u003e\n\u003cli\u003eWhelan, A.J. \u003cem\u003eUpdated guidance for medical students\u0026apos; roles in direct patient care. \u003c/em\u003e. 2020; Available from: https://www.aamc.org/news-insights/press-releases/covid-19-updated-guidance-medical-students-roles-direct-patient-care.\u003c/li\u003e\n\u003cli\u003eRose, S., \u003cem\u003eMedical Student Education in the Time of COVID-19.\u003c/em\u003e JAMA, 2020. \u003cstrong\u003e323\u003c/strong\u003e(21): p. 2131-2132.\u003c/li\u003e\n\u003cli\u003eCooke, M., D.M. Irby, and B.C. O\u0026apos;Brien, \u003cem\u003eEducating Physicians: A Call for Reform of Medical School and Residency\u003c/em\u003e. 1st ed. 2010: Jossey-Bass.\u003c/li\u003e\n\u003cli\u003eSkochelak, S.E. and S.J. Stack, \u003cem\u003eCreating the Medical Schools of the Future.\u003c/em\u003e Acad Med, 2017. \u003cstrong\u003e92\u003c/strong\u003e(1): p. 16-19.\u003c/li\u003e\n\u003cli\u003eHew, K.F. and C.K. Lo, \u003cem\u003eFlipped classroom improves student learning in health professions education: a meta-analysis.\u003c/em\u003e BMC Med Educ, 2018. \u003cstrong\u003e18\u003c/strong\u003e(1): p. 38.\u003c/li\u003e\n\u003cli\u003eDurfee, S.M., et al., \u003cem\u003eMedical Student Education Roadblock Due to COVID-19: Virtual Radiology Core Clerkship to the Rescue.\u003c/em\u003e Acad Radiol, 2020. \u003cstrong\u003e27\u003c/strong\u003e(10): p. 1461-1466.\u003c/li\u003e\n\u003cli\u003eHanson, A.E., et al., \u003cem\u003eDecline in Pediatric Shelf Examination Performance During COVID-19.\u003c/em\u003e Cureus, 2021. \u003cstrong\u003e13\u003c/strong\u003e(10): p. e18453.\u003c/li\u003e\n\u003cli\u003ePrigoff, J., M. Hunter, and R. Nowygrod, \u003cem\u003eMedical Student Assessment in the Time of COVID-19.\u003c/em\u003e J Surg Educ, 2021. \u003cstrong\u003e78\u003c/strong\u003e(2): p. 370-374.\u003c/li\u003e\n\u003cli\u003eRedinger, K.E. and J.D. Greene, \u003cem\u003eVirtual Emergency Medicine Clerkship Curriculum during the COVID-19 Pandemic: Development, Application, and Outcomes.\u003c/em\u003e West J Emerg Med, 2021. \u003cstrong\u003e22\u003c/strong\u003e(3): p. 792-798.\u003c/li\u003e\n\u003cli\u003eDominguez, M., et al., \u003cem\u003eA Neurology Clerkship Curriculum Using Video-Based Lectures and Just-in-Time Teaching (JiTT).\u003c/em\u003e MedEdPORTAL, 2018. \u003cstrong\u003e14\u003c/strong\u003e: p. 10691.\u003c/li\u003e\n\u003cli\u003eVilla, S., et al., \u003cem\u003eAn Emergency Medicine Virtual Clerkship: Made for COVID, Here to Stay.\u003c/em\u003e West J Emerg Med, 2021. \u003cstrong\u003e23\u003c/strong\u003e(1): p. 33-39.\u003c/li\u003e\n\u003cli\u003eRipkey, D.R., S.M. Case, and D.B. Swanson, \u003cem\u003eIdentifying students at risk for poor performance on the USMLE Step 2.\u003c/em\u003e Acad Med, 1999. \u003cstrong\u003e74\u003c/strong\u003e(10 Suppl): p. S45-8.\u003c/li\u003e\n\u003cli\u003eRipkey, D.R., S.M. Case, and D.B. Swanson, \u003cem\u003ePredicting performances on the NBME Surgery Subject Test and USMLE Step 2: the effects of surgery clerkship timing and length.\u003c/em\u003e Acad Med, 1997. \u003cstrong\u003e72\u003c/strong\u003e(10 Suppl 1): p. S31-3.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Flipped classroom, curriculum evaluation, neurology, nervous system, instructional design, NBME shelf examination","lastPublishedDoi":"10.21203/rs.3.rs-4631444/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4631444/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere is conflicting evidence whether decreased clerkship duration is associated with reduced NBME shelf examination performance. We hypothesized that scores would remain stable for students in a shortened 2-week flipped classroom-based virtual rotation as compared to the traditional 4-week Neurology clerkship.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere is conflicting evidence whether decreased clerkship duration is associated with reduced NBME shelf examination performance. We hypothesized that Neurology shelf exam scores would remain stable for students in a shortened 2-week flipped classroom-based virtual rotation as compared to the traditional 4-week Neurology clerkship with direct patient care supplemented by didactics.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis is a retrospective observational cohort study. In April 2020, the neurology clerkship at Rutgers Robert Wood Johnson Medical School switched from a traditional 4-week in-person clinical rotation supplemented by didactics to a 2-week flipped classroom-based virtual-only curriculum decoupled from a supplementary 2-week in-person clinical rotation to be completed later. Students were offered the opportunity to test after the 2-week virtual curriculum. NBME shelf scores for these “exam before clinical rotation” (EBC) students over the final quarter of the academic year 2019–2020 were compared to those of the “exam after clinical” (EAC) students undergoing the 4-week traditional rotation from July 2018 to March 2020.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e321 students completed the shelf exam between July 2018 and June 2020, of which 284 students comprised the EAC cohort and 37 students, the EBC cohort. Mean scores did not differ between the EBC and EAC cohorts (81.8 ± 6.7 versus 81.8 ± 6.5, p = 0.96). Additional analyses showed no significant differences in the performance of EBC students as compared to the EAC students that tested within the same academic year 2019–2020 or within the matched quarter of the prior academic year 2018–2019, and students completing the virtual rotation who delayed testing after the clinical experience.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTesting after a 2-week flipped classroom-based virtual curriculum in Neurology without additional clinical training did not negatively affect neurology NBME shelf scores. Decoupling clinical experiences from knowledge acquisition and the evaluation of that knowledge base by means of NBME shelf examination is not likely to disadvantage learners with respect to shelf examination performance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Trial Number:\u003c/strong\u003e Not applicable. This is not a clinical trial.\u003c/p\u003e","manuscriptTitle":"Impact of a 2-week Flipped Classroom Virtual Neurology Clerkship versus a Traditional 4- week Rotation on NBME Shelf Exam Scores","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-24 17:53:46","doi":"10.21203/rs.3.rs-4631444/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-07-04T03:04:32+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-02T10:27:52+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-02T10:27:06+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Education","date":"2024-06-24T16:15:27+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c84799c3-81d6-43e1-80d6-7225ac9e62bd","owner":[],"postedDate":"July 24th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-11-11T16:08:18+00:00","versionOfRecord":{"articleIdentity":"rs-4631444","link":"https://doi.org/10.1186/s12909-024-06239-8","journal":{"identity":"bmc-medical-education","isVorOnly":false,"title":"BMC Medical Education"},"publishedOn":"2024-11-05 15:57:07","publishedOnDateReadable":"November 5th, 2024"},"versionCreatedAt":"2024-07-24 17:53:46","video":"","vorDoi":"10.1186/s12909-024-06239-8","vorDoiUrl":"https://doi.org/10.1186/s12909-024-06239-8","workflowStages":[]},"version":"v1","identity":"rs-4631444","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4631444","identity":"rs-4631444","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}