The impact of deliberate reflection with WISE-MD™ modules on surgical clerkship students' critical thinking: a prospective, randomized controlled pilot study

Critical thinking underlies at least three of the Association of American Medical Colleges’ (AAMC) 13-core entrustable professional activities (EPA) for entering residency. 1 Critical thinking is required for physicians to competently and independently provide patient care. 2 While critical-thinking ability is clearly related to quality and safety in health care, 3 defining and measuring it continue to be a challenge for health professions’ educators, including medical faculty. 4 – 7 Defining critical thinking has been elusive for most of the recent century. 8 There is no consensus for an approved definition in the medical literature, 9 nor is there agreement on terminology to define the process and little evidence for best practices for teaching, measuring, and evaluating critical thinking. 4 – 6 Others question whether critical thinking can even be taught. 4 What it means to think critically may vary by discipline, practice settings, and contexts. 10 Critical thinking may be viewed as a variety of ways to think with various styles of reasoning, 11 and in the health sciences’ literature, critical thinking is often used interchangeably with clinical thinking, clinical reasoning, and diagnostic reasoning. 10 Because critical thinking is largely conceptual, measurement must be inferred from observable behaviors. 6 Educational strategies to reveal actual thought processes may include a standardized list of questions – necessitating verbal or written evidence for analysis as a requirement to enhance metacognition and make visible a student’s thought processing. 4 Huang et al 4 reported the following strategies for teaching critical thinking: 1) slowing down the pace of the learning process to enable students to digest and apply knowledge, 2) actively engaging the learner in tasks that require problems to be solved, 3) compelling students to justify how they arrived at decisions, 4) making thinking explicit, and 5) requiring self-reflection on the part of the learner. Given the literature and the above noted gaps, the authors wanted to test the integration of deliberate reflection (DR) with Web Initiative in Surgical Education (WISE-MD™) modules as a means to increase critical-thinking ability. Due to the timing of courses, semesters, and per the protocol submitted and approved as exempt by the institutional review board, the methodology was first tested with nurse practitioner students in an advanced health assessment course 39 followed by implementation with medical students during their surgical clerkship (SC). This study evaluated critical-thinking outcomes of SC students by adding metacognitive DR guidance to the learning strategy with WISE-MD™ simulation modules. The authors hypothesized that SC students in the intervention group who were exposed to the DR guide would show a significantly greater pre- to postintervention increase in critical-thinking ability than students in the control group who had no DR guidance. 39 The next section provides additional details regarding the development of the methodology. WISE-MD™ is a series of 35 case-based online teaching modules developed to fill in the gaps in surgical education created by shorter hospital stays along with more of the pre-and postoperative care occurring in outpatient services. 12 , 13 The American College of Surgeons and the Association of Surgical Education endorsed the WISE-MD™ modules, which were designed to develop clinical reasoning in medical students while seeking consistent, high-quality learning environments to ensure clinical competence. 12 , 13 , 39 Among the module topics are those particularly germane to the SC such as appendicitis, breast cancer, gall bladder disease, thyroid disease, and hernias. 12 , 13 The modules were created for independent study illustrated with video and animation using best practices for multimedia design. 12 , 13 , 39 Lasting ~1 hour, each module is introduced with a “fundamentals” section and depicts the patient’s experiences and interactions with the physician from initial presentation, history taking, physical examination, laboratory tests and radiological imaging to preoperative preparation, surgery, and recovery. 12 , 13 , 39 Professionalism and communication are emphasized throughout each module, which also includes a summary and key findings from the case. 12 , 13 The surgical procedure is presented with a graphic depiction alongside the actual surgery process overview, which is very helpful for medical student visualization of the virtual along with the actual surgery. 39 For the remainder of the document, the authors refer to the WISE-MD™ modules as the WISE modules. 39 Although the WISE modules have been used in medical education since 1998 and are used by >200 medical schools nationally and internationally, 12 , 13 little has been published regarding their use with medical students. One study found that medical students who viewed the WISE modules trended toward better knowledge and clinical reasoning than students who did not view the modules. 14 , 39 To be able to think critically, students must learn to routinely and critically examine their own thinking. 6 , 15 Requiring medical students to examine their own thinking through thoughtful reflection is an important component of medical education 16 – 18 as reflection requires the ability to think critically. 19 Reflective practice also promotes professional identity transformation from medical student to physician 16 and seeks to improve diagnostic accuracy. 20 , 21 The Accreditation Council for Graduate Medical Education (ACGME) core competencies and milestones 22 and the AAMC’s Physician Competency Reference Set (PCRS) 23 require physician trainees to reflect upon and analyze practice experiences. 17 Reflective practice can be fostered/enhanced when medical students perceive the case/situation as real, there is some conflict, critical questions are raised, and there is a structured process for reflecting. 24 Guided reflection is considered a key element of professional identity formation from medical student to physician 16 with an interactional aspect proposed to develop that identity. 25 Furthermore, clinical reasoning may be enhanced by the process of “think aloud”, which occurs when a student verbalizes his or her thoughts while doing an assignment. 26 Siddiqui 27 found think aloud to be valuable in identifying medical students’ critical-thinking strengths and weaknesses during their ICU rotation. Thus, given the potential value of both reflection and think aloud to make critical thinking overt and for the purposes of this study, author MQ created the term DR to reinforce the notion that the reflective thinking process is an overt skill that compliments the skill set of “deliberate practice” defined by Ericsson. 28 DR as a metacognitive learning innovation is tested in this study. The DR process is introduced for a number of reasons. It implies that the reflective process must be made overt to enhance learning 39 as suggested by Croskerry who defines “cognitive forcing strategies” as a means of de-biasing and preventing diagnostic error. 26 , 39 Although similar to the metacognitive strategy known as self-explanation, DR differs in that it is not restricted to inferences, clarifications, justifications, or monitoring of behavior as is inherent in the definitions of self-explanation. 29 – 31 , 39 Rather, DR includes the integration of previous experiences with current experiences and the application of strategic knowledge about self and learning including awareness of affective components such as confidence. 40 DR incorporates mental representation (selective encoding, combination, and comparison) (A Kalet, New York University, email communication, May 2011.) 30 , 39 and considers the temporal aspects of reflection – before action, during action, and after action. 32 , 39 Finally, DR includes a think-aloud or “verbal report” strategy that has been used in debriefing and other thought process research strategies. 6 , 39 Believing that the value of learning through simulation lies in debriefing and reflection on the simulation experience 33 and that structured reflection improves learning outcomes, 28 , 31 , 34 – 37 , 39 the authors reasoned that without specific, systematic instructions for learner reflection or self-debriefing, some of the educational value of the video-based simulation in the WISE modules might be lost. 39 The authors further speculated that the personalized, real-time, self-debriefing/reflective component of DR for the SC students might improve the WISE module learning experience and outcomes related to working memory and critical thinking. 39 Therefore, for this study, the authors developed specific DR instructions to guide the SC learner to apply the principles of reflective awareness to surgical content in the WISE-MD™ modules with the goal of promoting learning 38 and enhancing critical-thinking and learning outcomes. 39

Of the 31 SC study participants, 15 (48%) participants dropped out of the study. Of the 16 (52%) remaining par ticipants, 7 participants failed to complete all parts of the study. Nine (29%) students completed the entire study – four females and five males. Participants’ mean age was 25.77 years (range =23–29 years). All nine completers had a baccalaureate degree, and one had achieved a master’s degree. Eight were Caucasian and one was of Middle Eastern descent. Of those completing, four were from the intervention group and five were from the control group. The control and intervention groups showed no difference in pre- and posttest free-thought critical-thinking outcomes. SC students in the intervention group demonstrated a higher level of critical thought when prompted by questions in the DR guide. 39 Table 3 provides examples of SC students’ DR narratives from either the thyroid module or the appendicitis module that are representative of the critical-thinking category/subcategory. During the free-thought steps (step 1 [pretest] and step 4 [posttest]) of methods, students in both the control and intervention groups predominantly verbalized in the categories of description, learning style, and occasionally their past experiences with the module topic. They described what they were seeing and hearing in the modules as an ongoing commentary about each section of the module. They also described what they liked/did not like (in terms of their learning styles) about the module, the narrator, and the interaction between the physician and the patient. Some evaluated or summarized the overall value of the module for them at the end of their free-thought recording. The free-thought narratives did not demonstrate clinical reasoning. Because of the large dropout rate (of the 31 SC learner participants, 15 [48%] participants dropped out of the study, and of the 16 [52%] remaining participants, 7 participants failed to complete all parts of the study), the authors sought to determine 1) the reasons for the high dropout/failure to complete rate and 2) the value of the modules to all students. All participants (31) were surveyed poststudy for qualitative feedback and quantitative feedback regarding the study and the modules; 15 (48%) of the 31 participants returned the survey, but not all 15 respondents responded to every question.

Despite no difference in unprompted outcomes between groups, the intervention group verbalized more thoughtful clinical decision-making when following the DR protocol. 39 The authors now believe that limiting the application of DR with only two modules was not sufficient for students to internalize a new way of thinking about clinical cases. We suggest that DR could be integrated throughout medical education as a means to reinforce learning with a defined model to promote critical thinking for clinical reasoning. It would be of particular interest to see how many guided DR modules it might take for students to begin to verbalize and record their critical-thinking processes without prompting from the requirements of the DR protocol. 39 Faculty must identify and test strategies that will help learners develop and enhance good critical-thinking skills. 39 The value of this study is that it takes a legitimate and sponsored tool in surgical education (WISE-MD™) and poses the toughest criteria of evaluation of an educational intervention, ie, does it affect the way we think? and not just what we learn (but how we learn it)? 39 Finding out the answer is significant and will require more resources before we arrive at a definitive answer. 39

The authors were disappointed that after having been exposed twice to the DR guidance, the performance of the intervention group on the last two modules did not continue to demonstrate the same high level of critical thinking as during the DR-guided modules (steps 2 and 3). Despite the lack of support for our hypothesis, we believe that DR has value and generates the following additional questions for consideration in future studies: 1) were we measuring the right concepts? 2) are two guided DRs insufficient for students to internalize, without the DR guidance, the same high level clinical reasoning process? and 3) would a formal debriefing process (online or face to face) have enhanced the students’ experience? 39 Ibiapina et al, 21 in their 2014 study of free reflection, modeled reflection, and cued reflection, found that with a higher amount of guidance, the structured reflection groups performed significantly better than the free-reflection group. These findings are consistent with our findings in that students with the guided/DR demonstrated a higher level of critical thinking versus unprompted free reflection. 39 We implemented the poststudy survey as an opportunity to “debrief ” the students in terms of what they did and did not find to be useful about the modules and the study. The feedback from the students in our study was similar to that found in other studies related to multimedia education enhancement strategies. Examples were as follows: 1) because of the fast pace of medical education, students become highly strategic in their selection of learning resources and unnecessary information is not appreciated; 40 it appears from the postsurvey that the SC students perceived the WISE modules to be in addition to other course expectations and students trusted their usual mode of studying and preparing for the examinations using their textbooks; and 2) the pressure of end of clerkship examination and the National Board of Medical Examiners’ (NBME) subject examination along with their perception of limited time to try out a new way of learning created barriers to their enthusiasm for the modules. 41 This is consistent with Yavner et al, 40 who emphasized the importance of faculty making clear the purpose and value of any additional online initiatives. As in the Ellaway et al 41 study, the students found the modules to be very useful when they had adequate time to prepare for a known specific case with which they were going to be involved. Consistent with Ellaway et al, 41 SC students complained that even during “downtime” in clinical settings, they were not permitted to use electronic devices. The students expressed that this would have been an ideal time to review modules, particularly just prior to an upcoming case. Both the SC students and those of Ellaway appreciated the split screen videos of the virtual and actual surgery. 41 The authors believe after completing and analyzing the data that requiring the DR guidance for every module may have increased the likelihood of students developing and internalizing a thought pattern or process that enhanced their clinical reasoning. 39 In addition, critical thinking as a response style may only become a habit if practiced over time. Perhaps students would need more than two opportunities for the application of guided (deliberate) reflection 24 , 39 suggesting that learning a new automatic pattern of thinking is enhanced by practice and observing expert modeling of critical thinking over time. 39 , 42 There are limitations to this study. The pilot study was conducted at a single academic institution in New England, with a limited number of students, only 29% of whom completed the study. The time frame for the study was short, and the intervention group was instructed to use DR guidance with only two of the WISE modules. The outcomes measured were author-defined rather than consensually defined constructs: 1) critical thinking, which lacks an expert consensus definition; 2) the process of DR, developed by the study team; and 3) researcher-developed, not previously tested categories of critical thinking. 39 Additionally, this study was a one-time brief commitment of the entire clerkship curriculum for third-year medical students, thus, only a short-term “injection” within the four-year medical education process. 24 Despite the small number, high dropout rate, and lack of support for the hypothesis, we believe that this study has implications for health professions’ curricula 39 and provides a forum for discussion in medical education. Introducing this approach to – and level of thinking to – students ab initio might be the best way to ensure that DR remains with them throughout their academic and professional lives. Implementing the DR approach to problem solving beginning in year 1 and then threading it throughout all 4 years of medical school might enhance the impact of DR on critical thinking. The feedback from the post-study survey was valuable in terms of a debriefing strategy. Some of the students’ feedback was similar to that found in other studies 41 that participants in both the intervention and control groups valued using the WISE modules. The authors believe that the WISE modules offer SC students a unique resource for them to follow a complete interaction between an experienced surgeon and patient for the core disease processes in surgery. Students can pace themselves and use the resource 24/7. They can start and stop the program at will depending on available time to offer a chance for reflection on the covered material. 39 The modules are a key resource for students preparing for oral examinations as well as objective structured clinical examinations. The program is comprehensive such that all aspects of any given clinical problem are covered from epidemiology to symptoms, diagnostic workup, and treatment algorithms so that the material serves as good review for the written NBME examination as well. Many of the modules have videos and other graphic materials that can help the students review anatomy and acquire knowledge of the procedure prior to participation in the surgical suite.

We recruited the participants from two different blocks of SC students at a New England medical school during the fall semester. The study was presented to students during the first week of their rotation. Thirty-one (72%) of the 43 students volunteered to participate and were randomly assigned to either the control group (n=16) or the intervention group (n=15). Written informed consent was obtained from the participants. Participants were given access to all the WISE modules and provided with a digital recorder to record their thoughts as they completed several preselected modules: abdominal aortic aneurysm (AAA), cholecystitis, appendicitis, and thyroid nodule. 39 Participants received no compensation, viewed the modules, and completed assigned activities on their own time outside of class. Although participation would not influence their grade, SC students were told that the WISE modules might be seen as an advantage in terms of their overall learning. 39 As outlined in the WISE study flowchart in Figure 1 , there were four steps to the procedure for both the control and intervention groups ranging from pretest to posttest. 39 The intervention and control groups were divided into two subgroups as close to equal size as possible. One subgroup viewed the AAA module, and the other subgroup viewed the cholecystitis module. 39 Each student was provided with a digital recorder and asked to freely record their thoughts while viewing each module. 39 For this exercise, all students (both intervention and control) were provided with a “free-thought” guide requesting them to record out loud whatever happens to come across their mind at least three times while viewing the modules. The control group reviewed the thyroid module without using a digital recorder. 39 While viewing the same module, the intervention group used a digital recorder to answer questions from the DR think-aloud group instructions, which can be found in Table 1 . 39 These instructions asked the intervention group to complete the specific DR exercises at specific time points. 39 The two groups followed the same procedures as described in step 2 for a second module, appendicitis. 39 The two modules (AAA and cholecystitis) used in the pretest were again used in the posttest but switched. 39 Using the same free-thought guide as described in step 1, students in both the control and intervention groups were asked to record their thoughts freely while viewing the module. 39 Students submitted their digital recorders to the study coordinator (KS) for analysis once they completed the study. 39 All 31 study participants were sent a brief follow-up anonymous survey about their experience as a study participant, with the WISE modules. 39 Data from students’ digital recordings were transcribed into word documents and imported into NVIVO™ Version 10. 39 All authors iteratively coded the transcripts. They reviewed the critical-thinking literature for potential categories to reach con sensus on the final critical-thinking coding. Further description of the development of the categories can be found in the work of Terrien et al. 39 Final analysis determined five categories with 10 subcategories, which can be found in Table 2 . 39 The study was exempted from review (14811) by the UMass Medical School Committee for the Protection of Human Subjects in Research.