The Effect of Incentives on the Use of Successive Relearning for Retaining Statistics and Epidemiology Concepts in a Medical Research Course

The Effect of Incentives on the Use of Successive Relearning for Retaining Statistics and Epidemiology Concepts in a Medical Research Course | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Effect of Incentives on the Use of Successive Relearning for Retaining Statistics and Epidemiology Concepts in a Medical Research Course Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3061046/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Successive relearning (SR) combines retrieval practice across spaced study sessions. In particular, students attempt to recall to-be-learned information (with feedback) during each session until all concepts are correctly recalled and then return to the same material to repeat retrieval practice on multiple spaced practice sessions. Thus, students must begin using this technique several weeks before an exam, which may decrease their motivation for using it. The main question for the present research is: Will providing students with a small amount of class credit increase their likelihood of engaging in SR? First-year medical students in a Principles of Medical Research course were provided with an SR program that included (a) virtual flashcards containing definitions of statistical concepts (e.g., levels of measurement, central tendency, odds ratios, etc) that students practiced retrieving, (b) feedback after each retrieval attempt, and (c) schedules for using each virtual flashcard stack across three spaced practice sessions. Students received incentives in terms of class credit to complete SR sessions for half of the content but no incentive for the other half. A delayed practice test was administered to evaluate the impact of SR on retention. Using SR (vs. not using it) did boost retention of the concepts. And, most important, credit had a major impact, with students completing over 90 percent of the SR sessions that were assigned to receive credit but under 10 percent of the SR sessions that were optional. Medical education Successive relearning Retrieval practice Spaced practice Figures Figure 1 Figure 2 Figure 3 Introduction Understanding epidemiology is critical for medical practitioners because it allows them to understand the development of diseases, to diagnose them, and to reduce risk for patients. Epidemiology is a challenging course – many students struggle to memorize and understand the numerous concepts and equations that comprise epidemiology and biostatistics. Thus, exploring techniques that boost students’ learning and retention of epidemiological concepts is an important endeavor for helping students to become effective practitioners. Toward this goal, we investigated the degree to which one technique -- successive relearning -- improves students’ learning of key concepts in descriptive statistics and epidemiology in a medical research course. As we describe in detail below, successive relearning (SR) is a specific combination of two effective learning techniques (retrieval practice and spaced practice) that have been shown to improve students’ retention (for a review, see Rawson & Dunlosky, in press). Unfortunately, many students do not use successive relearning, even when they are instructed on how to use it and provided with scaffolds to use it successfully. Moreover, very few investigations have explored the impact of SR in classroom settings, and none have focused on epidemiology. Accordingly, in the current research, we evaluated whether SR improves students’ retention of epidemiology and descriptive statistics, and most important, the degree to which a relatively small incentive increases its rate of use. In the remainder of this introduction, we explain the components of SR, briefly describe prior research, and discuss why incentive may improve usage. Successive relearning involves attempting to recall information from memory (retrieval practice), followed by presentation of the correct answer as feedback. Students continue to practice retrieval with feedback until the target information has been correctly recalled. Then, they repeat this process with the same information in multiple sessions that are spaced across time (spaced practice). SR is perhaps best illustrated by students using a physical deck of flashcards. While using flashcards, the student would read the prompt on one side of the first card (e.g. “What is an odds ratio?”), retrieve an answer from their memory (which may be correct or incorrect), and then refer to the back of the card to evaluate if their answer is correct. If correct, they would set that card to the side and continue with the rest of the items. If incorrect, they would put that card at the back of the stack and continue cycling through the deck until they recall the correct answer one time for each item. Importantly, to implement SR, students would also return to practice the same deck of items again during one or more spaced practice sessions. This example focuses on a physical deck of flashcards, but successive relearning can be instantiated in many ways, such as using class notes (e.g., Cornell notes) or using available flashcard apps on-line (for a review, see Dunlosky & O’Brien, 2020). Thus, the critical issue is not what platform students use (i.e., handheld flashcards or apps) but how they use them. That is, to use SR with fidelity, they must engage in its two components – retrieval practice until a criterion is met and then relearning the same items during multiple spaced sessions. Unfortunately, although many college students report using flashcards to prepare for exams, most use them exclusively for studying foreign language-English translation equivalents and many students do not instantiate SR when they use flashcards (for details, see Zung et al., 2022; Wissman et al, 2012). Does SR improve student achievement in classroom settings? Whereas SR shows much promise, only a few studies have been conducted to evaluate its efficacy (for a recent review, see Rawson & Dunlosky, 2022). For instance, in Janes et al. ( 2020 ), students in an undergraduate biopsychology course were provided with a SR program that included multiple assignments, each of which consisted of 8–10 to-be-learned items from the course, such as declarative concepts (e.g. What is an axon?), labels for structures (e.g. labels for parts of the brain), and steps of biological processes (e.g., various steps of the action potential). In each assignment, students were prompted to answer a single question and were instructed to type the correct answer. They then scored their answer and could restudy the correct answer. Students continued practicing retrieval with feedback until they correctly recalled an answer, and then it was dropped from practice during that session. They continued until they correctly recalled all answers in a given stack. They were instructed to relearn the items from each assignment three more times, with two to three days between each session. Importantly, the assignments were separated into two sets, each with different content. Students were slated to complete one set of assignments using SR, and the other set was optional; the two sets were counterbalanced across students. As a baseline control, other content did not appear in the assignments but was included on the high-stakes, in-class exam. For the successively relearned items, students were given access to the assignments and a schedule that indicated when each assignment should be completed. For the optional items, students were only provided access to the assignment, but not a schedule on when (and how often) each assignment should be completed. Exam performance was significantly higher for questions that targeted successively relearned content compared to optional content. Such a benefit of using SR in a classroom setting has also been demonstrated in introductory psychology (e.g., Higham et al., 2021 ; Rawson et al., 2013) and cognitive psychology (Badali & Greve, under review). Despite the potential effectiveness of SR, an open issue is whether students will use it without external incentives, because in prior research, students were typically given an external incentive to use SR. For example, in Rawson et al. (2013), students were paid for each of the SR sessions they had completed. In Higham et al. (2022), students received course credit (20% of their final grade) for completing the SR sessions. As mentioned above, in Janes et al. ( 2020 ), students were also instructed to complete SR assignments; for completing some, they received course credit, whereas other assignments were optional. Most students completed the assignments that were slated for course credit but only 39 percent attempted the optional assignments. Of those who did engage in optional assignments, on average, they completed significantly fewer sessions (8.9) than when they received course credit (17.4). One potential conclusion is that students are less likely to engage in SR without an external incentive (see also, Nevid & Gordon, 2018). Note, however, none of the prior investigations purely manipulated incentive. Even in Janes et al. ( 2020 ) in which students received incentive to complete some assignments but not others, a key confound was present – for incentivized assignments, students also received a schedule about when to complete assignments, whereas for optional assignments, none of this scaffolding was provided. Although students may complete more SR assignments when they receive incentive to do so, prior research on the impact of incentives on homework completion has produced mixed results. One study found significantly more students completed homework assignments when receiving course credit compared to not receiving any points (Ryan & Hemmes, 2005). Others have found significant increases in homework completion with an increase in course credit (Kontur & Terry, 2014 ) and an increase in money (Gong, Liu, & Tang, 2021 ). Perhaps most relevant, Trumbo et al. (2016) reported that students completed significantly more practice quizzes when they were required than optional. Nevertheless, in an investigation by Planchard et al. (2015), students reported that course credit was an incentive to completing assignments, yet later analyses revealed no significant relationship between credit and actual homework completion. Moreover, Radhakrishnan, Lam, & Ho (2009) found no difference in homework completion for students who received 1.25% of their final grade from homework assignments compared to .45%. Given these results, it is unclear whether incentive will influence students use of SR. One reason why the impact of incentive may be minimal is that the students are generally motivated to excel in gateway science courses like epidemiology and biostatistics, and if so, they may use SR when given the opportunity to learn the optional concepts which would overshadow any potential impact of incentive. To explore these possibilities, we investigated the impact of using course credit as incentive by comparing for-credit versus optional assignments, with all else equal, to investigate the use of SR in improving medical students’ retention of concepts in an introductory medical research course. With respect to the use of SR in this course, we designed two sets of assignments that included approximately the same amount and kind of content. For each student, one set of assignments was designated for credit and the other was optional (whether a set was slated for credit or was optional was counterbalanced across students). The concept prompts and answers (separated by assignment and the two sets within each assignment) are presented in the Appendix . All assignments were delivered using the same SR program as Janes et al. ( 2020 ) in which students first used a drop-down menu to select whether they wanted to complete for-credit assignments or optional ones. The main dependent measure was the number of assignments (for-credit vs. optional) that students completed. Importantly, we also evaluated the degree to which students learned and retained the concepts by administering a closed-book interim test a few days before their final exam in which students were asked to recall concepts from sets that were assigned for credit and from sets that were optional. We also evaluated the impact of using SR on performance on two other tests -- the in-class final exam and a 3-month delayed retention test. Note that the in-class exam was an open-book test, so analysis of these outcomes was exploratory, and we did not expect to find any systematic effect of using SR. More important, the three-month delayed test may be more sensitive to revealing the effects of SR on memory for the content, as the effects of some interventions (and in particular, those based on retrieval practice and spaced practice) get larger with longer delays after the intervention (for a detailed discussion, see Soderstrom & Bjork, 2015). Materials and Methods Participants and Design Participants consisted of 99 first-year podiatric medical students in a blended Principles of Medical Research course taught primarily by a College of Podiatric Medicine faculty member The class lasted 10 weeks, with class sessions held once a week, and instruction about epidemiology and statistics occurred during a single two-hour block. Class work included pre-class quizzes, in class assignments, 1 final exam, online video lectures, and other types of work. To assess the impact of course credit on SR use, students in the course were randomly assigned to one of two conditions. In the for-credit condition, students received course credit for completing the assignments. Each assignment was worth 0.8% of their grade; thus, all six assignments were worth 4.6% of the total points possible. In the optional condition, the assignments were available to students, but they were not required to complete them. However, they were told that completing flashcards would help them and that they would still be tested on that material. Importantly, assignments in both conditions were also accompanied by a schedule detailing when to do the assignments. Materials Materials consisted of course content from the pre-class video lectures adapted into six stacks of flashcard assignments containing 6–10 items, all of which were cue-target definitions. See the Appendix or the list of items. Materials were chosen and edited in conjunction with the instructor of the course. The stacks were split into two sets, Set A and Set B. Across participants, these sets were counterbalanced in terms of which set was assigned for credit or was optional. Procedure Students could complete up to 12 assignments – they received credit for completing six of them and the other six were optional. As shown in Table 1 , each assignment contained 1–3 stacks of material. Across assignments, each stack occurred five times – three of which occurred prior to the interim test followed by two refresher assignments that occurred prior to the open-book exam. Table 1 Schedule of ideal study dates for the successive relearning sessions. Note that the interim test was made available after Assignment 4 but before the first refresher. Assignment Stacks Dates available Assignment 1 Basic Stat I 9/21–9/23 Assignment 2 Basic Stat I Epi I Epi 3 9/24–9/27 Assignment 3 Basic Stat I Epi 1 Epi 3 9/27–9/29 Assignment 4 Epi 1 Epi 3 9/30 − 10/4 Assignment 5 (refresher) Basic 1 Epi 1 Epi 3 11/15–11/17 Assignment 6 (refresher) Basic 1 Epi 1 Epi 3 11/18–11/22 Students were asked to repeat each stack during three sessions for a specified range of days (referred to as “ideal study dates” to the students). These sessions were spread out over four homework assignments and not all homework assignments contained the same stacks. Two additional homework assignments containing three stacks each were provided as refresher sessions before their final exam. Thus, across all 6 assignments, students received three stacks and repeated them five times for a total of 15 assigned stacks and 15 optional stacks. Students were given access to virtual flashcards through the website Study Buddy ( https://apps.kent.edu/studybuddy/Courses.aspx ). Flashcards became available to students on the first day of the ideal study dates. After becoming available, flashcards remained available for the duration of the course. To complete a stack, the student cycled through the free-response items, answering them one at a time. After answering an item correctly, it was dropped from the stack, and they would continue cycling through the remaining items. Students were asked to compare their answer to a correct answer that had been broken down into main ideas ( idea units ). The student would then verify if their answer contained all of those parts by checking a box. They needed to mark all idea units correct in order for that answer to be considered correct. Marking one or more idea units as incorrect resulted in that item being placed at the end of the stack. After evaluating their answer for the idea units, students were shown the correct answer. For example, in the stack including basic statistical concepts (see Set A, Basic Stats 1 in Appendix ), when students began, the first item presented “What is the definition of a dependent variable?” Below that, they would see an empty textbox with the header “Your answer:”. After typing an answer, students were shown their answer, along with two idea units: “Outcomes measured” and “in an experiment”. The students would then have to determine if their answer contained each of those two ideas. Regardless of what they selected, the next screen showed their answer again, and also the full correct answer (“Outcomes measured in an experiment”). If they had selected both idea units as correct, that item would be dropped from the stack. If one or both of the idea units was marked as incorrect, that item would return at the end of the stack. Ten days before their final exam, we administered a 22-item interim test to evaluate student knowledge after they had experienced the 3 successive relearning sessions (the four homework assignments), but before most students had started studying for their final exam. The questions consisted of a subset of the items used in the flashcard stacks. The test was administered as a survey via Qualtrics, and 99 students responded. Using the Qualtrics interface, we collected open-ended responses. Students received points equivalent to 1.5% of their total grade for completing this test. At the end of the course, students completed an in-class, open-book final exam. Three months after the end of the course, we administered the 22-item test again to evaluate long term retention of the material. Nineteen students agreed to participate, completed a consent form (IRB approved), and were compensated with a $ 10 gift card; they were also entered in a raffle for an additional $ 25 gift card. Results How Often Did Students Use Successive Relearning? To evaluate whether incentive impacted students’ use of SR, we computed the number of assignments that each participant completed, for those assignments that were slated for incentive (up to 6) versus those that were optional (up to 6). Most assignments (assignments 2–6) contained multiple flashcard stacks, and an assignment was considered completed if the student finished all stacks in that assignment. Mean values for percent of completed assignments (presented separately for Set A and Set B concepts) are shown in Fig. 1 . A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance revealed a significant main effect for credit, F (1, 100) = 8.38, p = .005, MSE = .004, partial η 2 = .077, but did not indicate a significant main effect of set, F (1,100) = 0.35, p = .56, MSE = .004, partial η 2 = .003. Most importantly, the analysis also indicated a significant interaction between credit and set, F (1,100) = 13,587.70, p < .001, MSE = .004, partial η 2 = .99, suggesting that students completed more assignments for credit than when assignments were optional. Because some students completed fewer than the full number of flashcard stacks in a given assignment, we also computed the percentage of stacks that each student completed. A similar pattern to completed assignments occurred: Students completed an average of 91.1% of for-credit stacks and 1.1% of optional stacks when Set A was assigned, and an average of 99.2% of for-credit stacks and 0.5% of optional stacks when Set B was assigned. Retention of Key Terms on the Interim Test To evaluate the impact of using SR on retention, the second author (who is the instructor for the course) scored the responses based on the presence of correct main ideas in the student response. In particular, each answer was separated into main idea units by the instructor in a manner that intuitively captured the key ideas in each concept (the idea units for each concept are presented in Appendix ). We computed the proportion of main ideas each student had in their answer (for example, if a student’s answer contained 1 of 3 idea units, that particular response received a score of .33). We then calculated mean proportion correct for each question (see Fig. 2 ). A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance did not reveal a significant main effect for credit, F (1,97) = 1.20, p = .28, MSE = .10, partial η 2 = .012, but did reveal a significant effect for set, which indicated that performance was lower for Set B than Set A content, F (1,97) = 18.31, p < .001, MSE = .02, partial η 2 = .16. Most important, the significant interaction indicated that performance was greater when students received credit for using successive relearning than when it was optional, F (1,97) = 34.72, p < .001, MSE = .02, partial η 2 = .26. In-Class, Open-Book Exam Performance For the open-book exam, we compared performance on exam questions that tapped into concepts that students received credit to successively relearn compared to concepts that were from optional assignments. Given that the exam was open book, performance was high (and in some cases, very close to the ceiling): For Set A concepts, correct performance was near the ceiling regardless of whether the concepts were assigned for credit ( M = 94.4, SEM = 1.4, Mdn = 1) or were optional ( M = 94.0, SEM = 1.6, Mdn = 1). For Set B concepts, mean percent correct performance was 85.4 (2.2) and 94.0 (1.6) when the concepts were assigned for credit or were optional (Medians = .95 and .90), respectively. A 2 X 2 ANOVA reveal only a significant effect for Set, F (1,99) = 16.18, p < .001, MSE = .01, partial η 2 = .14, indicating that questions tapping Set B concepts were more difficult than those tapping Set A concepts. The main effect of credit (credit versus optional) was not significant, F (1,99) = 2.1, p = .15, MSE = .02, partial η 2 = .02, nor was the interaction between credit and set, F (1,99) = 2.7, p = .10, MSE = .01, partial η 2 = .03. Retention After a Three-Month Delay As per the interim test, the second author scored the responses for the 3-month delayed test. As shown in Fig. 3 , the pattern of outcomes was similar to the interim test, although forgetting occurred over the three month delay. A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance did not reveal a significant main effect for credit, F (1, 16) = .31, p = .59, MSE = .017, partial η 2 = .019, but did reveal a significant effect for set, which indicated that performance was lower for Set B than Set A content, F (1, 16) = 14.13, p = .002, MSE = .141, partial η 2 = .469. Most important, the significant interaction indicated that performance was greater when students received credit for using successive relearning than when it was optional, F (1, 16) = 13.60, p = .002, MSE = .14, partial η 2 = .46 . Discussion In the current experiment, even when all assignments came with a schedule, students completed significantly more assignments when they received course credit versus when the assignments were made available but optional. And, for the latter, students completed close to none of the assignments and did not engage in the spaced practice of the same material across multiple sessions that is needed to implement successive relearning. Moreover, students performed significantly better on concepts that they had received credit for successively relearning as compared to concepts from the optional assignments, although the positive impact of successive relearning was relatively small for the more difficult item set (see Figs. 2 & 3 ). One explanation for why students did not complete most of the optional assignments is related to the design of Experiment 1. The website in which students completed assignments presented both types of assignments side by side. In particular, to complete an assignment, students were shown a drop-down menu where they could select to complete the for-credit assignments (labelled “Assigned”) or the optional assignments (labelled “Optional”). The simultaneous presentation (i.e., juxtaposing “Assigned” with “Optional” on the menu) may have led students to perceive the for-credit assignments as more worthwhile, which in turn may have led them to not want to do the optional assignments when they could do others for course credit. Likewise, after completing a for-credit assignment, students would need to go back to the main menu to select the optional assignment, and the extra effort may have curtailed their motivation to do more. Accordingly, if the two kinds of assignments were not contrasted in this manner, students might complete more optional assignments, which we evaluated in Experiment 2. Experiment 2 In Experiment 1, the interface of the website presented the two types of assignments side by side, which may have led students to under value the optional assignments. Thus, in Experiment 2, we evaluated whether students (from a different cohort) completed assignments when they did not have both kinds of assignments available to them. In particular, students first received optional assignments (and instructions on how to use the successive relearning program) at the beginning of the term, so as to evaluate how much they would use them when for-credit assignments were not available. Then, later in the semester, new assignments were provided in which students could earn course credit to ensure that the particular group of students would complete for-credit assignments as per usual. Method We conducted Experiment 2 in the same Principles of Medical Research course with a new sample of students in another semester. In the first half of the class, all students had access to four optional successive relearning assignments. In the second half of the class, all students were required to complete four different successive relearning assignments. The materials were identical to Experiment 1 but redistributed. About half of the content was used as the optional assignments in the first half of the class, and the other half was used as the for-credit assignments (see Appendix for how the content was distributed). The assignment protocol remained the same. Results How Often Did Students Use Successive Relearning? Students completed none ( M = 0, SEM = 0) of the optional assignments, whereas they completed the vast majority ( M = 96.9%) of assignments when given for-credit assignments later in the semester. Thus, consistent with Experiment 1 outcomes, even when optional assignments were the only ones available, students still did not complete them, suggesting that some incentive may be necessary to motivate students to use successive relearning. General Discussion Across two experiments, we investigated the extent to which students would engage in successive relearning when the assignments were for credit or optional. Students completed the vast majority of assignments when they received course credit, but completed almost none when the assignments were optional. Given the benefits of successive relearning on retention of material, students’ unwillingness to complete the optional assignments was rather surprising, especially given that (a) the instructor developed the content and indicated that the assignments included key concepts in the course and (b) Principles of Medical Research is a critical course for first year podiatric medical students. That is, both studies were conducted in a required course for these students, and failing this course had consequences for their progress. These outcomes and those from prior studies demonstrate that students are not always motivated to engage in extra study – even in the present case involving medical students – when it is optional. On the optimistic side, students were willing to complete the successive relearning assignments when they received a relatively small amount of course credit to complete them. Thus, instructors who desire to motivate their students to use a particular learning strategy for completing homework assignments can likely do so by providing a relatively small proportion of the class grade for completion. Nevertheless, strategies like successive relearning show a great deal of promise for helping students to gain mastery of a particular topic – both in obtaining the content as well as maintaining over time (for a review, see Rawson & Dunlosky, 2022), so it may benefit students to adopt this strategy to master any content even when they are not given an incentive to do so. A question becomes, how might students be motivated to use the strategy on their own? We suspect that answers to this question will involve enhancing students’ perceived value of successive relearning. How might this be done, and if so, what are possible tactics to increase its perceived value? First, some students may view successive relearning as having value at least in some contexts. For instance, in surveys about students use of flashcards (Zung et al., 2022; Wissman et al., 2012), the majority of students report using flashcards to learn simple associations, such as a foreign language. The surveys do not focus on successive relearning per se, but most students report using flashcards to practice until they can correctly recall all the to-be-learned material at least once. By contrast, very few students reported using flashcards to study more complicated material, such as the definitions of concepts (as in the present research). Lack of using successive relearning in this case may arise from several barriers: (a) students may not want to spend the time developing flashcards for complicated material, (b) they may not realize that this approach is beneficial for learning complex materials, or (c) they may view the time and effort required to test themselves on difficult definitions as being overly burdensome as compared to not studying or merely restudying the material. The first barrier is not relevant in the current experiments in which the flashcards were populated with concepts chosen by the instructor and were prepared ahead of time. The second two barriers may apply, however: Students may not believe that flashcards are useful for complex material, and learning a difficult concept well enough to correctly recall during a given session does take real time. Prescriptions on how to overcome these barriers are available from McDaniel and Einstein (2020), who described a framework for motivating students to adopt effective study strategies. The framework has four components: knowledge, belief that the learning strategy is valuable, commitment to using the strategy, and a plan of action for implementing it. Accordingly, to motivate students, an instructor could explain how to use successive relearning to give students the knowledge they need to complete successive relearning assignments. Second, to help students realize this strategy is effective, the instructor could give students experience with successive relearning, such as by requiring some assignments in the beginning of the class, and then switching to optional assignments after students have had time to experience its benefits. Third, students could build commitment to the strategy by responding to a short prompt asking them to draw an explicit connection between the learning strategy and how it can be used to achieve their educational goal. Finally, the instructor could have students write a brief outline of their plan for using successive relearning (e.g. where the student will get the materials for the questions and when they intend to complete spaced learning sessions). The degree to which such approaches improve students use of successive relearning is an important avenue for future classroom-based research. Finally, in the present context, it was evident that to retain the epidemiology principles, students would need to engage in more successive relearning. For instance, as shown in Fig. 3 , three months after class, students forgot a great deal of what they had initially learned. One recommendation here would be to include booster sessions in which students successively relearn the materials over longer-and-longer delays until it is fluently accessed even after long delays. Of course, such booster sessions would require planning and likely should focus just on the critical material that students need to fluently remember for subsequent courses and in their jobs. That is, given that obtaining and maintaining knowledge can take a meaningful amount of time, strategies like successive relearning may best be used for just the content that is most essential to remember over the long term. Conclusions While successive relearning is beneficial for medical student learning, many students are unlikely to complete successive relearning assignments unless they are given course credit. Fortunately, it appears a relatively trivial amount of credit is sufficient to motivate medical students to complete successive relearning assignments. Declarations Statements and Declarations Funding Writing of this manuscript was partially supported by the National Science Foundation under Grant IUSE-1914499. Competing interests The authors declare no other conflicts of interest. Ethics approval These experiments were approved by the Kent State University Institutional Review Board. Informed consent Participants were informed about the experiments, and informed consent was obtained from all participants. Consent to publish As the present experiments used aggregate, anonymized data, consent to publish was not obtained. Data and analyses Data and analyses are available upon request. Author contributions Conceptualization: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Methodology: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Formal analysis and investigation: Maren Greve, John Dunlosky; Writing – original draft preparation: Maren Greve, John Dunlosky; Writing – review and editing: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Funding acquisition: John Dunlosky References Badali, S., & Greve, M. (2023). Can successive relearning enhance performance on application-based exam questions ? Under review. Dunlosky, J., & O'Brien, A. (2020). The power of successive relearning and how to implement it with fidelity using pencil and paper and web-based programs. Scholarship of Teaching and Learning in Psychology, 8, 225-235. https://doi.org/10.1037/stl0000233 Gong, J., Liu, T. X., & Tang, J. (2021). How monetary incentives improve outcomes in MOOCs: Evidence from a field experiment. Journal of Economic Behavior & Organization, 190 , 905–921. https://doi.org/10.1016/j.jebo.2021.06.029 Higham, P. A., Zengel, B., Bartlett, L. K., & Hadwin, J. A. (2021). The benefits of successive relearning on multiple learning outcomes. Journal of Educational Psychology, 114, 928-944. https://doi.org/10.1037/edu0000693 Janes, J. L., Dunlosky, J., Rawson, K. A., & Jasnow, A. (2020). Successive relearning improves performance on a high‐stakes exam in a difficult biopsychology course. Applied Cognitive Psychology , 34 , 1118-1132. Kontur, F. J., & Terry, N. B. (2014). Motivating students to do homework. The Physics Teacher, 52. Nevid, J. S., & Gordon, A. J. (2018). Integrated Learning Systems: Is There a LearningBenefit? Teaching of Psychology , 45 , 340-345. Planchard, M., Daniel, K. L., Maroo, J., Mishra, C., & McLean, T. (2015). Homework, Motivation, and Academic Achievement in a College Genetics Course Radhakrishnan, P., Lam, D., & Ho, G. (2009). Giving University Students Incentives to do Homework Improves their Performance. Journal of Instructional Psychology, 36 , 219–225. Rawson, K. A., & Dunlosky, J. (2022). Successive relearning: an underexplored but potent technique for obtaining and maintaining knowledge. Current Directions in Psychological Science , 31, 362-268. doi.org/10.1177/09637214221100484 Rawson, K. A., Dunlosky, J., & Sciartelli, S. M. (2013). The power of successive relearning: Improving performance on course exams and long-term retention. Educational Psychology Review, 25, 523–548. https://doi-org.proxy.library.kent.edu/10.1007/s10648-013-9240-4 Ryan, C. S., & Hemmes, N. S. (2005). Effects of the contingency for homework submission on homework submission and quiz performance in a college course. Journal of Applied Behavior Analysis, 38 , 79-88. Soderstrom, N. C., & Bjork, R. A. (2015). Learning versus performance: An integrative review. Perspectives on Psychological Science, 10 , 176-199. Trumbo, M. C., Leiting, K. A., McDaniel, M. A., & Hodge, G. K. (2016). Effects of reinforcement on test-enhanced learning in a large, diverse introductory college psychology course. Journal of Experimental Psychology: Applied, 2 , 148-160. Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2012). How and when do students use flashcards? Memory, 20 , 568-579. Zung, I., Imundo, M. N., & Pan, S. C. (2022). How do college students use digital flashcards during self-regulated learning? Memory, 30 , 923–941. https://doi.org/10.1080/09658211.2022.2058553 Appendix The “prompt” column shows the question students were prompted to answer. The “answer” column shows the answer that students were shown as feedback. Slash marks indicate where the answer was split up to form idea units (no slash indicates that the entire answer fit into a single idea unit). Italics indicates this text was not included in the idea units (but was still shown in the answer feedback). Some idea units were changed (e.g. for formulas involving division, the idea units were written as “divided by” instead of the slash mark used in the full answer) or lightly paraphrased in order to fit into the maximum character limit of the idea units (e.g. “Probability that an individual would experience an outcome” was changed to “Probability of experiencing an outcome”). In Experiment 1, for the three stacks within a set (either Set A or Set B), a student received credit for completing assignments for one set (e.g., A) and did not receive credit for the other, optional set (i.e., B); which set was slated as for-credit or optional was counterbalanced across students. Both sets were presented simultaneously In Experiment 2, Set A was given as the optional assignments, after which Set B was assigned as the for-credit assignments. Set Stack Prompt Answer A Basic Stats 1 What is the definition of a continuous variable? Data can take on / any value / within a set range What is the definition of a discrete variable? Data can be only one of limited number of values What is the definition of a nominal variable? By name only, / including / two or more categories / without a meaningful order What is the definition of a dichotomous variable? A nominal variable / with only two categories What is the definition of an ordinal variable? Two or more categories / arranged in a meaningful order What is the definition of an independent variable? In an experiment, / the variable that is manipulated / to assess its effects What is the definition of a dependent variable? Outcomes measured / in an experiment Epi 1 What is the definition of epidemiology? Branch of medicine / that deals with the incidence, distributions and possible control of / diseases and other factors relating to health What does HRE stand for in epidemiology? Health-related event What is the definition of incidence? Number of new occurrences of an HRE in a population / during a specific time / (risk of contracting HRE) What is the definition of prevalence? Number of existing cases in a population / during a specific time / (how widespread the HRE is) What is the definition of relative risk? The ratio of / risk in the exposure group / to the risk in the control group What is the definition of "risk"? The probability of an event occurring What is the formula for probability? (# of cases of one event) / divided by / (overall # of cases) What does a relative risk above 1.0 mean? Risk of developing an HRE / is higher for exposure group What does a relative risk equals to 1.0 mean? Risk of developing an HRE / is the same for exposure group What does a relative risk below 1.0 mean? Risk of developing an HRE / is lower for exposure group Epi 3 What is the definition of sensitivity? Ability of a test to detect an HRE / in an individual when HRE is present Why is high sensitivity important? You will have few false negatives What is the definition of specificity? The ability of a test to indicate non-HRE / in an individual when no HRE is present Why is high specificity important? You will have few false positives What is positive predictive value? Proportion of individuals / who are screened positive by a test / and actually have the HRE What is negative predictive value? Proportion of individuals / who are screened negative by a test / and actually do NOT have the HRE B Basic Stats 2 What is the mean of a set of numbers? Sum of the set of values / divided by the number of values in the set What is the median of a set of number? The middle value / in a data set What is the mode? The most frequent score in a data set What is the range? A measure of dispersion that is equal to the / difference between the largest and smallest values / in a data set What is the variance? A measure of dispersion / that is the average of the squared differences of the means What is the standard deviation? A measure of dispersion / that is the square root of the variance What is a normal distribution? Symmetric bell-shaped curve / where the mean, median and mode are all equal Epi 2 What is the formula for the odds ratio? (odds of HRE in exposure group) / divided by / (odds of HRE in the control group) What is the definition of "odds"? The likelihood of an event occurring What is the formula for odds? (# of cases of one event) / divided by / (# of cases of alternate event) What does an odds ratio greater than 1 mean? Exposure associated with higher odds of outcome What does an odds ratio equal to 1 mean? Exposure does not affect odds of outcome What does an odds ratio less than 1 mean? Exposure associated with lower odds of outcome What is the definition of hazard? Probability of experiencing an outcome / at a specific point in time What is the formula for hazard ratio? (hazard in the exposure group) /divided by / ( hazard in the control group) Epi 4 What is likelihood ratio? Probability that a specific / patient has a specific HRE What are likelihood ratios used for? To assess the diagnostic accuracy of a test What is the formula for likelihood ratio? (prob ability a person w/HRE has a specific test result) / divided by / (prob ability a person without an HRE has the specific test result) What does a likelihood ratio equaling 1 mean? Diagnostic test has no value What is the formula for LR+? sensitivity / divided by / (1 – specificity) What does an LR+ between 5 and 10 mean? Test result has a moderate effect / on increasing probability of HRE What is the formula for LR-? (1 – sensitivity) / divided by / specificity What does an LR- between 0.5 and 0.1 mean? Test result has a moderate effect / on decreasing probability of HRE Footnotes Although similar, a stack and a program are not interchangeable terms. A stack is a set of to-be-learned items. Stacks are not necessarily used in a way that instantiates SR. A SR program uses stacks but also includes the infrastructure that supports the use of SR by scheduling distributed practice session, dropping items after reaching criterion in a given session, providing feedback, and so forth. Additional Declarations Competing interest reported. Writing of this manuscript was partially supported by the National Science Foundation under Grant IUSE-1914499. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3061046","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":209708197,"identity":"91c20d37-c98b-4652-a095-7e7e34d051fe","order_by":0,"name":"Maren Greve","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAyUlEQVRIiWNgGAWjYBACgwNgqkZOAsJnJqzFEqLlmDFQC2MDUVrsIVqYE2cQrcXs/NqHj3kq2NJntvcef8BQYZ3YQFDLjefGxjxnZHJn85wDqj6TToyWY2ySM9vYcudJ5Bg2MLYdJqzF4MYx9p8z/zGny8m/AWr5R4yW821sDB8bmBOkJXiAWhqIsoWNWeLDsWOGM3tyDGckHEs3JsKWY4wfEmpq5CWOnzH48KHGWpagFgaJBCROAg5FqID/AFHKRsEoGAWjYCQDALjLQr40eL7BAAAAAElFTkSuQmCC","orcid":"","institution":"Kent State University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Maren","middleName":"","lastName":"Greve","suffix":""},{"id":209708198,"identity":"aa525159-a35a-4b2c-b860-867d5496b243","order_by":1,"name":"Jill Kawalec","email":"","orcid":"","institution":"Kent State University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jill","middleName":"","lastName":"Kawalec","suffix":""},{"id":209708199,"identity":"b31a7b68-c44d-46e7-9a1e-12186dbec33e","order_by":2,"name":"Viveka Jenks","email":"","orcid":"","institution":"Kent State University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Viveka","middleName":"","lastName":"Jenks","suffix":""},{"id":209708200,"identity":"6bf4015d-ebc2-41e0-aa93-a68ca600ac64","order_by":3,"name":"John Dunlosky","email":"","orcid":"","institution":"Kent State University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Dunlosky","suffix":""}],"badges":[],"createdAt":"2023-06-14 05:29:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3061046/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3061046/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":38684375,"identity":"cce255cd-6ce8-4e3c-ac05-903d455dfd72","added_by":"auto","created_at":"2023-06-16 18:42:28","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":92321,"visible":true,"origin":"","legend":"\u003cp\u003ePercentage of assignments completed by students as a function of concept set and whether students received credit for completion. Error bars (not visible on the Optional assignment) are standard errors of each mean.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3061046/v1/9cc41f0a243a8026021af122.png"},{"id":38684374,"identity":"59f7a7c5-fd6e-4fba-a845-787bfe1d8ffc","added_by":"auto","created_at":"2023-06-16 18:42:28","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":28613,"visible":true,"origin":"","legend":"\u003cp\u003ePercentage correct performance on the interim test as a function of concept set and whether students received credit for completion. Error bars are standard errors of each mean.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3061046/v1/7bfd5a53b07fa9d446d4ca5b.png"},{"id":38684373,"identity":"2a012712-5ba3-4731-9d6c-04a4d8ed104a","added_by":"auto","created_at":"2023-06-16 18:42:28","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":13404,"visible":true,"origin":"","legend":"\u003cp\u003ePercentage correct performance on the (three-month) delayed test as a function of concept set and whether students received credit for completion. Error bars are standard errors of each mean.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3061046/v1/d0e3e0a94e4062f2ca56a68e.png"},{"id":39658335,"identity":"07f10b87-7eb7-4429-bef5-85882865f8c1","added_by":"auto","created_at":"2023-07-06 20:29:32","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":399248,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3061046/v1/6f1988b3-cb05-4bd5-88d8-d90cfc74e53b.pdf"}],"financialInterests":"Competing interest reported. Writing of this manuscript was partially supported by the National Science Foundation under Grant IUSE-1914499.","formattedTitle":"The Effect of Incentives on the Use of Successive Relearning for Retaining Statistics and Epidemiology Concepts in a Medical Research Course","fulltext":[{"header":"Introduction","content":"\u003cp\u003eUnderstanding epidemiology is critical for medical practitioners because it allows them to understand the development of diseases, to diagnose them, and to reduce risk for patients. Epidemiology is a challenging course \u0026ndash; many students struggle to memorize and understand the numerous concepts and equations that comprise epidemiology and biostatistics. Thus, exploring techniques that boost students\u0026rsquo; learning and retention of epidemiological concepts is an important endeavor for helping students to become effective practitioners. Toward this goal, we investigated the degree to which one technique -- successive relearning -- improves students\u0026rsquo; learning of key concepts in descriptive statistics and epidemiology in a medical research course. As we describe in detail below, \u003cem\u003esuccessive relearning\u003c/em\u003e (SR) is a specific combination of two effective learning techniques (retrieval practice and spaced practice) that have been shown to improve students\u0026rsquo; retention (for a review, see Rawson \u0026amp; Dunlosky, in press). Unfortunately, many students do not use successive relearning, even when they are instructed on how to use it and provided with scaffolds to use it successfully. Moreover, very few investigations have explored the impact of SR in classroom settings, and none have focused on epidemiology. Accordingly, in the current research, we evaluated whether SR improves students\u0026rsquo; retention of epidemiology and descriptive statistics, and most important, the degree to which a relatively small incentive increases its rate of use. In the remainder of this introduction, we explain the components of SR, briefly describe prior research, and discuss why incentive may improve usage.\u003c/p\u003e \u003cp\u003eSuccessive relearning involves attempting to recall information from memory (retrieval practice), followed by presentation of the correct answer as feedback. Students continue to practice retrieval with feedback until the target information has been correctly recalled. Then, they repeat this process with the same information in multiple sessions that are spaced across time (spaced practice). SR is perhaps best illustrated by students using a physical deck of flashcards. While using flashcards, the student would read the prompt on one side of the first card (e.g. \u0026ldquo;What is an odds ratio?\u0026rdquo;), retrieve an answer from their memory (which may be correct or incorrect), and then refer to the back of the card to evaluate if their answer is correct. If correct, they would set that card to the side and continue with the rest of the items. If incorrect, they would put that card at the back of the stack and continue cycling through the deck until they recall the correct answer one time for each item. Importantly, to implement SR, students would also return to practice the same deck of items again during one or more spaced practice sessions.\u003c/p\u003e \u003cp\u003eThis example focuses on a physical deck of flashcards, but successive relearning can be instantiated in many ways, such as using class notes (e.g., Cornell notes) or using available flashcard apps on-line (for a review, see Dunlosky \u0026amp; O\u0026rsquo;Brien, 2020). Thus, the critical issue is not what platform students use (i.e., handheld flashcards or apps) but how they use them. That is, to use SR with fidelity, they must engage in its two components \u0026ndash; retrieval practice until a criterion is met and then relearning the same items during multiple spaced sessions. Unfortunately, although many college students report using flashcards to prepare for exams, most use them exclusively for studying foreign language-English translation equivalents and many students do not instantiate SR when they use flashcards (for details, see Zung et al., 2022; Wissman et al, 2012).\u003c/p\u003e \u003cp\u003eDoes SR improve student achievement in classroom settings? Whereas SR shows much promise, only a few studies have been conducted to evaluate its efficacy (for a recent review, see Rawson \u0026amp; Dunlosky, 2022). For instance, in Janes et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), students in an undergraduate biopsychology course were provided with a SR program that included multiple assignments, each of which consisted of 8\u0026ndash;10 to-be-learned items from the course, such as declarative concepts (e.g. What is an axon?), labels for structures (e.g. labels for parts of the brain), and steps of biological processes (e.g., various steps of the action potential). In each assignment, students were prompted to answer a single question and were instructed to type the correct answer. They then scored their answer and could restudy the correct answer. Students continued practicing retrieval with feedback until they correctly recalled an answer, and then it was dropped from practice during that session. They continued until they correctly recalled all answers in a given stack.\u003ca class=\"FNLink\" href=\"#Fn1\" id=\"#FNLinkFn1\"\u003e\u003c/a\u003e They were instructed to relearn the items from each assignment three more times, with two to three days between each session. Importantly, the assignments were separated into two sets, each with different content. Students were slated to complete one set of assignments using SR, and the other set was optional; the two sets were counterbalanced across students. As a baseline control, other content did not appear in the assignments but was included on the high-stakes, in-class exam. For the successively relearned items, students were given access to the assignments and a schedule that indicated when each assignment should be completed. For the optional items, students were only provided access to the assignment, but not a schedule on when (and how often) each assignment should be completed. Exam performance was significantly higher for questions that targeted successively relearned content compared to optional content. Such a benefit of using SR in a classroom setting has also been demonstrated in introductory psychology (e.g., Higham et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Rawson et al., 2013) and cognitive psychology (Badali \u0026amp; Greve, under review).\u003c/p\u003e \u003cp\u003eDespite the potential effectiveness of SR, an open issue is whether students will use it without external incentives, because in prior research, students were typically given an external incentive to use SR. For example, in Rawson et al. (2013), students were paid for each of the SR sessions they had completed. In Higham et al. (2022), students received course credit (20% of their final grade) for completing the SR sessions. As mentioned above, in Janes et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), students were also instructed to complete SR assignments; for completing some, they received course credit, whereas other assignments were optional. Most students completed the assignments that were slated for course credit but only 39 percent attempted the optional assignments. Of those who did engage in optional assignments, on average, they completed significantly fewer sessions (8.9) than when they received course credit (17.4). One potential conclusion is that students are less likely to engage in SR without an external incentive (see also, Nevid \u0026amp; Gordon, 2018). Note, however, none of the prior investigations purely manipulated incentive. Even in Janes et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) in which students received incentive to complete some assignments but not others, a key confound was present \u0026ndash; for incentivized assignments, students also received a schedule about when to complete assignments, whereas for optional assignments, none of this scaffolding was provided.\u003c/p\u003e \u003cp\u003eAlthough students may complete more SR assignments when they receive incentive to do so, prior research on the impact of incentives on homework completion has produced mixed results. One study found significantly more students completed homework assignments when receiving course credit compared to not receiving any points (Ryan \u0026amp; Hemmes, 2005). Others have found significant increases in homework completion with an increase in course credit (Kontur \u0026amp; Terry, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) and an increase in money (Gong, Liu, \u0026amp; Tang, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Perhaps most relevant, Trumbo et al. (2016) reported that students completed significantly more practice quizzes when they were required than optional. Nevertheless, in an investigation by Planchard et al. (2015), students reported that course credit was an incentive to completing assignments, yet later analyses revealed no significant relationship between credit and actual homework completion. Moreover, Radhakrishnan, Lam, \u0026amp; Ho (2009) found no difference in homework completion for students who received 1.25% of their final grade from homework assignments compared to .45%. Given these results, it is unclear whether incentive will influence students use of SR. One reason why the impact of incentive may be minimal is that the students are generally motivated to excel in gateway science courses like epidemiology and biostatistics, and if so, they may use SR when given the opportunity to learn the optional concepts which would overshadow any potential impact of incentive.\u003c/p\u003e \u003cp\u003eTo explore these possibilities, we investigated the impact of using course credit as incentive by comparing for-credit versus optional assignments, with all else equal, to investigate the use of SR in improving medical students\u0026rsquo; retention of concepts in an introductory medical research course. With respect to the use of SR in this course, we designed two sets of assignments that included approximately the same amount and kind of content. For each student, one set of assignments was designated for credit and the other was optional (whether a set was slated for credit or was optional was counterbalanced across students). The concept prompts and answers (separated by assignment and the two sets within each assignment) are presented in the \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e. All assignments were delivered using the same SR program as Janes et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) in which students first used a drop-down menu to select whether they wanted to complete for-credit assignments or optional ones. The main dependent measure was the number of assignments (for-credit vs. optional) that students completed. Importantly, we also evaluated the degree to which students learned and retained the concepts by administering a closed-book interim test a few days before their final exam in which students were asked to recall concepts from sets that were assigned for credit and from sets that were optional.\u003c/p\u003e \u003cp\u003eWe also evaluated the impact of using SR on performance on two other tests -- the in-class final exam and a 3-month delayed retention test. Note that the in-class exam was an open-book test, so analysis of these outcomes was exploratory, and we did not expect to find any systematic effect of using SR. More important, the three-month delayed test may be more sensitive to revealing the effects of SR on memory for the content, as the effects of some interventions (and in particular, those based on retrieval practice and spaced practice) get larger with longer delays after the intervention (for a detailed discussion, see Soderstrom \u0026amp; Bjork, 2015).\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eParticipants and Design\u003c/h2\u003e \u003cp\u003eParticipants consisted of 99 first-year podiatric medical students in a blended Principles of Medical Research course taught primarily by a College of Podiatric Medicine faculty member The class lasted 10 weeks, with class sessions held once a week, and instruction about epidemiology and statistics occurred during a single two-hour block. Class work included pre-class quizzes, in class assignments, 1 final exam, online video lectures, and other types of work.\u003c/p\u003e \u003cp\u003eTo assess the impact of course credit on SR use, students in the course were randomly assigned to one of two conditions. In the for-credit condition, students received course credit for completing the assignments. Each assignment was worth 0.8% of their grade; thus, all six assignments were worth 4.6% of the total points possible. In the optional condition, the assignments were available to students, but they were not required to complete them. However, they were told that completing flashcards would help them and that they would still be tested on that material. Importantly, assignments in both conditions were also accompanied by a schedule detailing when to do the assignments.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eMaterials\u003c/h2\u003e \u003cp\u003eMaterials consisted of course content from the pre-class video lectures adapted into six stacks of flashcard assignments containing 6\u0026ndash;10 items, all of which were cue-target definitions. See the \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e or the list of items. Materials were chosen and edited in conjunction with the instructor of the course. The stacks were split into two sets, Set A and Set B. Across participants, these sets were counterbalanced in terms of which set was assigned for credit or was optional.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eProcedure\u003c/h2\u003e \u003cp\u003eStudents could complete up to 12 assignments \u0026ndash; they received credit for completing six of them and the other six were optional. As shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, each assignment contained 1\u0026ndash;3 stacks of material. Across assignments, each stack occurred five times \u0026ndash; three of which occurred prior to the interim test followed by two refresher assignments that occurred prior to the open-book exam.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSchedule of ideal study dates for the successive relearning sessions. Note that the interim test was made available after Assignment 4 but before the first refresher.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStacks\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDates available\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBasic Stat I\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9/21\u0026ndash;9/23\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBasic Stat I\u003c/p\u003e \u003cp\u003eEpi I\u003c/p\u003e \u003cp\u003eEpi 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9/24\u0026ndash;9/27\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBasic Stat I\u003c/p\u003e \u003cp\u003eEpi 1\u003c/p\u003e \u003cp\u003eEpi 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9/27\u0026ndash;9/29\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEpi 1\u003c/p\u003e \u003cp\u003eEpi 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9/30\u0026thinsp;\u0026minus;\u0026thinsp;10/4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 5 (refresher)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBasic 1\u003c/p\u003e \u003cp\u003eEpi 1\u003c/p\u003e \u003cp\u003eEpi 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e11/15\u0026ndash;11/17\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAssignment 6 (refresher)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBasic 1\u003c/p\u003e \u003cp\u003eEpi 1\u003c/p\u003e \u003cp\u003eEpi 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e11/18\u0026ndash;11/22\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eStudents were asked to repeat each stack during three sessions for a specified range of days (referred to as \u0026ldquo;ideal study dates\u0026rdquo; to the students). These sessions were spread out over four homework assignments and not all homework assignments contained the same stacks. Two additional homework assignments containing three stacks each were provided as refresher sessions before their final exam. Thus, across all 6 assignments, students received three stacks and repeated them five times for a total of 15 assigned stacks and 15 optional stacks.\u003c/p\u003e \u003cp\u003eStudents were given access to virtual flashcards through the website Study Buddy (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://apps.kent.edu/studybuddy/Courses.aspx\u003c/span\u003e\u003cspan address=\"https://apps.kent.edu/studybuddy/Courses.aspx\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e).\u003c/span\u003e Flashcards became available to students on the first day of the ideal study dates. After becoming available, flashcards remained available for the duration of the course. To complete a stack, the student cycled through the free-response items, answering them one at a time. After answering an item correctly, it was dropped from the stack, and they would continue cycling through the remaining items. Students were asked to compare their answer to a correct answer that had been broken down into main ideas (\u003cem\u003eidea units\u003c/em\u003e). The student would then verify if their answer contained all of those parts by checking a box. They needed to mark all idea units correct in order for that answer to be considered correct. Marking one or more idea units as incorrect resulted in that item being placed at the end of the stack. After evaluating their answer for the idea units, students were shown the correct answer.\u003c/p\u003e \u003cp\u003eFor example, in the stack including basic statistical concepts (see Set A, Basic Stats 1 in \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e), when students began, the first item presented \u0026ldquo;What is the definition of a dependent variable?\u0026rdquo; Below that, they would see an empty textbox with the header \u0026ldquo;Your answer:\u0026rdquo;. After typing an answer, students were shown their answer, along with two idea units: \u0026ldquo;Outcomes measured\u0026rdquo; and \u0026ldquo;in an experiment\u0026rdquo;. The students would then have to determine if their answer contained each of those two ideas. Regardless of what they selected, the next screen showed their answer again, and also the full correct answer (\u0026ldquo;Outcomes measured in an experiment\u0026rdquo;). If they had selected both idea units as correct, that item would be dropped from the stack. If one or both of the idea units was marked as incorrect, that item would return at the end of the stack.\u003c/p\u003e \u003cp\u003eTen days before their final exam, we administered a 22-item interim test to evaluate student knowledge after they had experienced the 3 successive relearning sessions (the four homework assignments), but before most students had started studying for their final exam. The questions consisted of a subset of the items used in the flashcard stacks. The test was administered as a survey via Qualtrics, and 99 students responded. Using the Qualtrics interface, we collected open-ended responses. Students received points equivalent to 1.5% of their total grade for completing this test. At the end of the course, students completed an in-class, open-book final exam. Three months after the end of the course, we administered the 22-item test again to evaluate long term retention of the material. Nineteen students agreed to participate, completed a consent form (IRB approved), and were compensated with a \u003cspan\u003e$\u003c/span\u003e10 gift card; they were also entered in a raffle for an additional \u003cspan\u003e$\u003c/span\u003e25 gift card.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results ","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eHow Often Did Students Use Successive Relearning?\u003c/h2\u003e \u003cp\u003eTo evaluate whether incentive impacted students\u0026rsquo; use of SR, we computed the number of assignments that each participant completed, for those assignments that were slated for incentive (up to 6) versus those that were optional (up to 6). Most assignments (assignments 2\u0026ndash;6) contained multiple flashcard stacks, and an assignment was considered completed if the student finished all stacks in that assignment. Mean values for percent of completed assignments (presented separately for Set A and Set B concepts) are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance revealed a significant main effect for credit, \u003cem\u003eF\u003c/em\u003e(1, 100)\u0026thinsp;=\u0026thinsp;8.38, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.005, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.004, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.077, but did not indicate a significant main effect of set, \u003cem\u003eF\u003c/em\u003e(1,100)\u0026thinsp;=\u0026thinsp;0.35, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.56, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.004, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.003. Most importantly, the analysis also indicated a significant interaction between credit and set, \u003cem\u003eF\u003c/em\u003e(1,100)\u0026thinsp;=\u0026thinsp;13,587.70, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.004, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.99, suggesting that students completed more assignments for credit than when assignments were optional.\u003c/p\u003e \u003cp\u003eBecause some students completed fewer than the full number of flashcard stacks in a given assignment, we also computed the percentage of stacks that each student completed. A similar pattern to completed assignments occurred: Students completed an average of 91.1% of for-credit stacks and 1.1% of optional stacks when Set A was assigned, and an average of 99.2% of for-credit stacks and 0.5% of optional stacks when Set B was assigned.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eRetention of Key Terms on the Interim Test\u003c/h2\u003e \u003cp\u003eTo evaluate the impact of using SR on retention, the second author (who is the instructor for the course) scored the responses based on the presence of correct main ideas in the student response. In particular, each answer was separated into main idea units by the instructor in a manner that intuitively captured the key ideas in each concept (the idea units for each concept are presented in \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e). We computed the proportion of main ideas each student had in their answer (for example, if a student\u0026rsquo;s answer contained 1 of 3 idea units, that particular response received a score of .33). We then calculated mean proportion correct for each question (see Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance did not reveal a significant main effect for credit, \u003cem\u003eF\u003c/em\u003e(1,97)\u0026thinsp;=\u0026thinsp;1.20, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.28, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.10, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.012, but did reveal a significant effect for set, which indicated that performance was lower for Set B than Set A content, \u003cem\u003eF\u003c/em\u003e(1,97)\u0026thinsp;=\u0026thinsp;18.31, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.02, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.16. Most important, the significant interaction indicated that performance was greater when students received credit for using successive relearning than when it was optional, \u003cem\u003eF\u003c/em\u003e(1,97)\u0026thinsp;=\u0026thinsp;34.72, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.02, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.26.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eIn-Class, Open-Book Exam Performance\u003c/h2\u003e \u003cp\u003eFor the open-book exam, we compared performance on exam questions that tapped into concepts that students received credit to successively relearn compared to concepts that were from optional assignments. Given that the exam was open book, performance was high (and in some cases, very close to the ceiling): For Set A concepts, correct performance was near the ceiling regardless of whether the concepts were assigned for credit (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;94.4, \u003cem\u003eSEM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1.4, \u003cem\u003eMdn\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1) or were optional (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;94.0, \u003cem\u003eSEM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1.6, \u003cem\u003eMdn\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1).\u003c/p\u003e \u003cp\u003eFor Set B concepts, mean percent correct performance was 85.4 (2.2) and 94.0 (1.6) when the concepts were assigned for credit or were optional (Medians\u0026thinsp;=\u0026thinsp;.95 and .90), respectively. A 2 X 2 ANOVA reveal only a significant effect for Set, \u003cem\u003eF\u003c/em\u003e(1,99)\u0026thinsp;=\u0026thinsp;16.18, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.01, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.14, indicating that questions tapping Set B concepts were more difficult than those tapping Set A concepts. The main effect of credit (credit versus optional) was not significant, \u003cem\u003eF\u003c/em\u003e(1,99)\u0026thinsp;=\u0026thinsp;2.1, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.15, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.02, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.02, nor was the interaction between credit and set, \u003cem\u003eF\u003c/em\u003e(1,99)\u0026thinsp;=\u0026thinsp;2.7, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.10, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.01, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.03.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eRetention After a Three-Month Delay\u003c/h2\u003e \u003cp\u003eAs per the interim test, the second author scored the responses for the 3-month delayed test. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, the pattern of outcomes was similar to the interim test, although forgetting occurred over the three month delay. A 2 (Credit: credit vs. optional) X 2 (Set: A vs. B) analysis of variance did not reveal a significant main effect for credit, \u003cem\u003eF\u003c/em\u003e(1, 16)\u0026thinsp;=\u0026thinsp;.31, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.59, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.017, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.019, but did reveal a significant effect for set, which indicated that performance was lower for Set B than Set A content, \u003cem\u003eF\u003c/em\u003e(1, 16)\u0026thinsp;=\u0026thinsp;14.13, \u003cem\u003ep\u0026thinsp;=\u003c/em\u003e\u0026thinsp;.002, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.141, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.469. Most important, the significant interaction indicated that performance was greater when students received credit for using successive relearning than when it was optional, \u003cem\u003eF\u003c/em\u003e(1, 16)\u0026thinsp;=\u0026thinsp;13.60, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.002, \u003cem\u003eMSE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.14, \u003cem\u003epartial η\u003c/em\u003e\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.46 .\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn the current experiment, even when all assignments came with a schedule, students completed significantly more assignments when they received course credit versus when the assignments were made available but optional. And, for the latter, students completed close to none of the assignments and did not engage in the spaced practice of the same material across multiple sessions that is needed to implement successive relearning. Moreover, students performed significantly better on concepts that they had received credit for successively relearning as compared to concepts from the optional assignments, although the positive impact of successive relearning was relatively small for the more difficult item set (see Figs.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e \u0026amp; \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eOne explanation for why students did not complete most of the optional assignments is related to the design of Experiment 1. The website in which students completed assignments presented both types of assignments side by side. In particular, to complete an assignment, students were shown a drop-down menu where they could select to complete the for-credit assignments (labelled \u0026ldquo;Assigned\u0026rdquo;) or the optional assignments (labelled \u0026ldquo;Optional\u0026rdquo;). The simultaneous presentation (i.e., juxtaposing \u0026ldquo;Assigned\u0026rdquo; with \u0026ldquo;Optional\u0026rdquo; on the menu) may have led students to perceive the for-credit assignments as more worthwhile, which in turn may have led them to not want to do the optional assignments when they could do others for course credit. Likewise, after completing a for-credit assignment, students would need to go back to the main menu to select the optional assignment, and the extra effort may have curtailed their motivation to do more. Accordingly, if the two kinds of assignments were not contrasted in this manner, students might complete more optional assignments, which we evaluated in Experiment 2.\u003c/p\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eExperiment 2\u003c/h2\u003e \u003cp\u003eIn Experiment 1, the interface of the website presented the two types of assignments side by side, which may have led students to under value the optional assignments. Thus, in Experiment 2, we evaluated whether students (from a different cohort) completed assignments when they did not have both kinds of assignments available to them. In particular, students first received optional assignments (and instructions on how to use the successive relearning program) at the beginning of the term, so as to evaluate how much they would use them when for-credit assignments were not available. Then, later in the semester, new assignments were provided in which students could earn course credit to ensure that the particular group of students would complete for-credit assignments as per usual.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eMethod\u003c/h2\u003e \u003cp\u003eWe conducted Experiment 2 in the same Principles of Medical Research course with a new sample of students in another semester. In the first half of the class, all students had access to four optional successive relearning assignments. In the second half of the class, all students were required to complete four different successive relearning assignments. The materials were identical to Experiment 1 but redistributed. About half of the content was used as the optional assignments in the first half of the class, and the other half was used as the for-credit assignments (see \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e for how the content was distributed). The assignment protocol remained the same.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eResults\u003c/h3\u003e\n\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eHow Often Did Students Use Successive Relearning?\u003c/h2\u003e \u003cp\u003eStudents completed none (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0, \u003cem\u003eSEM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0) of the optional assignments, whereas they completed the vast majority (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;96.9%) of assignments when given for-credit assignments later in the semester. Thus, consistent with Experiment 1 outcomes, even when optional assignments were the only ones available, students still did not complete them, suggesting that some incentive may be necessary to motivate students to use successive relearning.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eGeneral Discussion\u003c/h2\u003e \u003cp\u003eAcross two experiments, we investigated the extent to which students would engage in successive relearning when the assignments were for credit or optional. Students completed the vast majority of assignments when they received course credit, but completed almost none when the assignments were optional. Given the benefits of successive relearning on retention of material, students\u0026rsquo; unwillingness to complete the optional assignments was rather surprising, especially given that (a) the instructor developed the content and indicated that the assignments included key concepts in the course and (b) Principles of Medical Research is a critical course for first year podiatric medical students. That is, both studies were conducted in a required course for these students, and failing this course had consequences for their progress. These outcomes and those from prior studies demonstrate that students are not always motivated to engage in extra study \u0026ndash; even in the present case involving medical students \u0026ndash; when it is optional.\u003c/p\u003e \u003cp\u003eOn the optimistic side, students were willing to complete the successive relearning assignments when they received a relatively small amount of course credit to complete them. Thus, instructors who desire to motivate their students to use a particular learning strategy for completing homework assignments can likely do so by providing a relatively small proportion of the class grade for completion. Nevertheless, strategies like successive relearning show a great deal of promise for helping students to gain mastery of a particular topic \u0026ndash; both in obtaining the content as well as maintaining over time (for a review, see Rawson \u0026amp; Dunlosky, 2022), so it may benefit students to adopt this strategy to master any content even when they are not given an incentive to do so. A question becomes, how might students be motivated to use the strategy on their own?\u003c/p\u003e \u003cp\u003eWe suspect that answers to this question will involve enhancing students\u0026rsquo; perceived value of successive relearning. How might this be done, and if so, what are possible tactics to increase its perceived value? First, some students may view successive relearning as having value at least in some contexts. For instance, in surveys about students use of flashcards (Zung et al., 2022; Wissman et al., 2012), the majority of students report using flashcards to learn simple associations, such as a foreign language. The surveys do not focus on successive relearning per se, but most students report using flashcards to practice until they can correctly recall all the to-be-learned material at least once. By contrast, very few students reported using flashcards to study more complicated material, such as the definitions of concepts (as in the present research). Lack of using successive relearning in this case may arise from several barriers: (a) students may not want to spend the time developing flashcards for complicated material, (b) they may not realize that this approach is beneficial for learning complex materials, or (c) they may view the time and effort required to test themselves on difficult definitions as being overly burdensome as compared to not studying or merely restudying the material. The first barrier is not relevant in the current experiments in which the flashcards were populated with concepts chosen by the instructor and were prepared ahead of time. The second two barriers may apply, however: Students may not believe that flashcards are useful for complex material, and learning a difficult concept well enough to correctly recall during a given session does take real time.\u003c/p\u003e \u003cp\u003ePrescriptions on how to overcome these barriers are available from McDaniel and Einstein (2020), who described a framework for motivating students to adopt effective study strategies. The framework has four components: knowledge, belief that the learning strategy is valuable, commitment to using the strategy, and a plan of action for implementing it. Accordingly, to motivate students, an instructor could explain how to use successive relearning to give students the knowledge they need to complete successive relearning assignments. Second, to help students realize this strategy is effective, the instructor could give students experience with successive relearning, such as by requiring some assignments in the beginning of the class, and then switching to optional assignments after students have had time to experience its benefits. Third, students could build commitment to the strategy by responding to a short prompt asking them to draw an explicit connection between the learning strategy and how it can be used to achieve their educational goal. Finally, the instructor could have students write a brief outline of their plan for using successive relearning (e.g. where the student will get the materials for the questions and when they intend to complete spaced learning sessions). The degree to which such approaches improve students use of successive relearning is an important avenue for future classroom-based research.\u003c/p\u003e \u003cp\u003eFinally, in the present context, it was evident that to retain the epidemiology principles, students would need to engage in more successive relearning. For instance, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, three months after class, students forgot a great deal of what they had initially learned. One recommendation here would be to include booster sessions in which students successively relearn the materials over longer-and-longer delays until it is fluently accessed even after long delays. Of course, such booster sessions would require planning and likely should focus just on the critical material that students need to fluently remember for subsequent courses and in their jobs. That is, given that obtaining and maintaining knowledge can take a meaningful amount of time, strategies like successive relearning may best be used for just the content that is most essential to remember over the long term.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusions","content":"\u003cp\u003eWhile successive relearning is beneficial for medical student learning, many students are unlikely to complete successive relearning assignments unless they are given course credit. Fortunately, it appears a relatively trivial amount of credit is sufficient to motivate medical students to complete successive relearning assignments.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eStatements and Declarations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eWriting of this manuscript was partially supported by the National Science Foundation under Grant IUSE-1914499.\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003eThe authors declare no other conflicts of interest.\u003c/p\u003e\n\u003cp\u003eEthics approval\u003c/p\u003e\n\u003cp\u003eThese experiments were approved by the Kent State University Institutional Review Board.\u003c/p\u003e\n\u003cp\u003eInformed consent\u003c/p\u003e\n\u003cp\u003eParticipants were informed about the experiments, and informed consent was obtained from all participants.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent to publish\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAs the present experiments used aggregate, anonymized data, consent to publish was not obtained. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eData and analyses\u003c/p\u003e\n\u003cp\u003eData and analyses are available upon request.\u003c/p\u003e\n\u003cp\u003eAuthor contributions\u003c/p\u003e\n\u003cp\u003eConceptualization: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Methodology: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Formal analysis and investigation: Maren Greve, John Dunlosky; Writing \u0026ndash; original draft preparation: Maren Greve, John Dunlosky; Writing \u0026ndash; review and editing: Maren Greve, Jill Kawalec, Viveka Jenks, John Dunlosky; Funding acquisition: John Dunlosky\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBadali, S., \u0026amp; Greve, M. (2023). \u003cem\u003eCan successive relearning enhance performance on application-based exam questions\u003c/em\u003e? Under review.\u003c/li\u003e\n\u003cli\u003eDunlosky, J., \u0026amp; O\u0026apos;Brien, A. (2020). The power of successive relearning and how to implement it with fidelity using pencil and paper and web-based programs. \u003cem\u003eScholarship of Teaching and Learning in Psychology, 8,\u003c/em\u003e225-235. https://doi.org/10.1037/stl0000233\u003c/li\u003e\n\u003cli\u003eGong, J., Liu, T. X., \u0026amp; Tang, J. (2021). How monetary incentives improve outcomes in MOOCs: Evidence from a field experiment. \u003cem\u003eJournal of Economic Behavior \u0026amp; Organization, 190\u003c/em\u003e, 905\u0026ndash;921. https://doi.org/10.1016/j.jebo.2021.06.029\u003c/li\u003e\n\u003cli\u003eHigham, P. A., Zengel, B., Bartlett, L. K., \u0026amp; Hadwin, J. A. (2021). The benefits of successive relearning on multiple learning outcomes. \u003cem\u003eJournal of Educational Psychology, 114, \u003c/em\u003e928-944. https://doi.org/10.1037/edu0000693\u003c/li\u003e\n\u003cli\u003eJanes, J. L., Dunlosky, J., Rawson, K. A., \u0026amp; Jasnow, A. (2020). Successive relearning improves performance on a high‐stakes exam in a difficult biopsychology course. \u003cem\u003eApplied Cognitive Psychology\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e, 1118-1132. \u003c/li\u003e\n\u003cli\u003eKontur, F. J., \u0026amp; Terry, N. B. (2014). Motivating students to do homework. \u003cem\u003eThe Physics Teacher, \u003c/em\u003e52.\u003c/li\u003e\n\u003cli\u003eNevid, J. S., \u0026amp; Gordon, A. J. (2018). Integrated Learning Systems: Is There a LearningBenefit? \u003cem\u003eTeaching of Psychology\u003c/em\u003e, \u003cem\u003e45\u003c/em\u003e, 340-345. \u003c/li\u003e\n\u003cli\u003ePlanchard, M., Daniel, K. L., Maroo, J., Mishra, C., \u0026amp; McLean, T. (2015). Homework, Motivation, and Academic Achievement in a College Genetics Course\u003c/li\u003e\n\u003cli\u003eRadhakrishnan, P., Lam, D., \u0026amp; Ho, G. (2009). Giving University Students Incentives to do Homework Improves their Performance. \u003cem\u003eJournal of Instructional Psychology, 36\u003c/em\u003e, 219\u0026ndash;225.\u003c/li\u003e\n\u003cli\u003eRawson, K. A., \u0026amp; Dunlosky, J. (2022). Successive relearning: an underexplored but potent technique for obtaining and maintaining knowledge. \u003cem\u003eCurrent Directions in Psychological Science\u003c/em\u003e, 31, 362-268. doi.org/10.1177/09637214221100484\u003c/li\u003e\n\u003cli\u003eRawson, K. A., Dunlosky, J., \u0026amp; Sciartelli, S. M. (2013). The power of successive relearning: Improving performance on course exams and long-term retention. Educational Psychology Review, 25, 523\u0026ndash;548. https://doi-org.proxy.library.kent.edu/10.1007/s10648-013-9240-4\u003c/li\u003e\n\u003cli\u003eRyan, C. S., \u0026amp; Hemmes, N. S. (2005). Effects of the contingency for homework submission on homework submission and quiz performance in a college course. \u003cem\u003eJournal of Applied Behavior Analysis, 38\u003c/em\u003e, 79-88.\u003c/li\u003e\n\u003cli\u003eSoderstrom, N. C., \u0026amp; Bjork, R. A. (2015). Learning versus performance: An integrative review. \u003cem\u003ePerspectives on Psychological Science, 10\u003c/em\u003e, 176-199.\u003c/li\u003e\n\u003cli\u003eTrumbo, M. C., Leiting, K. A., McDaniel, M. A., \u0026amp; Hodge, G. K. (2016). Effects of reinforcement on test-enhanced learning in a large, diverse introductory college psychology course. \u003cem\u003eJournal of Experimental Psychology: Applied, 2\u003c/em\u003e, 148-160.\u003c/li\u003e\n\u003cli\u003eWissman, K. T., Rawson, K. A., \u0026amp; Pyc, M. A. (2012). How and when do students use flashcards? \u003cem\u003eMemory, 20\u003c/em\u003e, 568-579.\u003c/li\u003e\n\u003cli\u003eZung, I., Imundo, M. N., \u0026amp; Pan, S. C. (2022). How do college students use digital flashcards during self-regulated learning? \u003cem\u003eMemory, 30\u003c/em\u003e, 923\u0026ndash;941. https://doi.org/10.1080/09658211.2022.2058553\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Appendix","content":"\u003cp\u003eThe \u0026ldquo;prompt\u0026rdquo; column shows the question students were prompted to answer. The \u0026ldquo;answer\u0026rdquo; column shows the answer that students were shown as feedback. Slash marks indicate where the answer was split up to form idea units (no slash indicates that the entire answer fit into a single idea unit). Italics indicates this text was not included in the idea units (but was still shown in the answer feedback). Some idea units were changed (e.g. for formulas involving division, the idea units were written as \u0026ldquo;divided by\u0026rdquo; instead of the slash mark used in the full answer) or lightly paraphrased in order to fit into the maximum character limit of the idea units (e.g. \u0026ldquo;Probability that an individual would experience an outcome\u0026rdquo; was changed to \u0026ldquo;Probability of experiencing an outcome\u0026rdquo;).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn Experiment 1, for the three stacks within a set (either Set A or Set B), a student received credit for completing assignments for one set (e.g., A) and did not receive credit for the other, optional set (i.e., B); which set was slated as for-credit or optional was counterbalanced across students. \u0026nbsp;Both sets were presented simultaneously\u003c/p\u003e\n\u003cp\u003eIn Experiment 2, Set A was given as the optional assignments, after which Set B was assigned as the for-credit assignments.\u0026nbsp;\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"575\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.67361111111111%\" valign=\"top\"\u003e\n \u003cp\u003eSet\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"14.0625%\" valign=\"top\"\u003e\n \u003cp\u003eStack\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"38.888888888888886%\" valign=\"bottom\"\u003e\n \u003cp\u003ePrompt\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"34.375%\" valign=\"top\"\u003e\n \u003cp\u003eAnswer\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.67361111111111%\" rowspan=\"23\" valign=\"top\"\u003e\n \u003cp\u003e\u0026nbsp;A\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"14.0625%\" rowspan=\"7\" valign=\"top\"\u003e\n \u003cp\u003eBasic Stats 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"38.888888888888886%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of a continuous variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"34.375%\" valign=\"top\"\u003e\n \u003cp\u003eData can take on / any value / within a set range\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of a discrete variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eData can be\u003c/em\u003e only one of limited number of values\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of a nominal variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eBy name only, / including / two or more categories / without a meaningful order\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of a dichotomous variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eA\u003c/em\u003e nominal variable / \u003cem\u003ewith\u003c/em\u003e only two categories\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of an ordinal variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eTwo or more categories / arranged in a meaningful order\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of an independent variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eIn an experiment, / \u003cem\u003ethe\u003c/em\u003e variable that is manipulated / to assess its effects\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of a dependent variable?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eOutcomes measured / in an experiment\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"16.10337972166998%\" rowspan=\"10\" valign=\"top\"\u003e\n \u003cp\u003eEpi 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.53280318091451%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of epidemiology?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"39.363817097415506%\" valign=\"top\"\u003e\n \u003cp\u003eBranch of medicine / \u003cem\u003ethat deals with the\u003c/em\u003e incidence, distributions and possible control of / diseases and other factors relating to health\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does HRE stand for in epidemiology?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eHealth-related event\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of incidence?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eNumber of new occurrences of an HRE in a population / during a specific time / (risk of contracting HRE)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of prevalence?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eNumber of existing cases in a population / during a specific time / (how widespread the HRE is)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of relative risk?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eThe\u003c/em\u003e ratio of / risk in the exposure group / \u003cem\u003eto the\u003c/em\u003e risk in the control group\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of \u0026quot;risk\u0026quot;?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eThe probability of an event occurring\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for probability?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e(# of cases of one event) \u0026nbsp;/ \u0026nbsp;divided by / (overall # of cases)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does a relative risk above 1.0 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eRisk of developing an HRE / \u003cem\u003eis\u003c/em\u003e higher for exposure group\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does a relative risk equals to 1.0 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eRisk of developing an HRE / \u003cem\u003eis\u003c/em\u003e the same for exposure group\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does a relative risk below 1.0 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eRisk of developing an HRE / \u003cem\u003eis\u003c/em\u003e lower for exposure group\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"16.10337972166998%\" rowspan=\"6\" valign=\"top\"\u003e\n \u003cp\u003eEpi 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.53280318091451%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of sensitivity?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"39.363817097415506%\" valign=\"top\"\u003e\n \u003cp\u003eAbility of a test to detect an HRE / \u003cem\u003ein an individual\u003c/em\u003e when HRE is present\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhy is high sensitivity important?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eYou will have\u003c/em\u003e few false negatives\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of specificity?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eThe\u003c/em\u003e ability of a test to indicate non-HRE / \u003cem\u003ein an individual\u003c/em\u003e when no HRE is present\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhy is high specificity important?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eYou will have few\u003c/em\u003e false positives\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is positive predictive value?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eProportion of individuals / who \u003cem\u003eare\u003c/em\u003e screened positive by a test / \u003cem\u003eand\u003c/em\u003e actually have the HRE\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is negative predictive value?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eProportion of individuals / who \u003cem\u003eare\u003c/em\u003e screened negative by a test / \u003cem\u003eand actually\u003c/em\u003e do NOT have the HRE\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.67361111111111%\" rowspan=\"23\" valign=\"top\"\u003e\n \u003cp\u003eB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"14.0625%\" rowspan=\"7\" valign=\"top\"\u003e\n \u003cp\u003eBasic Stats 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"38.888888888888886%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the mean of a set of numbers?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"34.375%\" valign=\"top\"\u003e\n \u003cp\u003eSum of the set of values / divided by the number of values in the set\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the median of a set of number?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eThe\u003c/em\u003e middle value / in a data set\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the mode?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eThe most frequent score in a data set\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the range?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eA\u003c/em\u003e measure of dispersion \u003cem\u003ethat is equal to the\u003c/em\u003e / difference between the largest and smallest values / in a data set\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the variance?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eA\u003c/em\u003e measure of dispersion / \u003cem\u003ethat is the\u003c/em\u003e average of the squared differences of the means\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the standard deviation?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eA\u003c/em\u003e measure of dispersion /\u003cem\u003e\u0026nbsp;that is\u003c/em\u003e the square root of the variance\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is a normal distribution?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eSymmetric bell-shaped curve / \u003cem\u003ewhere the\u003c/em\u003e mean, median and mode are all equal\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"16.10337972166998%\" rowspan=\"8\" valign=\"top\"\u003e\n \u003cp\u003eEpi 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.53280318091451%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for the odds ratio?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"39.363817097415506%\" valign=\"top\"\u003e\n \u003cp\u003e(odds of HRE in exposure group) / divided by / (odds of HRE in the control group)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of \u0026quot;odds\u0026quot;?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cem\u003eThe\u003c/em\u003e likelihood of an event occurring\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for odds?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e(# of cases of one event) / divided by / (# of cases of alternate event)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does an odds ratio greater than 1 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eExposure associated with higher odds of outcome\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does an odds ratio equal to 1 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eExposure does not affect odds of outcome\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does an odds ratio less than 1 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eExposure associated with lower odds of outcome\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the definition of hazard?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eProbability of experiencing an outcome / at a specific point in time\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for hazard ratio?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e(hazard in the exposure group) /divided by / ( hazard in the control group)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"16.10337972166998%\" rowspan=\"8\" valign=\"top\"\u003e\n \u003cp\u003eEpi 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.53280318091451%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is likelihood ratio?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"39.363817097415506%\" valign=\"top\"\u003e\n \u003cp\u003eProbability that a specific / patient has a specific HRE\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat are likelihood ratios used for?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eTo assess the diagnostic accuracy of a test\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for likelihood ratio?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e(prob\u003cem\u003eability a\u003c/em\u003e person w/HRE has \u003cem\u003ea specific\u003c/em\u003e test result) / divided by / (prob\u003cem\u003eability\u003c/em\u003e a person without an HRE has the \u003cem\u003especific\u003c/em\u003e test result)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does a likelihood ratio equaling 1 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eDiagnostic test has no value\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for LR+?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003esensitivity / divided by / (1 \u0026ndash; specificity)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does an LR+ between 5 and 10 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eTest result has a moderate effect / on increasing probability of HRE\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat is the formula for LR-?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003e(1 \u0026ndash; sensitivity) / divided by / specificity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"53.08056872037915%\" valign=\"bottom\"\u003e\n \u003cp\u003eWhat does an LR- between 0.5 and 0.1 mean?\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"46.91943127962085%\" valign=\"top\"\u003e\n \u003cp\u003eTest result has a moderate effect / on decreasing probability of HRE\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Footnotes","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAlthough similar, a stack and a program are not interchangeable terms. A stack is a set of to-be-learned items. Stacks are not necessarily used in a way that instantiates SR. A SR program uses stacks but also includes the infrastructure that supports the use of SR by scheduling distributed practice session, dropping items after reaching criterion in a given session, providing feedback, and so forth.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Medical education, Successive relearning, Retrieval practice, Spaced practice","lastPublishedDoi":"10.21203/rs.3.rs-3061046/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3061046/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSuccessive relearning (SR) combines retrieval practice across spaced study sessions. In particular, students attempt to recall to-be-learned information (with feedback) during each session until all concepts are correctly recalled and then return to the same material to repeat retrieval practice on multiple spaced practice sessions. Thus, students must begin using this technique several weeks before an exam, which may decrease their motivation for using it. The main question for the present research is: Will providing students with a small amount of class credit increase their likelihood of engaging in SR? First-year medical students in a Principles of Medical Research course were provided with an SR program that included (a) virtual flashcards containing definitions of statistical concepts (e.g., levels of measurement, central tendency, odds ratios, etc) that students practiced retrieving, (b) feedback after each retrieval attempt, and (c) schedules for using each virtual flashcard stack across three spaced practice sessions. Students received incentives in terms of class credit to complete SR sessions for half of the content but no incentive for the other half. A delayed practice test was administered to evaluate the impact of SR on retention. Using SR (vs. not using it) did boost retention of the concepts. And, most important, credit had a major impact, with students completing over 90 percent of the SR sessions that were assigned to receive credit but under 10 percent of the SR sessions that were optional.\u003c/p\u003e","manuscriptTitle":"The Effect of Incentives on the Use of Successive Relearning for Retaining Statistics and Epidemiology Concepts in a Medical Research Course","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-06-16 18:42:23","doi":"10.21203/rs.3.rs-3061046/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5ec50820-58e2-4d82-b131-f7d012d12758","owner":[],"postedDate":"June 16th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-07-06T20:29:20+00:00","versionOfRecord":[],"versionCreatedAt":"2023-06-16 18:42:23","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3061046","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3061046","identity":"rs-3061046","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}