Automated Item Generation: – Impact of item variants on performance and standard setting

Background: Automated Item Generation (AIG) uses computer software to create multiple items from a single question model. Items generated using AIG software have been shown to be of similar quality to those produced using traditional item writing methods. However, there is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each. Methods: Pre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four ‘isomorphic’ 50-item MCQ tests using AIG software. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting). Results: 2218 students from 12 UK medical schools sat one of the four papers. The average facility of the four papers ranged from 0.55–0.61, and the cut score ranged from 0.58–0.61. Twenty item models had a facility difference >0.15 and 10 item models had a difference in standard setting of >0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility. Conclusions: Item facility varied to a greater extent than the standard set. This may relate to variants creating greater disruption of clinical reasoning strategies in novice learners as opposed to experts, in addition to the well documented tendency of standard setters to revert to the mean.