Generating Testlets to Measure Clinical Reasoning Skills in Medicine

Clinical reasoning is a core competency required by all medical professionals. A testlet is group of two or more items based on the same clinical scenario. A testlet can be used to measure clinical reasoning skills because each scenario is evaluated with two or more items. Unfortunately, testlets are challenging and time consuming to create. The purpose of our study is to address the item writing challenge by describing and illustrating testlet-based automatic item generation. We use this method to create testlets for evaluating clinical reasoning skills across four different scenarios in thoracic surgery. We created a testlet-based item model. The item model contains global and local variables. Global variables can be used to place content in any item across the testlet and hence are unique to testlet-based automatic item generation. Local variables are specific to each item model and can only be used for one specific item in the testlet. We generated 47 unique 3-item testlets. A sample of three, 3-item testlets, one from each clinical scenario, was independently evaluated by four surgical content experts and judged to be of high quality. Directions for future research are also discussed.