Diagnostic Assessment of Deep Understanding Using Cognitive Diagnostic Models: A Large-Scale Assessment to Promote the Use of Effective Learning Strategies

Recent educational goals have focused on achieving deep understanding and promoting the use of effective learning strategies. Previous studies showed that cognitive diagnostic models (CDMs) help diagnose students' depth of understanding. However, the approaches in specifying attributes (i.e., Q-matrix) to diagnose the depth of understanding remain unexplored; therefore, utilizing large-scale assessments to capture the general trends in the students' mastery of the depth of understanding is an uncharted area in this field. This study explores which attribute expression (i.e., linear hierarchy or polytomous attribute) is more appropriate, based on the CDM analysis of a large-scale assessment in mathematics. This is the first study to apply the variational Bayesian estimation to address the intensive computational load in CDM applications. The results indicate that a Q-matrix employing a linear hierarchy is more appropriate than employing polytomous attributes. The estimation results of the linear hierarchy suggest that less than 30% of the sampled students achieved a deep understanding of procedures/formulas, whereas more than half of the students achieved a shallow understanding. Regarding the understanding of terms, less than 35% of students achieved either shallow or deep understanding. These results may help design and improve future learning strategy instructions.