Development and Validation of the Anesthesia Critical-Thinking Test (ACTT)

Background: Despite critical thinking being a crucial skill in anesthesia practice, no specific tool exists to evaluate the skill in anesthesia training. The present study measured the validity, reliability and practicality of the Anesthesia Critical-Thinking Test (ACTT) for measuring the stages of critical thinking in anesthesia settings. Methods: : We developed the ACTT to evaluate critical-thinking skills by ranking learners into 6 critical-thinking stages: 1) unreflective thinker, 2) beginning critical thinker, 3) practicing critical thinker, 4) advanced critical thinker, 5) accomplished critical thinker and 6) challenged thinker. Content validity was assessed by 5 experts. Concurrent validity, reliability and practicality were tested by 2 evaluators. We retrospectively and randomly assessed 47 long-essay assignments written by 1 st –3 rd year anesthesia residents in 2018. The assignments were subsequently blindly reviewed by 2 researchers to assess the critical-thinking stages. Results: : The ACTT’s items were adjusted for content validity until the item-objective congruence was > 0.5. Concurrent validity between the scores and the critical-thinking stages showed moderate correlation by both evaluators (Evaluator 1 r s 0.574, P < 0.001; Evaluator 2 r s 0.603, P < 0.001). High-performance learners (stages 3, 4 and 5) had significantly higher scores than low-performance learners (stages 1 and 2), with P = 0.001. Interrater reliability was strongly correlated (r s 0.866, P < 0.001). Both evaluators were satisfied with the tool’s practicality. Conclusions: : The ACTT provides a valid, reliable and practical means of evaluating the critical-thinking skills of anesthesia residents.