Assessment of Entrustable Professional Activities among Dutch Endocrine Supervisors

Objective: Entrustable Professional Activities (EPAs) are an important tool to support individualization of medical training in a competency-based setting and are increasingly implemented in the clinical speciality training for endocrinologist. This study aims to assess interrater agreement and factors that potentially impact EPA scores. Design: A case-vignette study using standardized written cases Methods: Case vignettes (n=6) on the topics thyroid disease, pituitary disease, adrenal disease, calcium and bone disorders, diabetes mellitus, and gonadal disorders were written by two endocrinologists and a medical education expert and assessed by endocrinologists experienced in the supervision of residents in training. Primary outcome is the interrater agreement of entrustment decisions for endocrine EPAs among raters. Secondary outcomes included the dichotomous interrater agreement (entrusted vs. non-entrusted), and an exploration of factors that impact decision-making. The study protocol was registered and approved by the Ethical Review Board of the Netherlands Association for Medical Education (NVMO-ERB # 2020.2.5). Results: Nine endocrinologists from six different academic regions participated. Overall, the Fleiss Kappa for the EPA level was 0.11 (95% CI: 0.03-0.22) and for the entrustment decision 0.24 (95% CI 0.11-0.37). Of the five features that impacted the entrustment decision, capability was ranked as the most important by a majority of raters (56%- 67%) in every case. Conclusions: There is a considerable discrepancy between the EPA levels assigned by different raters. These findings emphasize the need to base entrustment decisions on multiple observations, made by a team of supervisors and enriched with factors other than direct medical competence.