An empirical approach of identifying subject matter experts (SMEs) for the development of situational judgment tests

The development of a scoring key for the situational judgment test (SJT) often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran-Weiss-Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS-index. Next, we demonstrate how the CWS-index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found the CWS index was associated with reflective thinking and intuition avoidance.