Score Comparability Under Changing Scoring Conditions in Rater-Scored Performance Assessment

Rater-scored performance assessments often treat observable fluency features (silent pauses, filled pauses, repetitions, and self-repairs) as stable indicators of underlying competence, assuming that their relationship to scores remains sufficiently invariant across scoring conditions (hereafter, scoring conditions): the stakes, instructions, rubrics, rater training, and task framing under which scores are produced and used. This paper argues that changes in scoring conditions can plausibly shift both (i) speakers’ internal thresholds for making speech difficulties visible and (ii) raters’ feature-weighting policies—that is, the relative importance raters assign to different observable performance features when assigning scores. The speaker-side claim is framed as a testable response-process hypothesis rather than as an established generalization. I introduce Retuning Tests (ReTs), design templates for diagnosing whether observed score differences reflect changes in performance, changes in rater feature weighting, or both, and illustrate their logic by re-examining published datasets and summary contrasts. The paper concludes with practical guidance for designing and reporting validity evidence for educational decisions under changing scoring conditions.