Testing Standards for AI-based Scores in Automated Essay Scoring

Recent developments in computer science and in particular in the field of artificial intelligence and machine learning allow the wide application of large language models for the evaluation of written text and other non-numerical data. When applied in the context of psychological and educational assessments, such models can be used for assigning scores to essays and other types of responses. In such as setting, traditional frameworks of test theory, such as classical test theory and item response theory, are no longer applicable, which leads to practical challenges in the evaluation of testing standards for scores obtained from AI models. To address these challenges, we discuss the evaluation of validity, fairness and reliability for scores obtained from models of artificial intelligence in the context of automated essay scoring. We further illustrate the proposed methods with an empirical example. The development of additional methods is suggested.