Evaluation of Transformer-Based Neural Language Models for Writing Feedback and Automated Essay Scoring

Abstract Writing remains a challenging skill for many students due to inadequate feedback tools. There is a need to develop more effective tools for supporting students’ writing skill development. This study aims to evaluate the effectiveness of transformer-based neural language models for assessing and automatically scoring argumentative essays written by 8th-12th grade English Language Learners (ELLs). The students’ English essays were assessed and scored based on six criteria, including cohesion, syntax, vocabulary, phraseology, grammar, and conventions. The models were trained on real teacher feedback from 2700 scored essays. We also compared various transformer-based neural language models to find the most effective model. Several metrics were used for evaluation, with the root mean square error (RMSE) as the primary measure. The results show that a specific model, DeBERTa-v3-large, outperforms others in most categories. In conclusion, this study suggests that transformer-based neural language models, especially when using the DeBERTa-v3-large model, hold significant promise in improving automated essay scoring and feedback, potentially leading to enhanced writing skills among English language learners.