From Summative Scores to Formative Growth: ChatGPT as a Feedback Agent in Education

Providing effective feedback is one of the most promising ways in which large language models (LLMs) can transform education, particularly by offering scalable and personalized support to learners. We present systematic evaluation of an LLM – ChatGPT – in delivering both summative and formative feedback, using two sequent studies to assess its ability to provide effective feedback. Study 1 employed a quantitative approach to examine the quality of ChatGPT's summative feedback by comparing it to well-established human ratings, using Intraclass Correlation Coefficients (ICCs) to evaluate interrater agreement and consistency. Results indicated moderate absolute agreement but high consistency, particularly when prompts were tailored with domain-specific information, suggesting that LLMs can reliably deliver feedback when aligned with specific assessment criteria. Study 2 adopted a single-blind randomized controlled design to investigate the impact of formative feedback generated by ChatGPT compared to a well-established automated writing evaluation (AWE) tool. Multiple measures, including writing quality, perceived feedback quality, and writing motivation, were used to assess the efficacy of the feedback. Both tools effectively improved writing quality; however, only ChatGPT's feedback significantly enhanced student motivation, underscoring the potential of LLMs to foster greater engagement in learning beyond performance improvement. These findings highlight the transformative potential of LLMs like ChatGPT to provide scalable, personalized, and consistent educational support, which could play a critical role in reducing educational inequalities. Further refinement is needed to enhance the accuracy of LLM-generated feedback and to address ethical and pedagogical considerations for real-world implementation.