Adjusting for Publication Bias Reveals No Evidence for the Effect of ChatGPT on Students’ Learning Performance, Learning Perception, and Higher-Order Thinking

Students increasingly use large language models such as ChatGPT to help them with their study tasks. Consequently, there is an acute interest in ascertaining and quantifying the effects of large language models in educational settings. In a recent article, Wang and Fan1 conducted a comprehensive meta-analysis on the impact of ChatGPT on students’ learning performance, learning perception, and higher-order thinking featuring 51 studies. Wang and Fan1 conclude that “ChatGPT has a large positive impact on improving learning performance (g = 0.867) and a moderately positive impact on enhancing learning perception (g = 0.456) and fostering higher-order thinking (g = 0.457).” Here we show that these effects greatly diminish once publication bias is accounted for, and the evidence in favor of the benefits disappears. In order to properly evaluate the benefit of large language models in educational settings, we believe that it is essential to design high-quality, pre-registered experiments.