Evaluating Large Language Models for Assisting in Meta-Analysis

Large language models (LLMs) are receiving increased attention in academia as aids for scientific research due to their superior performance in tasks related to natural language processing and understanding. Meta-analysis, a research method involving extensive text processing to extract and code qualitative and quantitative information from empirical studies, is particularly well-suited to the application of LLMs. In this study, we empirically evaluated the ability of LLMs to perform automatic coding tasks within meta-analytic contexts, using Bing Chat (based on GPT-4.0) and ChatPDF (based on GPT-3.5) as examples. Our findings indicate that Bing Chat outperformed ChatPDF in accurately extracting and coding qualitative information such as publication type, country, and survey methods. However, its performance decreased when handling quantitative data, such as correlation coefficients. We also noted an upward trend in Bing Chat's performance over time. The potential and utility of LLMs in facilitating meta-analysis from a researcher’s perspective are further discussed.