Are the Lists of Questionable Journals Reasonable: A Case Study of Early Warning Journal Lists

Abstract The lists of questionable journals are regarded a policy or tool to ensure research quality and integrity. However, due to their lack of clear criteria, they remain highly debated. Taking a typological perspective, we assess the reasonableness of the lists of questionable journals by examining how well it reflects the differences in bibliometric attributes among distinct groups when categorizing and labelling them, and whether these differences are consistent. Using the Early Warning Journal Lists released by the National Science Library of the Chinese Academy of Sciences as an example, we grouped listed journals based on warning levels and years. Subsequently, we compared them in groups to determine if there are differences in key academic indicators among different warning categories, thereby evaluating the reasonableness of the warning journal list. Our research findings suggest that Early Warning Journal Lists may have employed inconsistent criteria when assigning warning levels. Variations in the degrees of differences or the absence of differences were observed among groups across different key academic indicators. Additionally, citation metrics like journal impact factor and journal citation indicator might not have been treated as grouping criteria in the Early Warning Journal Lists, yet this lack of detailed explanation from the creators is evident. This highlights the need for a more scientific and meticulous assessment of the lists of questionable journals, along with a greater emphasis on sharing detailed standards and data. Furthermore, our study offers recommendations for future formulation of lists of questionable journals by various institutions.