Automated assessment of the trustworthiness of biomedical publications: a protocol for a scoping review

Background: Trustworthy scientific evidence can guide clinical practice and health policy. A range of terms, including methodological quality, rigor, confidence, internal validity, risk of bias, reproducibility, transparency, and statistical validity, have been used to refer to overlapping but distinct aspects of a study’s trustworthiness. Ambiguous language describing related concepts complicates human evaluation and poses challenges for the development of automated methods for assessing scientific evidence. Recent advances in natural language processing (NLP) and artificial intelligence (AI) have enabled large-scale analyses of biomedical literature, including automated detection of constructs related to trustworthiness. Objective: This scoping review aims to examine how studies developing or applying automated methods operationalize and assess constructs related to trustworthiness in biomedical literature. Methods: We will search PubMed, IEEE Xplore, ACM Digital Library, and ACL Anthology, with no restrictions on language or publication types. We will limit the search to articles published after 2000. We will include studies that evaluate computational methods and tools to assess biomedical publications for constructs related to trustworthiness. Two authors will independently screen records and extract data. We will extract study characteristics, method/tool and dataset details, how trustworthiness-related constructs are operationalized for automated assessment, and evaluation outcomes, including agreement metrics, performance metrics, and utility indicators. We will report and synthesize the findings using structured tables, visualizations, and qualitative analyses. Discussion: This scoping review will provide a comprehensive synthesis of datasets and computational tools for assessing constructs related to trustworthiness of biomedical studies, offering stakeholders a clear overview of existing approaches, highlighting commonalities and differences across the literature, and identifying priorities for future research.