Analysis of Evidence-Rating Systems Used in Meta-Analyses of Pharmacotherapy

Analysis of Evidence-Rating Systems Used in Meta-Analyses of Pharmacotherapy | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research article Analysis of Evidence-Rating Systems Used in Meta-Analyses of Pharmacotherapy Alberto Frutos Pérez-Surio, José Manuel Vinuesa Hernando, Mercedes Arene Mendoza, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-42029/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background: Evidence-rating systems (ERSs) provide a framework for the systematic evaluation of the quality of individual interventional or observational studies and the overall body of evidence in meta-analyses. Authors and users of meta-analyses require a familiarity with ERSs to determine the level of confidence in the application of results. Many ERSs have been published, but no consensus exists regarding best practice for their use. Objective: The aim is to describe patterns of use of ERSs in meta-analyses of drug therapy published in contemporary high-impact medical journals. Methods: We design a review. Medline / PubMed was searched to identify meta-analyses evaluating drug therapy from the top 5 ranked general medical journals from 2012 to 2016. Methods of full-texts were reviewed to ensure the meta-analyses evaluated drug therapy and to identify the ERS used to rate individual studies and the overall body of evidence. Frequency of ERS use was analyzed using descriptive statistics. Results: The top-ranked journals were Ann Intern Med, BMJ, JAMA, Lancet and PLoS Medicine . Of the 309 results, manual review excluded 111 meta-analyses. In 198 evaluated meta-analyses, 86.4% (171) utilized an ERS; the most commonly used was the Cochrane Risk of Bias Tool in 80.7% (138) of all meta-analyses. An ERS was used to evaluate the body of literature in 19.1% (38) of meta-analyses; the most commonly used of three systems was the GRADE methodology. Overall, 14 unique ERSs, including author-defined systems, were used Conclusions: Most meta-analyses of drug effects in high-impact medical journals evaluated individual studies with an ERS, most commonly the Cochrane Risk of Bias Tool, while the use of ERSs to evaluate the body of literature was less frequent. The familiarity of authors and users of meta-analyses with commonly used ERSs may facilitate the evaluation and application of findings of meta-analyses. Health Economics & Outcomes Research Evidence-based medicine Systematic review Meta-analysis Drug information Pharmacy practice Figures Figure 1 Figure 2 Impacts On Practice Experience is crucial for the evaluation and application of findings of meta-analyses. Meta-analyses evaluating individual studies with an Evidence-Rating Systems were frequent compared to those evaluating the body of literature, and modified versions of Evidence-Rating Systems, although used, may not be optimal. It could be beneficial for journals to be more explicit in terms of detail and risk of bias, so further work is required to develop a general framework of best practices in Evidence-Rating Systems. Background Evidence-based clinical decision-making relies on well-designed research that utilizes rigorous methodology [ 1 , 2 ]. Ranking highly among the various clinical research designs are meta-analyses, which have a goal of quantitatively integrating findings of individual studies [ 3 ]. The results of meta-analyses may be more useful than those of the individual studies they contain; the integrated result may include a more representative population, so it possesses greater statistical power. Therefore, this provides more confidence in decision-making through more precise results and offer a more efficient review of a full body of literature [ 4 ]. These benefits of meta-analyses are particularly relevant in the assessment of drug effects, which are part of a voluminous bibliography. A cursory Medline search of publications indexed in 2016 revealed approximately 10,500 results for publications on clinical trials related to drug therapy alone. Additionally, the Clinicaltrials.gov website lists 138,002 clinical trials on drugs or biologics registered as of May 09, 2019 [ 5 ]. In spite of these benefits of meta-analyses, their results are only as reliable as those of the individual studies they include. For this reason, meta-analyses should evaluate the risk of bias of the individual studies they include. This practice is recommended by various sources, including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement, the Meta-analysis Of Observational Studies in Epidemiology: a proposal for reporting (MOOSE) statement, and the Cochrane Handbook on Systematic Reviews of Interventions [ 6 – 8 ]. These sources are designed to improve reporting and utility of meta-analyses, and to delineate the preferred methodology for meta-analyses produced by some of the world’s most recognized and authoritative groups of authors [ 6 – 9 ]. A study’s risk for bias may be assessed through evaluation of elements of research design (e.g., randomization and blinding), and other design flaws that lead to increased risk for selection, performance, information, and other biases [ 6 – 8 ]. Nonetheless, the assessment of study quality may present difficulties because the concept of “quality” is not well defined, not surprisingly, some assessments of markers of study quality (e.g., blinding) are judged subjectively and have poor inter-rater reliability [ 10 , 11 ]. In attempts to provide greater objectivity and reproducibility to assessments of study quality, numerous evidence-rating systems (ERSs) have been developed. A 2002 Evidence Report Summary from the US Agency for Healthcare Research and Quality (AHRQ) found that among 1,602 reviewed publications, many ERSs were used for different purposes – 49 evaluated randomized controlled trials (RCTs), 19 evaluated observational studies, and 20 evaluated systematic reviews [ 12 ]. Several of these systems are often used in contemporary meta-analyses – for prospective trials, these include systems developed by AHRQ, Cochrane, and Jadad and colleagues. Underlying their familiarity, however, are significant differences in methodology. For example, the AHRQ and Cochrane tools provide non-numeric assessments to independent domains related to risks of bias, while the Jadad score generates a quantitative score reflecting methodological quality [ 8 , 13 ]. For observational studies, prevalent systems include the Downs-Black and Newcastle-Ottawa systems [ 14 – 16 ]. These also differ in that the Downs-Black system may be applied to both randomized and non-randomized studies and provide a semi-quantitative assessment, while the Newcastle-Ottawa system is applied only to those non-randomized trials as either a checklist or quantitative scale [ 16 ]. Additionally, tools that evaluate the quality of the overall body of evidence have been created to help summarize main findings, and their use is also recommended by PRISMA and Cochrane [ 8 , 17 ]. These assessments are similar to epidemiologic determinations of causality (e.g., Bradford Hill criteria), which consider elements of quantity, quality and consistency of evidence for each outcome across studies [ 18 ]. Although there is much overlap between systems that assess the risk of bias in individual studies in some domains, none of the ERS address all the necessary components [ 13 ]. Certainly, different study designs and assessment purposes dictate the need for different ERSs, and none are considered to be more appropriate [ 4 , 14 , 21 ]. The resulting variability among the numerous systems can be problematic for those conducting meta-analyses [ 14 ]. Further knowledge regarding trends in the use of specific ERSs in different scenarios may help identify commonly used systems. Because the generation of meta-analyses involves assessment of the quality of individual studies and the overall body of evidence, familiarity with commonly used ERSs may help both authors and users of meta-analyses to assess the level of confidence and application of results of meta-analyses. Therefore, there is abundance of published drug studies, and coexist numerous ERSs in use, with lack of consensus on best practices. Aim of the study The objective of this study is to describe the use of ERSs (including to evaluate de body of literature) in meta-analyses of drug therapy published in high-impact medical journals. Ethics Approval Not applicable. This research does not contain any studies with human participants conducted by any of the authors. Methods We performed a review of the use of ERSs in meta-analyses of drug effects published in high-impact medical journals. The highest-impact journals in the category of Medicine, General and Internal for the years 2012 through 2016 were determined using the Clarivate Analytics Journal Citations Report (JCR) database [ 22 ]. The journals ranked within the top five positions for this category in any year from 2012 through 2016 were selected as source journals for this analysis, similar to the methods utilized in other evaluations of reporting quality [ 23 ]. We searched for meta-analyses of drug effects published in these journals in Medline / PubMed using Medical Subject Heading (MeSH). The subheadings that were descriptive of drug effects included therapeutic use, adverse effects, drug therapy, and administration and dosage . Meta-analyses were targeted for searching the term “meta-analysis” in the title field or articles classified with the National Library of Medicine publication type meta-analysis. The time period 2012 to 2016 was selected to allow sufficient time for indexation with MeSH terms between the time of publication and the time of searching, because delays in indexing have previously been documented [ 24 ]. In the online supplementary appendix, we report the key terms (PRISMA flowchart as Fig. 1 ) outlining the search strategy. This research did not require further review by a local Institutional Review Board as the research does not meet the definition of human subject research. Methods of the analysis and inclusion criteria were specified in advance and documented in an internal non-registered protocol (available upon request). Medline data was downloaded from PubMed and entered into a standardized data collection form. The full-texts of all the articles was manually reviewed and categorized for inclusion criteria if they were determined to be meta-analyses (quantitative data), evaluated if the subject treated was drug therapy. The methods section, and when indicated, supplemental material, of included articles were reviewed and data was collected regarding the ERS that was used (if any) to rate the quality of evidence at the levels of study and the body of literature, methods used for assessing risk of bias of individual studies, and whether authors incorporated modifications to an ERS. A global assessment of risk of bias that may affect evidence was done. Descriptive statistics were calculated using SPSS to report frequencies. Principal summary measures may differ from that used in some of the included studies and may not be given for all studies. Exploratory analyses describing ERS use, stratified by journal, was also performed. The between-study variability (heterogeneity or inconsistency) of results may influence the decision of whether to combine results in a meta-analysis, so there were no planned methods for combining results. None sensitivity nor subgroup additional analyses was done. Results Review of the JCR database revealed that the medical journals ranked in the top 5 positions at any given time from 2012 through 2016 were BMJ, Lancet, Annals of Internal Medicine, JAMA , and PLoS Medicine . The targeted PubMed search of these journals yielded 309 results. Manual review excluded 111 publications that did not evaluate a drug intervention or exposure (n = 86), were not quantitative meta-analyses (n = 20), were not meta-analyses of trials (n = 3), or were retracted (n = 2), leaving 198 publications in the full analysis set from BMJ (n = 84), Lancet (n = 41), Annals of Internal Medicine (n = 39), JAMA (n = 23), and PLoS Medicine (n = 11). Overall, 86.4% (n = 171) of meta-analyses of drug effects published in high-impact medical journals utilized an ERS to evaluate interventional or observational studies. Fourteen unique ERSs were identified which included author-defined systems: these were utilized to evaluate interventional and observational studies in 12 and 2 meta-analyses, respectively. Among the meta-analyses reporting use of an ERS to rate interventional trials (n = 171), the most frequently used ERS was the Cochrane Risk of Bias Tool (80.7%; n = 138), followed by a variety of other systems (Table 1 ). Meta-analyses that reported ERSs for rating observational studies (n = 24) most frequently used the Newcastle-Ottawa scale (66.7%; n = 16; Table 2 ). Table 1 Use of ERSs among meta-analyses rating the quality of evidence of interventional trials (n = 171). ERS N (%) Cochrane Risk of Bias Tool 138 (80.7) Author-defined 16 (9.4) Jadad 11 (6.4) AHRQ 4 (2.3) USPSTF 2 (1.2) Abbreviations: AHRQ = Agency for Healthcare Research and Quality; ERS = Evidence-Rating System; GRADE = Grading of Recommendations Assessment, Development, and Evaluation. Table 2 Use of ERSs among meta-analyses rating the quality of evidence of observational studies (n = 24). ERS N (%) Newcastle-Ottawa 16 (66.7) Downs-Black 3 (12.5) Author-defined 3 (12.5) RoBANS 1 (4.2) ROBINS-I 1 (4.2) Abbreviations: ERS = evidence-rating system; RoBANS = Risk of Bias Assessment Tool for Nonrandomized Studies; ROBINS-I = Risk of Bias in Non-randomized Studies of Interventions. An ERS evaluating the body of literature was used by 38 meta-analyses, with the most commonly used system being the GRADE methodology (78.9%; n = 30; Table 3 ). Systems developed by AHRQ and the USPSTF were also utilized (n = 7 and n = 1, respectively). Table 3 Use of ERSs among meta-analyses rating the quality of evidence of a body of literature (n = 38). ERS N (%) GRADE 30 (78.94) AHRQ 7 (18.42) USPSTF 1 (2.63) Abbreviations: AHRQ = Agency for Healthcare Research and Quality; ERS = evidence-rating system; GRADE = Grading of Recommendations Assessment, Development, and Evaluation; USPSTF = US Preventative Services Task Force. Four meta-analyses incorporated a secondary ERS to evaluate interventional trials, including systems by Jadad (n = 2), AHRQ (n = 1), and McHarm (n = 1). No meta-analyses evaluating observational studies utilized a secondary ERS. Not all meta-analyses incorporated present results with confidence intervals and measures of consistency. Modifications of ERS were made in 11 meta-analyses. 8 meta-analyses were interventional studies; The ERS modified included the Cochrane Risk of Bias Tool (n = 7) and Jadad systems (n = 1). Observational ERSs were modified in 3 meta-analyses, including the Newcastle-Ottawa scale (n = 2) and ROBINS-I (n = 1). Most meta-analyses present results of any assessment of risk of bias across studies. No results from exploratory subgroup nor sensitivity analyses was done, bearing in mind the potential for multiple analyses to mislead. ERS use across journals indicated variations in whether any ERS was used and which ERS was used (Table 4 ; Fig. 2 ). Table 4 Meta-analyses using any ERS at the study level stratified by journal (n = 198). Journal N TOTAL N with ERS % with ERS Annals of Internal Medicine 39 39 100 BMJ 84 78 92.9 JAMA 23 23 100 Lancet 41 22 53.6 PLoS Medicine 11 9 81.8 TOTAL 198 171 86.4 Discusion This descriptive review found that a majority of meta-analyses of drug effects published in high-impact medical journals utilized a variety of ERSs, most commonly the Cochrane Risk of Bias Tool for interventional trials and the Newcastle-Ottawa scale for observational studies, and that systems rating the body of evidence are less commonly utilized. Notably, the meta-analysis reporting use of AHRQ methods referenced the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews; this document provides information about the process of evaluating risk of bias, but does not propose its own ERS per se, and did not explicitly state the actual ERS used to assess the risk of bias [ 14 ]. The variety of unique ERSs identified confirms there is no well-accepted gold standard system, consistent with statements to this effect by respected organizations commenting on this topic [ 14 ]. While most meta-analyses included an ERS to rate quality of individual studies, only 19.1% incorporated an ERS evaluating the body of literature. In addition, inter-journal practices varied, as noted by the disparity in the proportion of meta-analyses using any ERS, which was as low as 53.7% in Lancet, compared to 100% in Annals of Internal Medicine and JAMA . These findings indicate that adherence to the recommendations for the reporting of several aspects of meta-analytic designs might be suboptimal. For example, the PRISMA statement and Cochrane Handbook recommend that authors specify the assessments of risk of bias for each study, across studies, and at the outcome level. Similarly, MOOSE guidelines suggest less specifically that risk for bias should be discussed [ 6 – 8 ]. Our findings (88.4% of meta-analyses in high-impact medical journals) show variations in use across journals; these findings suggest standards is not consistently meet. Systematic reviewers should have flexibility to choose the tool that best matches their study and literature base. Other notable findings were that more than 10 meta-analyses used modifications of an ERS. This practice may not be optimal, as authors of the GRADE system have claimed such modifications undermining the goal of promoting a single system in which all readers can be familiar with [ 25 ]. Indeed, recent commentaries have questioned the rigor in using modified versions of ERS in meta-analyses, specifically the Newcastle-Ottawa scale [ 26 ]. Readers of meta-analyses should therefore be observant for such modifications and consider their effects on estimates of study quality. Additionally, author-defined systems were utilized to evaluate interventional and observational studies in 12 and 2 meta-analyses, respectively. One of these relied on methods in rating study quality based on those from a previous publication, which was not an ERS per se, but evaluated how specific elements of study design biased the estimates of an intervention’s effect [ 27 , 28 ]. In such instances, authors should be explicit in the methods used to assess the risk for bias to clearly explain the process to readers. Our findings suggest potential improvements in standards for publication of meta-analyses; it could be beneficial for journals to consider more explicit statements regarding the amount, and type of detail, regarding risk of bias assessments to improve reporting. For example, guidances from journals included in this analysis refer authors to PRISMA, MOOSE, and other relevant guidances in reporting of systematic reviews [ 29 – 33 ]. In addition to the commonly used system developed by GRADE Working Group, others have been developed by the AHRQ, the US Preventive Services Task Force and the Oxford Centre for Evidence-Based Medicine [ 18 – 20 ]. These systems evaluate different domains and incorporate their own processes of translating assessments of a body of literature into clinical recommendations, such as those provided by clinical practice guidelines. However, an assessment of 40 of these systems by the 2002 AHRQ report determined that they are less uniform than those used for assessing individual studies, which may complicate the selection of an appropriate system to rate a body of evidence [ 12 ]. Further instructions from journals may help address limitations identified in this study. For example, while certain ERSs will undoubtedly be preferred in different scenarios, journals may consider establishing a preferred ERS for meta-analyses characteristic of the journal’s scope, such as establishing a preferred ERS for interventional and another for observational studies. This may allow for more meaningful comparisons of estimates between meta-analyses published in the same journal, when for example, one meta-analysis evaluates efficacy, and another evaluates safety of the same drug. This task would be more complex if different meta-analyses used various systems to rate the quality of evidence. Journal-specific preference of particular ERSs may cultivate more awareness and familiarity among readers and facilitate application of evidence in practice. Our analysis has several limitations. Firstly, we only considered a narrow scope of journals, the top five journals in Medicine, General & Internal; this category covers resources on medical specialties such as general medicine, internal medicine, clinical physiology, pain management, military and hospital medicine, whereas, Pharmacology and Toxicology category are not included and covers resources on the discovery and testing of bioactive substances, including animal research, clinical experience, delivery systems, and dispensing of drugs. This category also includes resources on the biochemistry, metabolism, and toxic or adverse effects of drugs. These findings may not be representative of non-medical journals or medical journals with lower impact factors or from specialized practice areas, and Embase search should also have been made. Secondly, the editorial and peer review standards of higher-impact journals in this analysis may have produced findings more reflective of the “best practice” in meta-analysis production. Among these journals, a large number of meta-analyses originated in BMJ, and the requirements of this journal may disproportionately influence our overall findings. Thirdly, we only evaluated meta-analyses of drug effects, and our conclusions are not generalizable to meta-analyses of other interventions. Combined together, these limitations indicate there may be a greater variety of ERS utilization outside journals and interventions considered in this review. Future research, conducting an overview of systematic reviews would be needed. The 40 fields of the protocol should be prospectively registered on PROSPERO, an international prospective database of registered systematic reviews, developed and managed by the Centre for Reviews and Dissemination (CRD) at the University of York. Our review represents, to the authors’ knowledge, the first description of the frequency of use of ERSs in the medical literature. Further research should address these findings to develop a general framework for best practices in this field. Conclusions Most meta-analyses of drug effects in high-impact medical journals evaluated individual studies with an ERS, most commonly the Cochrane Risk of Bias Tool, while use of ERSs to evaluate the body of literature was less frequent. The evaluation and application of findings of meta-analyses may be facilitated by familiarity of authors and users of meta-analyses with commonly used ERSs, as well as more specific guidance for journal submissions. Abbreviations AHRQ Agency for Healthcare Research and Quality BMJ British Medical Journal ERS Evidence-Rating System GRADE Grading of Recommendations Assessment, Development, and Evaluation JAMA Journal of the American Medical Association MeSH Medical Subject Heading MOOSE Meta-analysis Of Observational Studies in Epidemiology PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses RCT Randomized Controlled Trial RoBANS Risk of Bias Assessment Tool for Nonrandomized Studies ROBINS-I Risk of Bias in Nonrandomized Studies of Interventions USPSTF US Preventative Services Task Force Declarations - Ethics approval and consent to participate Not applicable - Consent to publish Not applicable - Availability of data and materials All data generated or analysed during this study are included in this published article [and its supplementary information files]. - Competing interests The authors declare that they have no competing interests. - Funding This study was not funded, nor received any grant. - Authors' Contributions AFPS conceived the study and designed the paper. AFPS, JMVH, MAM and MAAB drafted the manuscript, analyzed and interpreted the data. MAAL and MTSG helped with the manuscript. All authors were involved in the development of the concept of the article and drafting the article. All authors read and approved the final manuscript. - Acknowledgements Joshua Ryan Comra (BYU pre-med student), Aric Aiton (UWindsor pre-med student), Sandra Peña (PIMA Medical Institute pre-med student). References Agency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence. Agency for Healthcare Research and Quality website. https://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf . Accessed Sep 13, 2019. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6. Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29–37. Murad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA. 2014;312(2):171–9. Anonymous. Trends, charts and maps. Clinicaltrials.gov website. https://clinicaltrials.gov/ct2/resources/trends#TypesOfRegisteredStudies . Updated November 2018. Accessed Sep 13, 2019. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283(15):2008–12. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. Higgins PT, Gree S. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Collaboration website. http://handbook-5-1.cochrane.org/ . Updated March 2011. Accessed Sep 13, 2019. Anonymous. About us. Cochrane Collaboration website. https://www.cochrane.org/about-us . Accessed Sep 13, 2019. Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. West S, King V, Carey TS, et al. Systems to rate the strength of scientific evidence: Full Report. Agency for Healthcare Research and Quality website. https://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf . Accessed Sep 13, 2019. Viswanathan M, Berkman ND, Dryden DM, Hartling L. Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank. In: AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013. Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. Agency for Healthcare Research and Quality website. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf . Published January 2014. Accessed Sep 13, 2019. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377–84. Wells GA, Shea B, O'Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. website. http://www.ohri.ca/programs/clinical_epidemiology/nos_manual.pdf . Accessed Sep 13, 2019. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700. Agency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence: Summary. Agency for Healthcare Research and Quality website. https://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf . Accessed Sep 13, 2019. Harris RP, Helfand M, Woolf SH, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(3 Suppl):21–35. Anonymous. OCEBM Levels of Evidence. Oxford website. https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ . Accessed Sep 13, 2019. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42–6. Anonymous. Journal Citation Reports. Clarivate Analytics website. https://jcr.incites.thomsonreuters.com/ . Accessed August 31, 2018. Hays M, Andrews M, Wilson R, Callender D, O'Malley PG, Douglas K. Reporting quality of randomised controlled trial abstracts among high-impact general medical journals: a review and analysis. BMJ Open. 2016;6:e011082. Rodriguez R. Comparison of indexing times among articles from medical, nursing, and pharmacy journals. Am J Health Syst Pharm. 2016;73(8):569–75. Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64:395–400. Wang L, Gu C. How Would You Evaluate the Quality of Uncontrolled Studies in a Meta-Analysis? Crit Care Med. 2018;46(8):e822–3. Savovic J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157(6):429–38. Stagg HR, Zenner D, Harris RJ, Munoz L, Lipman MC, Abubakar I. Treatment of latent tuberculosis infection: a network meta-analysis. Ann Intern Med. 2014;161:419–28. Anonymous. Information for authors. Annals of Internal Medicine website. https://annals.org/aim/pages/authors . Accessed Sep 13, 2019. Anonymous. Instructions for Authors. Journal of the American Medical Association website. https://jamanetwork.com/journals/jama/pages/instructions-for-authors . Accessed Sep 13, 2019. Anonymous. Resources for authors. The British Medical Journal website. https://www.bmj.com/about-bmj/resources-authors/article-types . Accessed Sep 13, 2019. Anonymous. Submission Guidelines. PLoS Medicine website website. https://journals.plos.org/plosmedicine/s/submission-guidelines . Accessed Sep 13, 2019. Anonymous. Systematic reviews and meta-analyses in The Lancet: formatting guidelines. The Lancet website. https://www.thelancet.com/pb/assets/raw/Lancet/authors/metaguidelines.pdf . Accessed Sep 13, 2019. Supplementary Files PRISMA2009checklist.doc AppendixSupplementarySearchstrategyPrismaflowchart.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-42029","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research article","associatedPublications":[],"authors":[{"id":948946,"identity":"485e193c-6eef-4633-aa68-f953c523c988","order_by":0,"name":"Alberto Frutos Pérez-Surio","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABC0lEQVRIie2RMUsDMRiG33BQl5SsJ4L9CykHupzmr+QopMsNBaF0rIuOXQPej9DF+coH3iItbjeedL1BcHHTXHURidwomAdCCOThfZMPCAT+JOtmvwmAlRj0UkgCGjhcAn2V6FORZV9FXNLgdfiWqmRLDfE5RqfX1dNuhlT5lLik6IZrk93XRhLfYFw85heJhcmWPqfcUsQ16ZMakoZXYBa5OeIg7TNGLqVTVLKqXjpFWdF2yru3mPxS2C3yfUpm4+mDU0rmKzZ2CiuMyWydz9bFJp7YuI0SLifetxzXFKFNUyVW1V3TztMzK6bPO7449xZzX/b96BZ3k5JewQ3mZ/5B88v9QCAQ+Id8ABCeVp3Gq/xAAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-9962-3387","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Alberto","middleName":"Frutos","lastName":"Pérez-Surio","suffix":""},{"id":948947,"identity":"905e887f-46ec-4849-b6f2-6937c7f20126","order_by":1,"name":"José Manuel Vinuesa Hernando","email":"","orcid":"","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"José","middleName":"Manuel Vinuesa","lastName":"Hernando","suffix":""},{"id":948948,"identity":"31726c3d-deab-4dfe-ac0b-3d813fed1087","order_by":2,"name":"Mercedes Arene Mendoza","email":"","orcid":"","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Mercedes","middleName":"Arene","lastName":"Mendoza","suffix":""},{"id":948949,"identity":"aa7362a9-0c89-4281-80ca-21bbe79a9a67","order_by":3,"name":"María Ángeles Allende Bandrés","email":"","orcid":"","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"María","middleName":"Ángeles Allende","lastName":"Bandrés","suffix":""},{"id":948950,"identity":"291b53cf-0bab-43a5-b433-5ed8c393da74","order_by":4,"name":"María Aránzazu Alcácera López","email":"","orcid":"","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"María","middleName":"Aránzazu Alcácera","lastName":"López","suffix":""},{"id":948951,"identity":"3f68bfb2-9959-43bc-963e-e2d440865ce4","order_by":5,"name":"Tránsito Salvador Gómez","email":"","orcid":"","institution":"Hospital Clinico Universitario Lozano Blesa","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Tránsito","middleName":"Salvador","lastName":"Gómez","suffix":""}],"badges":[],"createdAt":"2020-07-13 18:45:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-42029/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-42029/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":1647823,"identity":"e4905e59-75e4-4758-b3d5-18e8d7dc12bf","added_by":"auto","created_at":"2020-07-22 14:18:33","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":32391,"visible":true,"origin":"","legend":"PRISMA flow diagram","description":"","filename":"Fig1.JPG","url":"https://assets-eu.researchsquare.com/files/rs-42029/v1/Fig1.JPG"},{"id":1647824,"identity":"9b061806-6211-412c-a2a4-7bdc01b31eab","added_by":"auto","created_at":"2020-07-22 14:18:33","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":31383,"visible":true,"origin":"","legend":"ERS use across journals for interventional and observational studies.","description":"","filename":"Fig2.JPG","url":"https://assets-eu.researchsquare.com/files/rs-42029/v1/Fig2.JPG"},{"id":13554837,"identity":"d9a4f405-7a58-4f63-a3e1-7af3c5f3e6d2","added_by":"auto","created_at":"2021-09-17 02:42:44","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":279570,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-42029/v1/7dab2cf9-8732-4a42-92a8-0f1e2b182f17.pdf"},{"id":1647826,"identity":"143b5207-57a3-4d60-bec4-374f07bdc9b6","added_by":"auto","created_at":"2020-07-22 14:18:34","extension":"doc","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":66048,"visible":true,"origin":"","legend":"","description":"","filename":"PRISMA2009checklist.doc","url":"https://assets-eu.researchsquare.com/files/rs-42029/v1/PRISMA2009checklist.doc"},{"id":1647827,"identity":"3b634b6e-cd37-445c-a479-c726bef94f69","added_by":"auto","created_at":"2020-07-22 14:18:34","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":55944,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixSupplementarySearchstrategyPrismaflowchart.docx","url":"https://assets-eu.researchsquare.com/files/rs-42029/v1/AppendixSupplementarySearchstrategyPrismaflowchart.docx"}],"financialInterests":"","formattedTitle":"Analysis of Evidence-Rating Systems Used in Meta-Analyses of Pharmacotherapy","fulltext":[{"header":"Impacts On Practice","content":" \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eExperience is crucial for the evaluation and application of findings of meta-analyses. Meta-analyses evaluating individual studies with an Evidence-Rating Systems were frequent compared to those evaluating the body of literature, and modified versions of Evidence-Rating Systems, although used, may not be optimal.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIt could be beneficial for journals to be more explicit in terms of detail and risk of bias, so further work is required to develop a general framework of best practices in Evidence-Rating Systems.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e "},{"header":"Background","content":" \u003cp\u003eEvidence-based clinical decision-making relies on well-designed research that utilizes rigorous methodology [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Ranking highly among the various clinical research designs are meta-analyses, which have a goal of quantitatively integrating findings of individual studies [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. The results of meta-analyses may be more useful than those of the individual studies they contain; the integrated result may include a more representative population, so it possesses greater statistical power. Therefore, this provides more confidence in decision-making through more precise results and offer a more efficient review of a full body of literature [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThese benefits of meta-analyses are particularly relevant in the assessment of drug effects, which are part of a voluminous bibliography. A cursory Medline search of publications indexed in 2016 revealed approximately 10,500 results for publications on clinical trials related to drug therapy alone. Additionally, the Clinicaltrials.gov website lists 138,002 clinical trials on drugs or biologics registered as of May 09, 2019 [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn spite of these benefits of meta-analyses, their results are only as reliable as those of the individual studies they include. For this reason, meta-analyses should evaluate the risk of bias of the individual studies they include. This practice is recommended by various sources, including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement, the Meta-analysis Of Observational Studies in Epidemiology: a proposal for reporting (MOOSE) statement, and the Cochrane Handbook on Systematic Reviews of Interventions [\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. These sources are designed to improve reporting and utility of meta-analyses, and to delineate the preferred methodology for meta-analyses produced by some of the world\u0026rsquo;s most recognized and authoritative groups of authors [\u003cspan additionalcitationids=\"CR7 CR8\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA study\u0026rsquo;s risk for bias may be assessed through evaluation of elements of research design (e.g., randomization and blinding), and other design flaws that lead to increased risk for selection, performance, information, and other biases [\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Nonetheless, the assessment of study quality may present difficulties because the concept of \u0026ldquo;quality\u0026rdquo; is not well defined, not surprisingly, some assessments of markers of study quality (e.g., blinding) are judged subjectively and have poor inter-rater reliability [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn attempts to provide greater objectivity and reproducibility to assessments of study quality, numerous evidence-rating systems (ERSs) have been developed. A 2002 Evidence Report Summary from the US Agency for Healthcare Research and Quality (AHRQ) found that among 1,602 reviewed publications, many ERSs were used for different purposes \u0026ndash; 49 evaluated randomized controlled trials (RCTs), 19 evaluated observational studies, and 20 evaluated systematic reviews [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eSeveral of these systems are often used in contemporary meta-analyses \u0026ndash; for prospective trials, these include systems developed by AHRQ, Cochrane, and Jadad and colleagues. Underlying their familiarity, however, are significant differences in methodology. For example, the AHRQ and Cochrane tools provide non-numeric assessments to independent domains related to risks of bias, while the Jadad score generates a quantitative score reflecting methodological quality [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. For observational studies, prevalent systems include the Downs-Black and Newcastle-Ottawa systems [\u003cspan additionalcitationids=\"CR15\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. These also differ in that the Downs-Black system may be applied to both randomized and non-randomized studies and provide a semi-quantitative assessment, while the Newcastle-Ottawa system is applied only to those non-randomized trials as either a checklist or quantitative scale [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Additionally, tools that evaluate the quality of the overall body of evidence have been created to help summarize main findings, and their use is also recommended by PRISMA and Cochrane [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. These assessments are similar to epidemiologic determinations of causality (e.g., Bradford Hill criteria), which consider elements of quantity, quality and consistency of evidence for each outcome across studies [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAlthough there is much overlap between systems that assess the risk of bias in individual studies in some domains, none of the ERS address all the necessary components [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Certainly, different study designs and assessment purposes dictate the need for different ERSs, and none are considered to be more appropriate [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. The resulting variability among the numerous systems can be problematic for those conducting meta-analyses [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Further knowledge regarding trends in the use of specific ERSs in different scenarios may help identify commonly used systems.\u003c/p\u003e \u003cp\u003eBecause the generation of meta-analyses involves assessment of the quality of individual studies and the overall body of evidence, familiarity with commonly used ERSs may help both authors and users of meta-analyses to assess the level of confidence and application of results of meta-analyses. Therefore, there is abundance of published drug studies, and coexist numerous ERSs in use, with lack of consensus on best practices.\u003c/p\u003e \u003cp\u003eAim of the study\u003c/p\u003e \u003cp\u003eThe objective of this study is to describe the use of ERSs (including to evaluate de body of literature) in meta-analyses of drug therapy published in high-impact medical journals.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eEthics Approval\u003c/strong\u003e \u003cp\u003eNot applicable. This research does not contain any studies with human participants conducted by any of the authors.\u003c/p\u003e \u003c/p\u003e "},{"header":"Methods","content":" \u003cp\u003eWe performed a review of the use of ERSs in meta-analyses of drug effects published in high-impact medical journals. The highest-impact journals in the category of Medicine, General and Internal for the years 2012 through 2016 were determined using the Clarivate Analytics Journal Citations Report (JCR) database [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. The journals ranked within the top five positions for this category in any year from 2012 through 2016 were selected as source journals for this analysis, similar to the methods utilized in other evaluations of reporting quality [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. We searched for meta-analyses of drug effects published in these journals in Medline / PubMed using Medical Subject Heading (MeSH). The subheadings that were descriptive of drug effects included \u003cem\u003etherapeutic use, adverse effects, drug therapy, and administration and dosage\u003c/em\u003e. Meta-analyses were targeted for searching the term \u0026ldquo;meta-analysis\u0026rdquo; in the title field or articles classified with the National Library of Medicine publication type meta-analysis. The time period 2012 to 2016 was selected to allow sufficient time for indexation with MeSH terms between the time of publication and the time of searching, because delays in indexing have previously been documented [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. In the online supplementary appendix, we report the key terms (PRISMA flowchart as Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) outlining the search strategy. This research did not require further review by a local Institutional Review Board as the research does not meet the definition of human subject research. Methods of the analysis and inclusion criteria were specified in advance and documented in an internal non-registered protocol (available upon request).\u003c/p\u003e \u003cp\u003eMedline data was downloaded from PubMed and entered into a standardized data collection form. The full-texts of all the articles was manually reviewed and categorized for inclusion criteria if they were determined to be meta-analyses (quantitative data), evaluated if the subject treated was drug therapy. The \u003cspan refid=\"Sec3\" class=\"InternalRef\"\u003emethods\u003c/span\u003e section, and when indicated, supplemental material, of included articles were reviewed and data was collected regarding the ERS that was used (if any) to rate the quality of evidence at the levels of study and the body of literature, methods used for assessing risk of bias of individual studies, and whether authors incorporated modifications to an ERS. A global assessment of risk of bias that may affect evidence was done. Descriptive statistics were calculated using SPSS to report frequencies. Principal summary measures may differ from that used in some of the included studies and may not be given for all studies. Exploratory analyses describing ERS use, stratified by journal, was also performed. The between-study variability (heterogeneity or inconsistency) of results may influence the decision of whether to combine results in a meta-analysis, so there were no planned methods for combining results. None sensitivity nor subgroup additional analyses was done.\u003c/p\u003e "},{"header":"Results","content":" \u003cp\u003eReview of the JCR database revealed that the medical journals ranked in the top 5 positions at any given time from 2012 through 2016 were \u003cem\u003eBMJ, Lancet, Annals of Internal Medicine, JAMA\u003c/em\u003e, and \u003cem\u003ePLoS Medicine\u003c/em\u003e. The targeted PubMed search of these journals yielded 309 results. Manual review excluded 111 publications that did not evaluate a drug intervention or exposure (n\u0026thinsp;=\u0026thinsp;86), were not quantitative meta-analyses (n\u0026thinsp;=\u0026thinsp;20), were not meta-analyses of trials (n\u0026thinsp;=\u0026thinsp;3), or were retracted (n\u0026thinsp;=\u0026thinsp;2), leaving 198 publications in the full analysis set from \u003cem\u003eBMJ\u003c/em\u003e (n\u0026thinsp;=\u0026thinsp;84), \u003cem\u003eLancet\u003c/em\u003e (n\u0026thinsp;=\u0026thinsp;41), \u003cem\u003eAnnals of Internal Medicine\u003c/em\u003e (n\u0026thinsp;=\u0026thinsp;39), JAMA (n\u0026thinsp;=\u0026thinsp;23), and \u003cem\u003ePLoS Medicine\u003c/em\u003e (n\u0026thinsp;=\u0026thinsp;11).\u003c/p\u003e \u003cp\u003eOverall, 86.4% (n\u0026thinsp;=\u0026thinsp;171) of meta-analyses of drug effects published in high-impact medical journals utilized an ERS to evaluate interventional or observational studies. Fourteen unique ERSs were identified which included author-defined systems: these were utilized to evaluate interventional and observational studies in 12 and 2 meta-analyses, respectively.\u003c/p\u003e \u003cp\u003eAmong the meta-analyses reporting use of an ERS to rate interventional trials (n\u0026thinsp;=\u0026thinsp;171), the most frequently used ERS was the Cochrane Risk of Bias Tool (80.7%; n\u0026thinsp;=\u0026thinsp;138), followed by a variety of other systems (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Meta-analyses that reported ERSs for rating observational studies (n\u0026thinsp;=\u0026thinsp;24) most frequently used the Newcastle-Ottawa scale (66.7%; n\u0026thinsp;=\u0026thinsp;16; Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eUse of ERSs among meta-analyses rating the quality of evidence of interventional trials (n\u0026thinsp;=\u0026thinsp;171).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eERS\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCochrane Risk of Bias Tool\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e138 (80.7)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAuthor-defined\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e16 (9.4)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJadad\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e11 (6.4)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAHRQ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4 (2.3)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUSPSTF\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2 (1.2)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003eAbbreviations: AHRQ\u0026thinsp;=\u0026thinsp;Agency for Healthcare Research and Quality; ERS\u0026thinsp;=\u0026thinsp;Evidence-Rating System; GRADE\u0026thinsp;=\u0026thinsp;Grading of Recommendations Assessment, Development, and Evaluation.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eUse of ERSs among meta-analyses rating the quality of evidence of observational studies (n\u0026thinsp;=\u0026thinsp;24).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eERS\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNewcastle-Ottawa\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e16 (66.7)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDowns-Black\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3 (12.5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAuthor-defined\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3 (12.5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRoBANS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1 (4.2)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eROBINS-I\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1 (4.2)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003eAbbreviations: ERS\u0026thinsp;=\u0026thinsp;evidence-rating system; RoBANS\u0026thinsp;=\u0026thinsp;Risk of Bias Assessment Tool for Nonrandomized Studies; ROBINS-I\u0026thinsp;=\u0026thinsp;Risk of Bias in Non-randomized Studies of Interventions.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAn ERS evaluating the body of literature was used by 38 meta-analyses, with the most commonly used system being the GRADE methodology (78.9%; n\u0026thinsp;=\u0026thinsp;30; Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Systems developed by AHRQ and the USPSTF were also utilized (n\u0026thinsp;=\u0026thinsp;7 and n\u0026thinsp;=\u0026thinsp;1, respectively).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eUse of ERSs among meta-analyses rating the quality of evidence of a body of literature (n\u0026thinsp;=\u0026thinsp;38).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eERS\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGRADE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e30 (78.94)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAHRQ\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7 (18.42)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUSPSTF\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1 (2.63)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003eAbbreviations: AHRQ\u0026thinsp;=\u0026thinsp;Agency for Healthcare Research and Quality; ERS\u0026thinsp;=\u0026thinsp;evidence-rating system; GRADE\u0026thinsp;=\u0026thinsp;Grading of Recommendations Assessment, Development, and Evaluation; USPSTF\u0026thinsp;=\u0026thinsp;US Preventative Services Task Force.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFour meta-analyses incorporated a secondary ERS to evaluate interventional trials, including systems by Jadad (n\u0026thinsp;=\u0026thinsp;2), AHRQ (n\u0026thinsp;=\u0026thinsp;1), and McHarm (n\u0026thinsp;=\u0026thinsp;1). No meta-analyses evaluating observational studies utilized a secondary ERS. Not all meta-analyses incorporated present results with confidence intervals and measures of consistency.\u003c/p\u003e \u003cp\u003eModifications of ERS were made in 11 meta-analyses. 8 meta-analyses were interventional studies; The ERS modified included the Cochrane Risk of Bias Tool (n\u0026thinsp;=\u0026thinsp;7) and Jadad systems (n\u0026thinsp;=\u0026thinsp;1). Observational ERSs were modified in 3 meta-analyses, including the Newcastle-Ottawa scale (n\u0026thinsp;=\u0026thinsp;2) and ROBINS-I (n\u0026thinsp;=\u0026thinsp;1). Most meta-analyses present results of any assessment of risk of bias across studies. No results from exploratory subgroup nor sensitivity analyses was done, bearing in mind the potential for multiple analyses to mislead.\u003c/p\u003e \u003cp\u003eERS use across journals indicated variations in whether any ERS was used and which ERS was used (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e; Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eMeta-analyses using any ERS at the study level stratified by journal (n\u0026thinsp;=\u0026thinsp;198).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJournal\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN TOTAL\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eN with ERS\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e% with ERS\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eAnnals of Internal Medicine\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e100\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eBMJ\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e92.9\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eJAMA\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e100\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eLancet\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e53.6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003ePLoS Medicine\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e81.8\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eTOTAL\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e198\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e171\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e86.4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e "},{"header":"Discusion","content":" \u003cp\u003eThis descriptive review found that a majority of meta-analyses of drug effects published in high-impact medical journals utilized a variety of ERSs, most commonly the Cochrane Risk of Bias Tool for interventional trials and the Newcastle-Ottawa scale for observational studies, and that systems rating the body of evidence are less commonly utilized.\u003c/p\u003e \u003cp\u003eNotably, the meta-analysis reporting use of AHRQ methods referenced the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews; this document provides information about the process of evaluating risk of bias, but does not propose its own ERS per se, and did not explicitly state the actual ERS used to assess the risk of bias [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. The variety of unique ERSs identified confirms there is no well-accepted gold standard system, consistent with statements to this effect by respected organizations commenting on this topic [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. While most meta-analyses included an ERS to rate quality of individual studies, only 19.1% incorporated an ERS evaluating the body of literature. In addition, inter-journal practices varied, as noted by the disparity in the proportion of meta-analyses using any ERS, which was as low as 53.7% in Lancet, compared to 100% in \u003cem\u003eAnnals of Internal Medicine\u003c/em\u003e and \u003cem\u003eJAMA\u003c/em\u003e.\u003c/p\u003e \u003cp\u003eThese findings indicate that adherence to the recommendations for the reporting of several aspects of meta-analytic designs might be suboptimal. For example, the PRISMA statement and Cochrane Handbook recommend that authors specify the assessments of risk of bias for each study, across studies, and at the outcome level. Similarly, MOOSE guidelines suggest less specifically that risk for bias should be discussed [\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Our findings (88.4% of meta-analyses in high-impact medical journals) show variations in use across journals; these findings suggest standards is not consistently meet. Systematic reviewers should have flexibility to choose the tool that best matches their study and literature base.\u003c/p\u003e \u003cp\u003eOther notable findings were that more than 10 meta-analyses used modifications of an ERS. This practice may not be optimal, as authors of the GRADE system have claimed such modifications undermining the goal of promoting a single system in which all readers can be familiar with [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Indeed, recent commentaries have questioned the rigor in using modified versions of ERS in meta-analyses, specifically the Newcastle-Ottawa scale [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Readers of meta-analyses should therefore be observant for such modifications and consider their effects on estimates of study quality. Additionally, author-defined systems were utilized to evaluate interventional and observational studies in 12 and 2 meta-analyses, respectively. One of these relied on methods in rating study quality based on those from a previous publication, which was not an ERS per se, but evaluated how specific elements of study design biased the estimates of an intervention\u0026rsquo;s effect [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. In such instances, authors should be explicit in the methods used to assess the risk for bias to clearly explain the process to readers.\u003c/p\u003e \u003cp\u003eOur findings suggest potential improvements in standards for publication of meta-analyses; it could be beneficial for journals to consider more explicit statements regarding the amount, and type of detail, regarding risk of bias assessments to improve reporting. For example, guidances from journals included in this analysis refer authors to PRISMA, MOOSE, and other relevant guidances in reporting of systematic reviews [\u003cspan additionalcitationids=\"CR30 CR31 CR32\" citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. In addition to the commonly used system developed by GRADE Working Group, others have been developed by the AHRQ, the US Preventive Services Task Force and the Oxford Centre for Evidence-Based Medicine [\u003cspan additionalcitationids=\"CR19\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. These systems evaluate different domains and incorporate their own processes of translating assessments of a body of literature into clinical recommendations, such as those provided by clinical practice guidelines. However, an assessment of 40 of these systems by the 2002 AHRQ report determined that they are less uniform than those used for assessing individual studies, which may complicate the selection of an appropriate system to rate a body of evidence [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Further instructions from journals may help address limitations identified in this study. For example, while certain ERSs will undoubtedly be preferred in different scenarios, journals may consider establishing a preferred ERS for meta-analyses characteristic of the journal\u0026rsquo;s scope, such as establishing a preferred ERS for interventional and another for observational studies. This may allow for more meaningful comparisons of estimates between meta-analyses published in the same journal, when for example, one meta-analysis evaluates efficacy, and another evaluates safety of the same drug. This task would be more complex if different meta-analyses used various systems to rate the quality of evidence. Journal-specific preference of particular ERSs may cultivate more awareness and familiarity among readers and facilitate application of evidence in practice.\u003c/p\u003e \u003cp\u003eOur analysis has several limitations. Firstly, we only considered a narrow scope of journals, the top five journals in Medicine, General \u0026amp; Internal; this category covers resources on medical specialties such as general medicine, internal medicine, clinical physiology, pain management, military and hospital medicine, whereas, Pharmacology and Toxicology category are not included and covers resources on the discovery and testing of bioactive substances, including animal research, clinical experience, delivery systems, and dispensing of drugs. This category also includes resources on the biochemistry, metabolism, and toxic or adverse effects of drugs. These findings may not be representative of non-medical journals or medical journals with lower impact factors or from specialized practice areas, and Embase search should also have been made. Secondly, the editorial and peer review standards of higher-impact journals in this analysis may have produced findings more reflective of the \u0026ldquo;best practice\u0026rdquo; in meta-analysis production. Among these journals, a large number of meta-analyses originated in BMJ, and the requirements of this journal may disproportionately influence our overall findings. Thirdly, we only evaluated meta-analyses of drug effects, and our conclusions are not generalizable to meta-analyses of other interventions. Combined together, these limitations indicate there may be a greater variety of ERS utilization outside journals and interventions considered in this review.\u003c/p\u003e \u003cp\u003eFuture research, conducting an overview of systematic reviews would be needed. The 40 fields of the protocol should be prospectively registered on PROSPERO, an international prospective database of registered systematic reviews, developed and managed by the Centre for Reviews and Dissemination (CRD) at the University of York.\u003c/p\u003e \u003cp\u003eOur review represents, to the authors\u0026rsquo; knowledge, the first description of the frequency of use of ERSs in the medical literature. Further research should address these findings to develop a general framework for best practices in this field.\u003c/p\u003e "},{"header":"Conclusions","content":" \u003cp\u003eMost meta-analyses of drug effects in high-impact medical journals evaluated individual studies with an ERS, most commonly the Cochrane Risk of Bias Tool, while use of ERSs to evaluate the body of literature was less frequent. The evaluation and application of findings of meta-analyses may be facilitated by familiarity of authors and users of meta-analyses with commonly used ERSs, as well as more specific guidance for journal submissions.\u003c/p\u003e "},{"header":"Abbreviations","content":" \u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAHRQ\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eAgency for Healthcare Research and Quality\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eBMJ\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eBritish Medical Journal\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eERS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eEvidence-Rating System\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eGRADE\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eGrading of Recommendations Assessment, Development, and Evaluation\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eJAMA\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eJournal of the American Medical Association\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eMeSH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eMedical Subject Heading\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eMOOSE\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eMeta-analysis Of Observational Studies in Epidemiology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePRISMA\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ePreferred Reporting Items for Systematic Reviews and Meta-Analyses\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRCT\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRandomized Controlled Trial\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRoBANS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRisk of Bias Assessment Tool for Nonrandomized Studies\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eROBINS-I\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRisk of Bias in Nonrandomized Studies of Interventions\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eUSPSTF\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eUS Preventative Services Task Force\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e "},{"header":"Declarations","content":"\u003cp\u003e- Ethics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e- Consent to publish\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e- Availability of data and materials\u003c/p\u003e\n\u003cp\u003eAll data generated or analysed during this study are included in this published article [and its supplementary information files].\u003c/p\u003e\n\u003cp\u003e- Competing interests\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003e- Funding\u003c/p\u003e\n\u003cp\u003eThis study was not funded, nor received any grant.\u003c/p\u003e\n\u003cp\u003e- Authors' Contributions\u003c/p\u003e\n\u003cp\u003eAFPS conceived the study and designed the paper. AFPS, JMVH, MAM and MAAB drafted the manuscript, analyzed and interpreted the data. MAAL and MTSG helped with the manuscript. All authors were involved in the development of the concept of the article and drafting the article. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e- Acknowledgements\u003c/p\u003e\n\u003cp\u003eJoshua Ryan Comra (BYU pre-med student), Aric Aiton (UWindsor pre-med student), Sandra Pe\u0026ntilde;a (PIMA Medical Institute pre-med student).\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e \u003cspan\u003eAgency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence. Agency for Healthcare Research and Quality website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eGuyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924\u0026ndash;6.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHaidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29\u0026ndash;37.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eMurad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA. 2014;312(2):171\u0026ndash;9.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Trends, charts and maps. Clinicaltrials.gov website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://clinicaltrials.gov/ct2/resources/trends#TypesOfRegisteredStudies\u003c/span\u003e\u003c/span\u003e. Updated November 2018. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eStroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283(15):2008\u0026ndash;12.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eMoher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHiggins PT, Gree S. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Collaboration website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://handbook-5-1.cochrane.org/\u003c/span\u003e\u003c/span\u003e. Updated March 2011. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. About us. Cochrane Collaboration website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cochrane.org/about-us\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHiggins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWest S, King V, Carey TS, et al. Systems to rate the strength of scientific evidence: Full Report. Agency for Healthcare Research and Quality website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eViswanathan M, Berkman ND, Dryden DM, Hartling L. Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank. In: AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAgency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. Agency for Healthcare Research and Quality website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf\u003c/span\u003e\u003c/span\u003e. Published January 2014. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eDowns SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377\u0026ndash;84.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWells GA, Shea B, O'Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.ohri.ca/programs/clinical_epidemiology/nos_manual.pdf\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eLiberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAgency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence: Summary. Agency for Healthcare Research and Quality website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHarris RP, Helfand M, Woolf SH, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(3 Suppl):21\u0026ndash;35.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. OCEBM Levels of Evidence. Oxford website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cebm.net/2016/05/ocebm-levels-of-evidence/\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eJuni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42\u0026ndash;6.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Journal Citation Reports. Clarivate Analytics website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://jcr.incites.thomsonreuters.com/\u003c/span\u003e\u003c/span\u003e. Accessed August 31, 2018.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHays M, Andrews M, Wilson R, Callender D, O'Malley PG, Douglas K. Reporting quality of randomised controlled trial abstracts among high-impact general medical journals: a review and analysis. BMJ Open. 2016;6:e011082.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eRodriguez R. Comparison of indexing times among articles from medical, nursing, and pharmacy journals. Am J Health Syst Pharm. 2016;73(8):569\u0026ndash;75.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eGuyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64:395\u0026ndash;400.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWang L, Gu C. How Would You Evaluate the Quality of Uncontrolled Studies in a Meta-Analysis? Crit Care Med. 2018;46(8):e822\u0026ndash;3.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eSavovic J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157(6):429\u0026ndash;38.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eStagg HR, Zenner D, Harris RJ, Munoz L, Lipman MC, Abubakar I. Treatment of latent tuberculosis infection: a network meta-analysis. Ann Intern Med. 2014;161:419\u0026ndash;28.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Information for authors. Annals of Internal Medicine website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://annals.org/aim/pages/authors\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Instructions for Authors. Journal of the American Medical Association website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://jamanetwork.com/journals/jama/pages/instructions-for-authors\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Resources for authors. The British Medical Journal website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.bmj.com/about-bmj/resources-authors/article-types\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Submission Guidelines. PLoS Medicine website website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://journals.plos.org/plosmedicine/s/submission-guidelines\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAnonymous. Systematic reviews and meta-analyses in The Lancet: formatting guidelines. The Lancet website. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.thelancet.com/pb/assets/raw/Lancet/authors/metaguidelines.pdf\u003c/span\u003e\u003c/span\u003e. Accessed Sep 13, 2019.\u003c/span\u003e \u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Evidence-based medicine, Systematic review, Meta-analysis, Drug information, Pharmacy practice","lastPublishedDoi":"10.21203/rs.3.rs-42029/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-42029/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground: Evidence-rating systems (ERSs) provide a framework for the systematic evaluation of the quality of individual interventional or observational studies and the overall body of evidence in meta-analyses. Authors and users of meta-analyses require a familiarity with ERSs to determine the level of confidence in the application of results. Many ERSs have been published, but no consensus exists regarding best practice for their use. \u003c/p\u003e\u003cp\u003eObjective: The aim is to describe patterns of use of ERSs in meta-analyses of drug therapy published in contemporary high-impact medical journals.\u003c/p\u003e\u003cp\u003eMethods: We design a review. Medline / PubMed was searched to identify meta-analyses evaluating drug therapy from the top 5 ranked general medical journals from 2012 to 2016. Methods of full-texts were reviewed to ensure the meta-analyses evaluated drug therapy and to identify the ERS used to rate individual studies and the overall body of evidence. Frequency of ERS use was analyzed using descriptive statistics.\u003c/p\u003e\u003cp\u003eResults: The top-ranked journals were \u003cem\u003eAnn Intern Med, BMJ, JAMA, Lancet \u003c/em\u003eand\u003cem\u003e PLoS Medicine\u003c/em\u003e. Of the 309 results, manual review excluded 111 meta-analyses. In 198 evaluated meta-analyses, 86.4% (171) utilized an ERS; the most commonly used was the Cochrane Risk of Bias Tool in 80.7% (138) of all meta-analyses. An ERS was used to evaluate the body of literature in 19.1% (38) of meta-analyses; the most commonly used of three systems was the GRADE methodology. Overall, 14 unique ERSs, including author-defined systems, were used\u003c/p\u003e\u003cp\u003eConclusions: Most meta-analyses of drug effects in high-impact medical journals evaluated individual studies with an ERS, most commonly the Cochrane Risk of Bias Tool, while the use of ERSs to evaluate the body of literature was less frequent. The familiarity of authors and users of meta-analyses with commonly used ERSs may facilitate the evaluation and application of findings of meta-analyses.\u003c/p\u003e","manuscriptTitle":"Analysis of Evidence-Rating Systems Used in Meta-Analyses of Pharmacotherapy","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2020-07-22 14:08:02","doi":"10.21203/rs.3.rs-42029/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6fccb807-1837-4d6c-8bff-849d6ce147b2","owner":[],"postedDate":"July 22nd, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":188131,"name":"Health Economics \u0026 Outcomes Research"}],"tags":[],"updatedAt":"2020-09-18T17:27:48+00:00","versionOfRecord":[],"versionCreatedAt":"2020-07-22 14:08:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-42029","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-42029","identity":"rs-42029","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}