The Comparability of Manual vs. Algorithm-Based Calculation of Clinical Trial Methodological Quality Indices

The Comparability of Manual vs. Algorithm-Based Calculation of Clinical Trial Methodological Quality Indices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Comparability of Manual vs. Algorithm-Based Calculation of Clinical Trial Methodological Quality Indices OSCAR ALATERAS, Courtney Chesser, Isabelle Burke, Benjamin Hampstead, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5486560/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Background Evidence hierarchies guide evidence-based practice by ranking forms of evidence to support translation and clinical decision-making. Systematic reviews and meta-analyses (SRMAs) represent the highest form of evidence but are time and resource-intensive, contributing to the estimated 17-year lag in the translation of evidence into practice. Tools that automate aspects of the systematic review processes aim to shorten this gap. Specifically, algorithm-based evaluation of study quality, as performed in the CogTale evidence synthesis platform, accelerates such processes relative to manual methods, leading to a more rapid synthesis of the evidence. In this study, we assessed the agreement between CogTale’s algorithm-based scoring of the PEDro and RoB scales with manual scoring of these scales. Methods We selected 37 randomised controlled trials (RCTs) with PEDro scores available on the NeuroBITE Platform and 37 trials with Risk of Bias (RoB) scores available in Cochrane meta-analyses. Agreement for individual PEDro and RoB items was evaluated using Gwet’s AC1, while a Bland-Altman plot assessed total PEDro score agreement. Results The Bland-Altman analysis showed an average difference in PEDro scores of 0.92 between CogTale and NeuroBITE, with limits of agreement from − 2.09 to 3.93. Gwet's AC1 revealed almost perfect agreement for PEDro items P1, P2, and P11; substantial agreement for P5, P6, P7, and P10; moderate agreement for P3 and P4; slight agreement for P8; and poor agreement for P9. For RoB domains, substantial agreement was found for random sequence generation, allocation concealment, and detection bias, with fair agreement in other domains. Conclusions Overall, CogTale's algorithm-based PEDro and RoB scores align well with manual scores, despite some discrepancies in specific PEDro (P3, P8, P9) and RoB items, likely due to systematic scoring criteria variations. CogTale shows promise in automating quality assessments, potentially reducing time for evidence synthesis while maintaining accuracy. Future research should address key limitations by examining how scoring differences impact meta-analytic outcomes and evaluate CogTale's performance with larger datasets as more evidence accumulates on the platform. Systematic reviews meta-analysis methodological quality PEDro scale Risk of Bias (RoB) algorithm-based scoring CogTale platform Figures Figure 1 1. Background Hierarchies of evidence, such as the Oxford Levels of Evidence (Burns et al., 2011 ), are fundamental to evidence-based practice. They serve as a critical framework for systematically ranking the strength of evidence in addressing health-related questions—whether concerning diagnosis, treatment, prognosis, or other clinical issues. By prioritising higher levels of evidence in clinical practice guidelines, these hierarchies ensure that recommendations are founded on the most rigorous and reliable data, thereby fostering consistency and integrity in evidence-based practice. Systematic reviews and meta-analyses (SRMAs) are regarded as the pinnacle of the evidence hierarchy within evidence-based medicine, offering the most robust form of evidence (Myung, 2023 ). By aggregating data from multiple studies, SRMAs provide a comprehensive and reliable summary of the existing research on a given topic. This method reduces individual study biases and enhances the statistical power to detect true effects, making SRMAs highly valuable in guiding clinical decision-making (Ganeshkumar & Gopalakrishnan, 2013 ). Over time, the methodological standards for conducting SRMAs have become more rigorous. The process now involves exhaustive literature searches and a lengthy article screening process, stringent selection criteria, detailed data extraction, complex statistical analysis, and grading of the evidence to name a few. These increasing demands have made the production of high-quality SRMAs more resource-intensive and time-consuming. Researchers must dedicate substantial effort and time to ensure their reviews adhere to these stringent standards, which, although essential for maintaining quality, also prolongs the time required to complete these reviews. Simultaneously, the rate at which new evidence is generated, especially from randomised controlled trials (RCTs), continues to accelerate. This rapid influx of new studies often results in SRMAs quickly becoming outdated, which poses a challenge for maintaining up-to-date clinical guidelines. Compounding this issue is the sluggish pace of translating research findings into clinical practice. There is an estimated 17-year delay from the time research is conducted to when it is implemented in clinical settings (Morris et al., 2011 ). This significant lag underscores the need for more efficient approaches to synthesising and applying evidence to ensure that clinical practices are informed by the latest and most reliable research. These developments have led to a growing interest in accelerating the evidence synthesis process by automating key stages of reviews to improve efficiency, reduce researcher workload, minimise human errors, and enhance reproducibility through decreased exclusive reliance on subjective human judgment (Toth et al., 2024). A crucial aspect of the complex SRMA process is the critical evaluation of the methodological quality of primary trials, often carried out using validated scales such as the PEDro scale (De Morton, 2009 ) and the Cochrane Risk of Bias (RoB) tool (Higgins, 2011). The PEDro scale is a validated instrument designed to assess the methodological quality of clinical trials. It evaluates key methodological elements, providing explicit criteria that studies must meet to guide reviewers in their evaluations. The PEDro scale offers clear guidelines that help ensure consistent and objective assessment (De Morton, 2009 ). It has demonstrated "fair" to "excellent" inter-rater reliability for both physiotherapy and pharmacological trials, with individual items ranging in reliability from "fair" to "almost perfect" (Cashin & McAuley, 2019 ). Furthermore, the PEDro scale exhibits strong construct and convergent validity, distinguishing effectively between high- and low-quality trials and aligning well with other established quality rating systems (Cashin, 2020). The Cochrane Risk of Bias (RoB) tool assesses potential biases in clinical trials across several domains to determine how these biases might affect the study's validity and outcomes. (Higgins, 2011). The RoB tool outlines specific criteria rater’s use in the evaluation of a study’s potential bias across several domains. Unlike PEDro ratings, RoB ratings go beyond merely identifying the presence of potential methodological threats, as it also requires evidence that these threats are likely to actually bias the results of a study. Accordingly, the effective use of the tool requires a degree of expertise in a particular field. These tools are frequently employed in systematic reviews. For example, research databases such as NeuroBITE (NeuroRehab Evidence Resource, 2024) and SMRAs performed as part of the Cochrane Database of Systematic Reviews (Cochrane Evidence Synthesis and Methods, 2024) utilise the PEDro scale and the Cochrane RoB tool, respectively, to evaluate the methodological rigour and reliability of trials. Importantly, both tools/scales are manually scored. Obtaining a PEDro score requires a yes or no answer to the 11 items on the scale based on specific criteria. In contrast, the RoB tool requires a more nuanced interpretation, as reviewers must judge whether potential sources of bias are present in the study and whether they are likely to have impacted the study's findings, demanding a more advanced understanding of the field from the reviewer. In the case of the RoB tool, it is standard practice in Cochrane Reviews for such ratings to be made by two reviewers independently. Although no formal data is available on the time it takes to complete these rating scales manually, RoB ratings are estimated to take between 30–120 minutes to complete, whereas PEDro ratings might take between 10–30 minutes to complete. Factors such as the complexity of trials, experience of the rater’s, and frequency of the need to consult formal manuals may all affect these estimates. Recognising the effort and time involved in these manual evaluations, tools are now being developed to accelerate and automate the scoring and rating process. Some tools use Artificial Intelligence to perform the rating process. For example, RobotReviewer (Marshall et al. 2017) uses Machine Learning and Natural Language Processing (NLP) to extract data from RCTs and to calculate Risk of Bias. Recent studies (Armijo-Olivo et al. 2020 ; Tian et al. 2024 ) found that RobotReviewer performed moderately well relative to humans on the domains of Random Sequence Generation and Blinding of Participants and Personnel , but that agreement was poor for Allocation Concealment and Blinding of Outcomes Assessors and both reviews concluded that the tool was not a suitable replacement for human raters. Alternatively, algorithm-based approaches use scoring rules to calculate item and total scores and can be used after data have been manually extracted. We focus on CogTale ( www.cogtale.com ), a comprehensive online repository and platform designed for the evaluation, synthesis, and dissemination of evidence from cognition-oriented treatments (COTs) studies in the older adult population (Sabates et al., 2021 ). The CogTale database provides detailed information on the design and methodology of intervention studies based on extensive data extraction by trained human reviewers. Using an algorithm-based approach, CogTale calculates several methodological quality scores, including the PEDro and Cochrane Risk of Bias (RoB) scores. The CogTale platform includes numerous other features, most importantly, the ability to rapidly conduct a meta-analysis of studies included in the platform (for more details, see Sabates et al. 2021 ). CogTale offers an alternative avenue for expediting systematic reviews and meta-analyses in the field of COTs in older adults. Although AI-based approaches to data extraction on the CogTale platform have not yet reached acceptable standards (Rasool et al. 2024 ), the algorithm-based approach it employs significantly speeds up the calculation of methodological quality scores. This reduces the time needed to produce systematic reviews and meta-analyses which in turn has the potential to reduce the previously mentioned translational time lag, allowing evidence-based practice to be implemented more rapidly and ensuring that systematic reviews and meta-analyses remain up to date. As CogTale’s algorithm-based approach has not yet been formally evaluated, the aim of the current study is to evaluate the level of agreement between PEDro and RoB scores produced by CogTale and those produced manually. 2. Method 2.1 Study Selection A convenience sample of randomised controlled trials (RCTs) was selected to evaluate the effects of COTs in older adults. Studies were chosen based on two separate criteria: (1) availability of PEDro scores on CogTale and/or NeuroBITE or (2) presence of Cochrane risk of bias (RoB) scores in both CogTale and Cochrane meta-analyses. It was not necessary for studies to meet both criteria to be included. This selection process resulted in two datasets: 37 studies with Pedro scores and 37 studies with RoB scores. Some studies met both criteria and thus appeared in both datasets, while others had only one type of score. The full dataset can be found in supplementary materials. To assess the adequacy of our sample, we conducted a post-hoc power analysis which indicated that our sample of 37 studies provided 80% power to detect correlations of ≥ 0.4 (α = 0.05), sufficient to detect agreement coefficients above chance levels. Representativeness was evaluated by comparing methodological quality metrics between our sample and the broader CogTale repository (N = 399). Statistical comparison of PEDro scores showed no significant difference in distribution (Kolmogorov-Smirnov test, p = .135). For Risk of Bias assessments, our sample showed similar distributions in three of six domains. Additionally, all our ROB sample studies were included in published Cochrane meta-analyses focused on cognitive rehabilitation and training for people with mild to moderate dementia (Bahar-Fuchs et al., 2019 ; Kudlicka et al., 2023 ), indicating they adequately represent the types of evidence synthesised in this field. Overall, our sample reflects the quality distribution of studies in the CogTale repository, which comprehensively catalogues cognition-oriented treatment research, making our selection appropriate for evaluating quality assessment tools in this specific research domain. 2.2 Methodological Quality Scores We assessed methodological quality using both the RoB and PEDro scales. The RoB scores were sourced from CogTale and compared with scores from two relevant Cochrane meta-analyses (Bahar-Fuchs et al., 2019 ; Kudlicka et al., 2023 ). Similarly, the PEDro scores were obtained from CogTale and compared with those for the same studies published on the NeuroBITE platform. Importantly, although scoring of PEDro items on the CogTale platform was generally based on the PEDro manual (used by NeuroBITE), there are nonetheless several differences in how certain items are calculated, as shown in Table 1 . Table 1 PEDro Item Criteria PEDro Item Source Criteria P1. Eligibility Criteria CogTale Inclusion and exclusion criteria specified as well as the recruitment source. NeuroBITE Same criteria as CogTale. P2 Random Allocation CogTale Subjects randomly allocated to interventions. NeuroBITE Same criteria as CogTale. P3. Allocation Concealment CogTale Person responsible for determining eligibility of participants unaware of participant intervention allocation, at the time of decision. NeuroBITE Person responsible for determining eligibility of participants unaware of participant intervention allocation, at the time of decision, or, it is clear that participants were only allocated to groups after they were enrolled. P4. Baseline Comparability CogTale Intervention conditions were comparable at baseline with either no differences between conditions or differences found on no more than one demographic or outcome variable. NeuroBITE Intervention conditions were comparable (no statistically significant difference) at baseline for relevant outcomes P5. Participant Blinding CogTale Participants were blinded in the study NeuroBITE Participants were blinded in the study; criteria is met even if blinding was not 100% successful P6. Therapist Blinding CogTale Therapists were blinded in the study NeuroBITE Therapists blinded from group allocation; criteria is met even if the blinding was not 100% successful P7. Assessor Blinding CogTale Assessors were blinded in the study NeuroBITE Assessors were blinded in the study; criteria is met even if the blinding was not 100% successful P8. Retention Rate CogTale The proportion of participants retained post treatment in the both the experimental and control group was above 85% NeuroBITE Explicitly stated that at least one post-treatment outcome obtained results from more than 85% of participants and was this explicitly stated P9. Intention to Treat CogTale Intention to treat analyses was used or was the intervention delivered to each participant as intended NeuroBITE Intention to treat analyses was used or was it explicitly stated that post-treatment data has been analyses according to initial condition allocation P10. Between Group Analyses CogTale Between-group statistical comparisons were reported for at least one outcome NeuroBITE Same criteria as CogTale. P11. Statistics Reported CogTale Study reported both point measures and measures of variability for at least one outcome measure. NeuroBITE Same criteria as CogTale. Note. Table 1 provides a shorthand (e.g. P1. Eligibility Criteria) for each PEDro item and provides information on how CogTale and NeuroBITE each assess these items. Differences in construct definition between CogTale and NeuroBITE are highlighted in bold. 2.2.1 CogTale CogTale generated both PEDro and RoB scores through a systematic and algorithm-driven process. Data for trials included in this study was extracted by coders who were graduate students taking part in the CogTale Internship – a program through which they received further training in understanding trial design, evaluation, and evidence synthesis. The CogTale data extraction form captures numerous design features necessary for quality assessment, but coders are unaware of or have access to the scoring algorithm, effectively blinding them to how their responses would impact the final scores. Further reducing bias, the lead investigators who developed CogTale were not involved in either the data extraction process or statistical analyses. The first author, who has no affiliation with CogTale, conducted an additional independent review on a subset of these extraction forms to ensure accuracy, and all data analysis was performed using anonymised datasets to prevent identification of specific studies during comparison. CogTale's algorithm was then used to automatically calculate quality scores based on specific responses from the extraction form. For example, the PEDro item P1 (Eligibility Criteria) is determined by responses to two specific questions: " Was eligibility criteria specified? " and " Was the source of participants specified? ". If both questions are answered "yes," P1 receives a score of 1; otherwise, it scores 0. This systematic approach is applied consistently across all PEDro and RoB items, ensuring objectivity in the scoring process. 2.2.2 NeuroBITE PEDro Scores NeuroBITE used the PEDro-P scale, which is a refinement of the original PEDro scale specifically designed to rate studies on NeuroBITE. The PEDro-P scale has adapted and re-worded the same 11 criteria from the original PEDro scale to ensure they apply unambiguously to trials on NeuroBITE. Reviewers are trained to complete manual PEDro ratings based on specific criteria for each item, as shown in Table 1 , though it is unclear how many raters scored the PEDro scale in NeuroBITE. 2.2.3 Cochrane Meta-Analyses RoB Scores Studies from Cochrane meta-analyses included in the current study were assessed using the Cochrane RoB 1 Tool (Higgins, 2011). The process involves a detailed manual evaluation of each study, where usually two trained reviewers independently assess and rate the risk of bias. After completing their individual assessments, the reviewers discuss any discrepancies to reach a consensus on the final scores, ensuring consistency and accuracy in the evaluation. 2.3 Statistical Analyses We analysed the level of agreement between PEDro total scores using a Bland-Altman plot. This analysis provided critical insights into both the bias and variability between the two sets of scores. The average bias, represented by the mean difference, indicated the systematic difference between the two measurement tools, showing whether one consistently rated higher or lower than the other. The limits of agreement gave a measure of variability, showing the range within which most of the score differences lay. This is essential for understanding how much variation exists between the two scoring methods and assessing if the differences are clinically acceptable. Additionally, by plotting the differences against the average of the two scores, we were able to detect any potential trends, such as proportional bias, where the differences between the scores vary across the scale. This is crucial to determine whether there is consistent agreement across the full scoring range or if the agreement breaks down at higher or lower scores. Overall, the Bland-Altman plot offered a comprehensive understanding of both the extent of bias and the variability, allowing us to evaluate the level of agreement between the two PEDro total scores and their potential interchangeability. To assess the level of agreement between PEDro items, we used unweighted kappa, which measures the degree of agreement between two raters beyond what would be expected by chance. This statistic tells us how consistently the raters scored each item to determine whether there is meaningful agreement. However, due to the binary nature of the data, the kappa statistic is vulnerable to the kappa prevalence paradox, where high or low prevalence of one category can lead to a misleadingly low kappa value despite high observed agreement (Zec et al., 2017 ). For example, if both raters consistently agree that an item is 'absent' for most cases, the kappa value may appear low, even though their agreement is strong. To address this limitation, we also employed Gwet’s AC1 and Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) as more robust metrics for assessing agreement. Gwet’s AC1 adjusts for the prevalence paradox by considering the probability of chance agreement differently, offering a more stable estimate of agreement across different prevalence rates (Wongpakaran et al., 2013 ). PABAK similarly adjusts for prevalence and bias by recalculating the kappa statistic to account for these factors, making it less sensitive to distributional imbalances in the data (Chen et al., 2009 ). Gwet’s AC1 and PABAK were reported as the primary statistics, as they offer a more accurate reflection of agreement in this context. Although Gwet’s AC1 and PABAK are not exact substitutes for unweighted kappa, they still provide valuable insights into the level of agreement between the two measures by offering complementary perspectives that mitigate the limitations of kappa. The inclusion of these metrics to accommodate for the kappa prevalence paradox is supported by their use in other studies (e.g, Boye et al., 2020 ), making this approach both robust and well-validated in agreement analyses. To assess the level of agreement between Risk of Bias items, we used weighted kappa since this provides harsher penalties for disagreements when studies were rated as either low or high risk of bias, as these discrepancies are more critical than those involving an unclear risk of bias. By using a squared weighting matrix, we assigned greater penalties to larger differences in ratings, ensuring that the level of disagreement between extreme ratings (low vs. high risk) was more heavily reflected in the final agreement statistic. This approach provided a more nuanced evaluation of agreement, considering the relative importance of different types of disagreements. We also reported Gwet’s AC1 to control for the kappa prevalence paradox, as the ordinal data only had three categories, making it vulnerable to this issue. Gwet’s AC1 used a similar weighting matrix as weighted kappa to reflect the importance of larger discrepancies. PABAK was not reported for Risk of Bias items, as it could not be weighted. Gwet’s AC1 will be reported as the primary statistic, providing a more stable estimate of agreement. 3. Results 3.1 Level of Agreement Between PEDro Total Scores The Bland-Altman plot (Fig. 1 ) revealed the mean difference (bias) between the two measures was M = 0.92 (95% CI: 0.36 to 1.38). The limits of agreement, defined as the mean difference ± 1.96 times the standard deviation of the differences, were − 2.09 to 3.93. The plot also revealed greater variability at the extremes of the measurement range such that at both lower and higher means, the differences between the two measures are more pronounced and approach the limits of agreement. These results suggest that although the measures are generally in agreement, there is increased variability in the agreement at the extremes of the score range. 3.2 Level of Agreement Between PEDro Items Raw values for Gwet’s AC1 and PABAK for each of the 11 PEDro items are reported in Table 2 . Overall, items P1, P2, and P11 demonstrated almost perfect agreement between raters, as indicated by Gwet’s AC1 values. Items P5, P6, P7, and P10 exhibited substantial agreement, while P3 and P4 showed moderate agreement. Additionally, item P8 had slight agreement, and P9 showed poor agreement, suggesting substantial variability in rater judgments for these items. Table 2 PEDro Item Level of Agreement Statistics PEDro Item PA UK Gwet AC1 PABAK P1. Eligibility Criteria 84.21 0.16 (.280) 0.81 (< .001) 0.68 P2. Random Allocation 94.60 0.77 (< .001) 0.93 (< .001) 0.89 P3. Allocation Concealment 70.26 0.32 (.040) 0.48 (.003) 0.41 P4. Baseline Comparability 68.59 0.22 (.112) 0.46(.005) 0.35 P5. Participant Blinding 81.08 0(1.000) 0.77(< .001) 0.62 P6. Therapist Blinding 81.08 -0.08(0.565) 0.77(< .001) 0.62 P7. Assessor Blinding 89.19 0.78(< .001) 0.79(< .001) 0.78 P8. Retention Rate 56.76 0.13(.420) 0.15 (.378) 0.14 P9. Intention to Treat 40.54 0.08 (.379) -0.19 (.258) -0.19 P10. Between-Group Analyses 83.78 0.17(.295) 0.80 (< .001) 0.67 P11. Statistics Reported 83.78 -0.04(.689) 0.81(< .001) 0.68 Note. PA = Percentage Agreement; UK = Unweighted Kappa; PABAK = Prevalence Adjusted and Bias Adjusted Kappa. The table provides raw statistics for the level of agreement between CogTale and NeuroBITE PEDro items. Both UK and Gwet’s AC1 are presented with their corresponding coefficients and p-values. 3.3 Level of Agreement Between Risk of Bias Items Raw statistics for Gwet’s AC1 are reported in Table 3 . The results show that random sequence generation, allocation concealment, and detection bias had substantial levels of agreement, while performance bias, attrition bias, and reporting bias demonstrated fair levels of agreement. Table 3 Level of Agreement Between Risk of Bias Items for CogTale and Cochrane Meta-Analyses Item PA WK Gwet AC1 Random Sequence 67.57 0.40(.011) 0.69(< .001) Allocation Concealment 62.16 0.422(.002) 0.68(< .001) Detection Bias 64.86 0.363(.008) 0.68(< .001) Performance Bias 29.73 − .087(.302) 0.35(.003) Attrition Bias 21.62 .082(.410) 0.26(.002) Reporting Bias 45.95 − .020(.888) 0.35(.022) Note. PA = Percentage Agreement; WK = Weighted Kappa. The table provides raw statistics for the level of agreement between CogTale and Cochrane Risk of Bias items. Both UK and Gwet’s AC1 are presented with their corresponding coefficients and p-values. 4. Discussion The aim of this study was to investigate how well the CogTale PEDro and Risk of Bias scores align with manual scoring of these indices. The Bland-Altman analysis shows that, on average, CogTale produced slightly higher PEDro total scores compared to NeuroBITE. However, the relatively small bias suggests that the two tools are generally aligned when assessing PEDro scores. Despite this general alignment, there is substantial variability in the individual differences between the tools' scores, as indicated by the wide LoA. The variability is especially pronounced at both the lower and higher ends of the score range, where differences between CogTale and NeuroBITE are more substantial. In contrast, the middle range of scores shows tighter clustering of differences, suggesting better consistency between the tools in this range. This pattern of variability implies that while the tools show good agreement on average, they may not be interchangeable across the entire range of scores. Particularly, users should be cautious when interpreting CogTale scores at the extreme ends of the score spectrum, as the differences between the tools become more pronounced in these regions. To further investigate the underlying causes of these discrepancies, we conducted an item-level analysis to assess the level of agreement between CogTale and NeuroBITE for PEDro scores. While the majority of items showed substantial to almost perfect agreement, items P3, P4, P8, and P9 displayed only slight to moderate agreement, contributing significantly to the discrepancies in total scores. A critical examination revealed that these discrepancies stem from how the CogTale algorithm translates abstract quality concepts into concrete decision rules, creating different operational definitions. For instance, in P9 ("Intention to Treat"), CogTale accepts either the use of intention to treat or evidence that the intervention was delivered as allocated, whereas NeuroBITE requires explicit mention of intention to treat or clear statement that post-treatment data was analysed according to initial allocations. This definitional difference results in CogTale providing more lenient ratings for P9, as our data confirms. Conversely, for P3 ('Allocation Concealment'), CogTale applies stricter standards, requiring that randomisation outcomes were concealed until intervention commencement, while NeuroBITE accepts studies where participants were assigned to groups after enrolment without explicit mention of concealment methods. Similar definitional differences exist in the algorithmic operationalisation of P4 ('Baseline Comparability') and P8 ('Retention Rate'), creating subtle differences in construct definitions that led to either more lenient or stricter CogTale ratings compared to NeuroBITE—patterns consistently observed in our data. These inconsistencies have significant implications: either the CogTale algorithm needs revision to better align with standard PEDro definitions, or alternatively, CogTale's PEDro scores should be considered a modified version of the scale rather than directly interchangeable with NeuroBITE. The level of agreement was substantial on three Risk of Bias domains (e.g., Random Sequence, Allocation Concealment, and Detection Bias), but was only fair for the other three domains (e.g., performance, attrition, and reporting bias). These differences, however, are not necessarily indicative of invalid assessments by CogTale but are more likely to reflect conceptual differences in how the items are evaluated. CogTale's algorithm focuses on identifying whether certain risk of bias criteria is present in a study, while Cochrane meta-analyses take an additional step by evaluating the potential impact of these criteria on the study's outcomes to determine the risk of bias (Higgins, 2011). This added layer of subjective expert interpretation in Cochrane’s approach, absent in CogTale, likely accounts for the discrepancies between the two tools in risk of bias assessments. For example, when assessing performance bias, both Cochrane meta-analyses and CogTale examine whether the participants were blinded. However, Cochrane also considers whether the lack of blinding would likely bias the results, while CogTale does not assess the likelihood of this bias occurring, suggesting a systematic difference between the tools. In cognitive interventions, the blinding of participants may not always lead to performance bias if the intervention relies on self-driven cognitive tasks, where awareness of the intervention's purpose is less likely to influence outcomes. In contrast, blinding may be crucial in studies where participant expectations can strongly influence results, such as pharmacological interventions. These interpretation differences suggest that the poor agreement is likely due to systematic error arising from how each construct is assessed, rather than random error. Although the more nuanced approach required for the assessment of Risk of Bias in Cochrane Reviews is useful when it comes to drawing a distinction between threats to bias and actual bias, the approach requires greater subject matter expertise, and given the duplicate nature of RoB assessments in Cochrane Reviews, may also increase the frequency of inter-rater discrepancies, owing in part to differences in rater’s experience. The algorithm-based approach implemented in the CogTale platform, in contrast, is not influenced by subjective factors and in that regard is likely to yield more consistent scores. Importantly, the algorithm-based approach to RoB ratings in the CogTale Platform performed better overall than current AI/NLP-based approaches such as RobotReviewer. While RobotReviewer showed moderate agreement with human raters for Random Sequence Generation and Blinding of Participants and Personnel, it exhibited poor agreement for Allocation Concealment and Blinding of Outcome Assessors (Tian et al., 2024 ). Conversely, CogTale demonstrated higher agreement across these domains, performing especially well in areas where RobotReviewer encountered difficulties, such as Allocation Concealment. These performance differences likely stem from fundamental methodological distinctions between the systems. RobotReviewer employs a fully automated AI and machine learning approach that analyses full-text articles to generate risk classifications without human intervention (Tian et al., 2024 ). In contrast, CogTale uses a semi-automated approach where human reviewers manually extract data, which is then processed through structured questions and pre-defined decision rules and without incremental improvement/adaptation of the model. This human-in-the-loop design allows CogTale to capture key methodological elements more systematically than fully automated systems while potentially achieving greater consistency than fully manual human ratings. These results suggest that CogTale's algorithm-based system offers a more suitable alternative for accelerating the Risk of Bias rating process, addressing the shortcomings seen in previous AI-driven approaches like RobotReviewer. The findings of this study have important implications for the validity and broader application of CogTale’s algorithm based PEDro and Risk of Bias scores. While systematic differences in the assessment criteria between CogTale and manual scoring methods like NeuroBITE and Cochrane were observed, these discrepancies are primarily attributable to variations in evaluation approaches, rather than a lack of validity in the CogTale system. The overall consistency of scores, particularly for the PEDro scale, supports the accuracy and utility of CogTale’s automated assessments. Although certain items—such as P3 (Allocation Concealment) and P9 (Intention to Treat)—revealed divergent scoring tendencies due to differences in evaluation criteria, the observed patterns reflect expected systematic variations in criteria, rather than random error. Thus, while CogTale's scores should be regarded as a modified version of traditional manual assessments, the high level of agreement for most items reinforces the tool’s potential to serve as a valid and reliable alternative for methodological evaluation. These findings also highlight the potential of CogTale to substantially expedite the meta-analysis process by automating the traditionally time-consuming methodological assessment of studies. Manual scoring of PEDro and Risk of Bias items is often laborious, contributing to delays in the synthesis and dissemination of research findings. CogTale’s algorithm-based approach offers a streamlined solution, enabling faster generation of methodological scores without significant loss of validity. This increased efficiency is particularly relevant for systematic reviews and meta-analyses, where methodological assessments can represent a bottleneck. By reducing the time required for these evaluations, CogTale has the potential to accelerate evidence synthesis, ensuring that systematic reviews remain timely and that clinical guidelines can be updated more rapidly. Ultimately, the integration of CogTale into evidence synthesis workflows in the field of COTs in ageing and dementia, could help bridge the gap between research and practice, facilitating the timely implementation of evidence-based interventions in clinical settings and improving the overall responsiveness of healthcare to emerging research. Despite the important findings of this study, several limitations must be considered. One limitation of this study is the relatively small sample size of studies included in the analysis, which may limit the generalisability of the findings. This limitation arises from the restricted availability of studies that are accessible with Risk of Bias and/or PEDro scores on both the CogTale and NeuroBITE or Cochrane platforms. As a result, the comparisons made between the scoring systems might not fully capture the broader range of possible variations across different types of studies. This could mean that the observed differences between the tools are more reflective of the specific set of studies analysed rather than being indicative of the tools’ performance across all possible contexts. As evidence keeps being added to the CogTale platform, these analyses can be repeated with larger datasets. Another limitation of this study is its focus on score alignment without considering how discrepancies in scoring between CogTale and NeuroBITE might influence the outcomes of systematic reviews and meta-analyses. Since CogTale incorporates PEDro scores as part of its overall methodological grading, the slightly more lenient scoring by CogTale could potentially lead to more favourable evaluations of study quality in these analyses. This leniency might result in higher methodological ratings for some studies, affecting the subsequent grading of the evidence and conclusions drawn from meta-analyses. Work is currently underway to evaluate the extent to which meta-analytic studies conducted in CogTale broadly replicates the findings from published meta-analyses (Hiskens-Raven, In Preparation ), and this will further clarify this issue. It is important to note, however, that PEDro scores are just one of several factors contributing to CogTale’s overall grading framework. Therefore, while CogTale may apply slightly more lenient scoring in this domain, the broader grading process remains comprehensive, reducing the likelihood that this leniency will substantially bias the overall quality assessments. Declarations Ethics approval and consent to participate. Not applicable. Consent for publication. Not applicable. Availability of data and materials All data used during this study is included in the Supplementary Information files. Competing interests ABF, BMH, SB, and SSS are the scientific developers of the CogTale platform, which was evaluated in this study. The first author (OA) and other co-authors (CC, IB) have no competing interests to declare." Funding Funding in support of this project was provided by NIH-NIA R35 Award (to Benjamin Hampstead). Authors’ contributions OA led the analysis and manuscript preparation. CC assisted with analyses, writing the manuscript preparation. IS assisted with the acquisition of the data. BMH, SB and SSS all contributed to the development of the CogTale database and manuscript preparation. ABF designed the study, led the development of the CogTale database, and contributed to manuscript preparation. All authors have reviewed and approved the manuscript. Acknowledgments The CogTale team would like to acknowledge the contributions of a range of trainee coders through the CogTale internship who have assisted with establishing and growing the CogTale database. References Armijo-Olivo S, Craig R, Campbell S. Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials. Res Synthesis Methods. 2020;11(3):484–93. https://doi.org/10.1002/jrsm.1398 . Bahar-Fuchs A, Martyr A, Goh AM, Sabates J, Clare L. Cognitive training for people with mild to moderate dementia. Cochrane Libr. 2019. https://doi.org/10.1002/14651858.cd013069.pub2 . Boye KS, Matza LS, Currie BM, Coyne KS. Validity and analysis of the Diabetes Injection Device Preference Questionnaire (DID-PQ). J Patient-Reported Outcomes. 2020;4(1). https://doi.org/10.1186/s41687-020-00266-x . Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in Evidence-Based Medicine. Plast Reconstr Surg. 2011;128(1):305–10. https://doi.org/10.1097/prs.0b013e318219c171 . Cashin AG, McAuley JH. Clinimetrics: Physiotherapy Evidence Database (PEDRO) scale. J Physiotherapy. 2019;66(1):59. https://doi.org/10.1016/j.jphys.2019.08.005 . Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol. 2009;9(1). https://doi.org/10.1186/1471-2288-9-5 . Cochrane Evidence Synthesis and Methods . (2024). Cochrane. Retrieved September 21, 2024, from https://www.cochrane.org/ De Morton NA. The PEDro scale is a valid measure of the methodological quality of clinical trials: a demographic study. Australian J Physiotherapy. 2009;55(2):129–33. https://doi.org/10.1016/s0004-9514(09)70043-1 . Ganeshkumar P, Gopalakrishnan S. Systematic reviews and meta-analysis: Understanding the best evidence in primary healthcare. J Family Med Prim Care. 2013;2(1):9. https://doi.org/10.4103/2249-4863.109934 . Higgins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne J. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343(oct18 2):d5928. https://doi.org/10.1136/bmj.d5928 . a. C. Kudlicka A, Martyr A, Bahar-Fuchs A, Sabates J, Woods B, Clare L. Cognitive rehabilitation for people with mild to moderate dementia. Cochrane Libr. 2023;2023(6). https://doi.org/10.1002/14651858.cd013388.pub2 . Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20. https://doi.org/10.1258/jrsm.2011.110180 . Marshall IJ, Kuiper J, Banner E, Wallace BC. (2017, July). Automating biomedical evidence synthesis: RobotReviewer. Proceedings of the Conference of the Association for Computational Linguistics Meeting , 2017, 7–12. https://doi.org/10.18653/v1/P17-4002 Myung S. How to review and assess a systematic review and meta-analysis article: a methodological study (secondary publication). J Educational Evaluation Health Professions. 2023;20:24. https://doi.org/10.3352/jeehp.2023.20.24 . NeuroRehab Evidence Resource . (2024, September 16). NeuroBITE. Retrieved September 21, 2024, from https://neurorehab-evidence.com/web/cms/content/home Rasool Z, Kurniawan S, Balugo S, Barnett S, Vasa R, Chesser C, Bahar-Fuchs A. Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset. Nat Lang Process J. 2024;100083. https://doi.org/10.1016/j.nlp.2024.100083 . Sabates J, Belleville S, Castellani M, Dwolatzky T, Hampstead BM, Lampit A, Simon S, Anstey K, Goodenough B, Mancuso S, Marques D, Sinnott R, Bahar-Fuchs A. CogTale: an online platform for the evaluation, synthesis, and dissemination of evidence from cognitive interventions studies. Syst Reviews. 2021;10(1). https://doi.org/10.1186/s13643-021-01787-2 . Tian Y, Yang X, Doi SA, Furuya-Kanamori L, Lin L, Kwong JS, Xu C. Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans. Res Synthesis Methods. 2024. https://doi.org/10.1002/jrsm.1761 . Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Reviews. 2024;13(1). https://doi.org/10.1186/s13643-024-02592-3 . Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1). https://doi.org/10.1186/1471-2288-13-61 . Zec S, Soriani N, Comoretto R, Baldi I. High agreement and high prevalence: the paradox of Cohen’s Kappa. Open Nurs J. 2017;11(1):211–8. https://doi.org/10.2174/1874434601711010211 . Supplementary Files Supplementarydataset.xls Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 16 Apr, 2025 Reviewers invited by journal 12 Apr, 2025 Editor assigned by journal 02 Apr, 2025 First submitted to journal 01 Apr, 2025 Editorial decision: Minor revision 10 Mar, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5486560","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":442160548,"identity":"0f2e63da-6b59-4f20-9ca3-b550106fbc25","order_by":0,"name":"OSCAR ALATERAS","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAUlEQVRIiWNgGAWjYFAC5gYIzd5gwMBwQALENCCghbGB4QCI5jlAshaJBJAWBsJa+NkPNn7+UHFPXn7m440PfpyxyGNgb94mwVBzGKcWyZ7EZokDZ4oNN9xOKzbsuSFRzMBzrEyC4RhuLQYHEhskDrYlMG6QzjGT4PkgAeQCGQxseLScf9j84+C/BPv5M8+Y//wD0iL/BqjlHx4tNxLbJA42JCQ23OAxY+a5AbKFx0yCsQ2PX2Y8bLM4cywhecOZtGJpmTMSiW08acUWiX3pOLXw8ycfvlFRk2A7v/3wxo9vjtUl9rMf3njjwzdrnFowARuISCBBwygYBaNgFIwCTAAAcjRb7nI8dxIAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0008-8409-8960","institution":"Deakin University","correspondingAuthor":true,"prefix":"","firstName":"OSCAR","middleName":"","lastName":"ALATERAS","suffix":""},{"id":442160549,"identity":"47cf38d6-0301-425d-8600-582a09efdccf","order_by":1,"name":"Courtney Chesser","email":"","orcid":"","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Courtney","middleName":"","lastName":"Chesser","suffix":""},{"id":442160550,"identity":"86fa9beb-c011-43e3-a191-575d0321479c","order_by":2,"name":"Isabelle Burke","email":"","orcid":"","institution":"Deakin University - Melbourne Burwood Campus","correspondingAuthor":false,"prefix":"","firstName":"Isabelle","middleName":"","lastName":"Burke","suffix":""},{"id":442160551,"identity":"214c5a24-7e22-4a38-8172-30e7ff0558db","order_by":3,"name":"Benjamin Hampstead","email":"","orcid":"","institution":"University of Michigan","correspondingAuthor":false,"prefix":"","firstName":"Benjamin","middleName":"","lastName":"Hampstead","suffix":""},{"id":442160552,"identity":"cbd5418c-38d7-425b-93aa-ef547b27ecb4","order_by":4,"name":"Sylvie Belleville","email":"","orcid":"","institution":"Universite de Montreal","correspondingAuthor":false,"prefix":"","firstName":"Sylvie","middleName":"","lastName":"Belleville","suffix":""},{"id":442160553,"identity":"b281a688-fbf6-43c0-a5fd-8bae16f4b9b0","order_by":5,"name":"Sharon Sanz-Simon","email":"","orcid":"","institution":"Rutgers New Jersey Medical School","correspondingAuthor":false,"prefix":"","firstName":"Sharon","middleName":"","lastName":"Sanz-Simon","suffix":""},{"id":442160554,"identity":"109a724c-9a81-4062-911a-aeee146c0c7e","order_by":6,"name":"Alex Bahar-Fuchs","email":"","orcid":"https://orcid.org/0000-0002-9248-6057","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Alex","middleName":"","lastName":"Bahar-Fuchs","suffix":""}],"badges":[],"createdAt":"2024-11-20 00:31:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5486560/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5486560/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80624957,"identity":"f3abeb39-4b59-47ab-9d4e-284e4ff854d0","added_by":"auto","created_at":"2025-04-15 10:37:21","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":99260,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eBland-Altman Plot for CogTale and NeuroBITE PEDro Total Scores\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eNote. \u003c/em\u003eBland-Altman plot comparing the total PEDro scores from CogTale and NeuroBITE. The solid red line represents the mean difference (bias) between the two measurements, indicating an average bias of 0.92. The dashed black lines indicate the limits of agreement (LoA), which range from -2.09 to 3.93. The plot shows the difference between the two scores plotted against their mean, with most points falling within the limits of agreement, indicating acceptable agreement between the two measures.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5486560/v1/a497bbdb936137aba774898f.png"},{"id":80627242,"identity":"7ccec1ea-5020-48c8-b4e0-08458d971ed6","added_by":"auto","created_at":"2025-04-15 11:01:21","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":971832,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5486560/v1/dfb14e4e-cf8f-4f38-bee2-92a27a30834a.pdf"},{"id":80626001,"identity":"c2a021ad-9b70-4d5d-b8f7-d2db43991444","added_by":"auto","created_at":"2025-04-15 10:45:21","extension":"xls","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":60928,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementarydataset.xls","url":"https://assets-eu.researchsquare.com/files/rs-5486560/v1/7e42aeeac09a4d988308ec2e.xls"}],"financialInterests":"","formattedTitle":"The Comparability of Manual vs. Algorithm-Based Calculation of Clinical Trial Methodological Quality Indices","fulltext":[{"header":"1. Background","content":"\u003cp\u003eHierarchies of evidence, such as the Oxford Levels of Evidence (Burns et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2011\u003c/span\u003e), are fundamental to evidence-based practice. They serve as a critical framework for systematically ranking the strength of evidence in addressing health-related questions\u0026mdash;whether concerning diagnosis, treatment, prognosis, or other clinical issues. By prioritising higher levels of evidence in clinical practice guidelines, these hierarchies ensure that recommendations are founded on the most rigorous and reliable data, thereby fostering consistency and integrity in evidence-based practice.\u003c/p\u003e \u003cp\u003eSystematic reviews and meta-analyses (SRMAs) are regarded as the pinnacle of the evidence hierarchy within evidence-based medicine, offering the most robust form of evidence (Myung, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). By aggregating data from multiple studies, SRMAs provide a comprehensive and reliable summary of the existing research on a given topic. This method reduces individual study biases and enhances the statistical power to detect true effects, making SRMAs highly valuable in guiding clinical decision-making (Ganeshkumar \u0026amp; Gopalakrishnan, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). Over time, the methodological standards for conducting SRMAs have become more rigorous. The process now involves exhaustive literature searches and a lengthy article screening process, stringent selection criteria, detailed data extraction, complex statistical analysis, and grading of the evidence to name a few. These increasing demands have made the production of high-quality SRMAs more resource-intensive and time-consuming. Researchers must dedicate substantial effort and time to ensure their reviews adhere to these stringent standards, which, although essential for maintaining quality, also prolongs the time required to complete these reviews. Simultaneously, the rate at which new evidence is generated, especially from randomised controlled trials (RCTs), continues to accelerate. This rapid influx of new studies often results in SRMAs quickly becoming outdated, which poses a challenge for maintaining up-to-date clinical guidelines. Compounding this issue is the sluggish pace of translating research findings into clinical practice. There is an estimated 17-year delay from the time research is conducted to when it is implemented in clinical settings (Morris et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). This significant lag underscores the need for more efficient approaches to synthesising and applying evidence to ensure that clinical practices are informed by the latest and most reliable research. These developments have led to a growing interest in accelerating the evidence synthesis process by automating key stages of reviews to improve efficiency, reduce researcher workload, minimise human errors, and enhance reproducibility through decreased exclusive reliance on subjective human judgment (Toth et al., 2024).\u003c/p\u003e \u003cp\u003eA crucial aspect of the complex SRMA process is the critical evaluation of the methodological quality of primary trials, often carried out using validated scales such as the PEDro scale (De Morton, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2009\u003c/span\u003e) and the Cochrane Risk of Bias (RoB) tool (Higgins, 2011). The PEDro scale is a validated instrument designed to assess the methodological quality of clinical trials. It evaluates key methodological elements, providing explicit criteria that studies must meet to guide reviewers in their evaluations. The PEDro scale offers clear guidelines that help ensure consistent and objective assessment (De Morton, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2009\u003c/span\u003e). It has demonstrated \"fair\" to \"excellent\" inter-rater reliability for both physiotherapy and pharmacological trials, with individual items ranging in reliability from \"fair\" to \"almost perfect\" (Cashin \u0026amp; McAuley, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Furthermore, the PEDro scale exhibits strong construct and convergent validity, distinguishing effectively between high- and low-quality trials and aligning well with other established quality rating systems (Cashin, 2020).\u003c/p\u003e \u003cp\u003eThe Cochrane Risk of Bias (RoB) tool assesses potential biases in clinical trials across several domains to determine how these biases might affect the study's validity and outcomes. (Higgins, 2011). The RoB tool outlines specific criteria rater\u0026rsquo;s use in the evaluation of a study\u0026rsquo;s potential bias across several domains. Unlike PEDro ratings, RoB ratings go beyond merely identifying the presence of potential methodological threats, as it also requires evidence that these threats are likely to actually bias the results of a study. Accordingly, the effective use of the tool requires a degree of expertise in a particular field.\u003c/p\u003e \u003cp\u003eThese tools are frequently employed in systematic reviews. For example, research databases such as NeuroBITE (NeuroRehab Evidence Resource, 2024) and SMRAs performed as part of the Cochrane Database of Systematic Reviews (Cochrane Evidence Synthesis and Methods, 2024) utilise the PEDro scale and the Cochrane RoB tool, respectively, to evaluate the methodological rigour and reliability of trials. Importantly, both tools/scales are manually scored. Obtaining a PEDro score requires a yes or no answer to the 11 items on the scale based on specific criteria. In contrast, the RoB tool requires a more nuanced interpretation, as reviewers must judge whether potential sources of bias are present in the study and whether they are likely to have impacted the study's findings, demanding a more advanced understanding of the field from the reviewer. In the case of the RoB tool, it is standard practice in Cochrane Reviews for such ratings to be made by two reviewers independently. Although no formal data is available on the time it takes to complete these rating scales manually, RoB ratings are estimated to take between 30\u0026ndash;120 minutes to complete, whereas PEDro ratings might take between 10\u0026ndash;30 minutes to complete. Factors such as the complexity of trials, experience of the rater\u0026rsquo;s, and frequency of the need to consult formal manuals may all affect these estimates.\u003c/p\u003e \u003cp\u003eRecognising the effort and time involved in these manual evaluations, tools are now being developed to accelerate and automate the scoring and rating process. Some tools use Artificial Intelligence to perform the rating process. For example, RobotReviewer (Marshall et al. 2017) uses Machine Learning and Natural Language Processing (NLP) to extract data from RCTs and to calculate Risk of Bias. Recent studies (Armijo-Olivo et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Tian et al. \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) found that RobotReviewer performed moderately well relative to humans on the domains of \u003cem\u003eRandom Sequence Generation\u003c/em\u003e and \u003cem\u003eBlinding of Participants and Personnel\u003c/em\u003e, but that agreement was poor for \u003cem\u003eAllocation Concealment\u003c/em\u003e and \u003cem\u003eBlinding of Outcomes Assessors\u003c/em\u003e and both reviews concluded that the tool was not a suitable replacement for human raters.\u003c/p\u003e \u003cp\u003eAlternatively, algorithm-based approaches use scoring rules to calculate item and total scores and can be used after data have been manually extracted. We focus on CogTale (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.cogtale.com\u003c/a\u003e\u003c/span\u003e\u003cspan address=\"http://www.cogtale.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), a comprehensive online repository and platform designed for the evaluation, synthesis, and dissemination of evidence from cognition-oriented treatments (COTs) studies in the older adult population (Sabates et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). The CogTale database provides detailed information on the design and methodology of intervention studies based on extensive data extraction by trained human reviewers. Using an algorithm-based approach, CogTale calculates several methodological quality scores, including the PEDro and Cochrane Risk of Bias (RoB) scores. The CogTale platform includes numerous other features, most importantly, the ability to rapidly conduct a meta-analysis of studies included in the platform (for more details, see Sabates et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2021\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eCogTale offers an alternative avenue for expediting systematic reviews and meta-analyses in the field of COTs in older adults. Although AI-based approaches to data extraction on the CogTale platform have not yet reached acceptable standards (Rasool et al. \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), the algorithm-based approach it employs significantly speeds up the calculation of methodological quality scores. This reduces the time needed to produce systematic reviews and meta-analyses which in turn has the potential to reduce the previously mentioned translational time lag, allowing evidence-based practice to be implemented more rapidly and ensuring that systematic reviews and meta-analyses remain up to date.\u003c/p\u003e \u003cp\u003eAs CogTale\u0026rsquo;s algorithm-based approach has not yet been formally evaluated, the aim of the current study is to evaluate the level of agreement between PEDro and RoB scores produced by CogTale and those produced manually.\u003c/p\u003e"},{"header":"2. Method","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Study Selection\u003c/h2\u003e \u003cp\u003eA convenience sample of randomised controlled trials (RCTs) was selected to evaluate the effects of COTs in older adults. Studies were chosen based on two separate criteria: (1) availability of PEDro scores on CogTale and/or NeuroBITE or (2) presence of Cochrane risk of bias (RoB) scores in both CogTale and Cochrane meta-analyses. It was not necessary for studies to meet both criteria to be included. This selection process resulted in two datasets: 37 studies with Pedro scores and 37 studies with RoB scores. Some studies met both criteria and thus appeared in both datasets, while others had only one type of score. The full dataset can be found in supplementary materials.\u003c/p\u003e \u003cp\u003eTo assess the adequacy of our sample, we conducted a post-hoc power analysis which indicated that our sample of 37 studies provided 80% power to detect correlations of \u0026ge;\u0026thinsp;0.4 (α\u0026thinsp;=\u0026thinsp;0.05), sufficient to detect agreement coefficients above chance levels. Representativeness was evaluated by comparing methodological quality metrics between our sample and the broader CogTale repository (N\u0026thinsp;=\u0026thinsp;399). Statistical comparison of PEDro scores showed no significant difference in distribution (Kolmogorov-Smirnov test, p\u0026thinsp;=\u0026thinsp;.135). For Risk of Bias assessments, our sample showed similar distributions in three of six domains. Additionally, all our ROB sample studies were included in published Cochrane meta-analyses focused on cognitive rehabilitation and training for people with mild to moderate dementia (Bahar-Fuchs et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Kudlicka et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), indicating they adequately represent the types of evidence synthesised in this field. Overall, our sample reflects the quality distribution of studies in the CogTale repository, which comprehensively catalogues cognition-oriented treatment research, making our selection appropriate for evaluating quality assessment tools in this specific research domain.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Methodological Quality Scores\u003c/h2\u003e \u003cp\u003eWe assessed methodological quality using both the RoB and PEDro scales. The RoB scores were sourced from CogTale and compared with scores from two relevant Cochrane meta-analyses (Bahar-Fuchs et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Kudlicka et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Similarly, the PEDro scores were obtained from CogTale and compared with those for the same studies published on the NeuroBITE platform. Importantly, although scoring of PEDro items on the CogTale platform was generally based on the PEDro manual (used by NeuroBITE), there are nonetheless several differences in how certain items are calculated, as shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003ePEDro Item Criteria\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePEDro Item\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSource\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eCriteria\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP1. Eligibility Criteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eInclusion and exclusion criteria specified as well as the recruitment source.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eSame criteria as CogTale.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP2 Random Allocation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eSubjects randomly allocated to interventions.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eSame criteria as CogTale.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP3. Allocation Concealment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003ePerson responsible for determining eligibility of participants unaware of participant intervention allocation, at the time of decision.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003ePerson responsible for determining eligibility of participants unaware of participant intervention allocation, at the time of decision, or,\u003c/p\u003e \u003cp\u003e\u003cb\u003eit is clear that participants were only allocated to groups after they were enrolled.\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP4. Baseline Comparability\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003eIntervention conditions were comparable at baseline with either no differences between conditions or \u003cb\u003edifferences found on no more than one demographic or outcome variable.\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eIntervention conditions were comparable (no statistically significant difference) at baseline for relevant outcomes\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP5. Participant Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eParticipants were blinded in the study\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eParticipants were blinded in the study; \u003cb\u003ecriteria is met even if blinding was not 100% successful\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP6. Therapist Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTherapists were blinded in the study\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTherapists blinded from group allocation; \u003cb\u003ecriteria is met even if the blinding was not 100% successful\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP7. Assessor Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAssessors were blinded in the study\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAssessors were blinded in the study; \u003cb\u003ecriteria is met even if the blinding was not 100% successful\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP8. Retention Rate\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe proportion of participants retained post treatment in the \u003cb\u003eboth the experimental and control group\u003c/b\u003e was above 85%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExplicitly stated that at least one post-treatment outcome obtained results from more than 85% of participants and was this explicitly stated\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP9. Intention to Treat\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eIntention to treat analyses was used or was the intervention delivered to each participant as intended\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eIntention to treat analyses was used or \u003cb\u003ewas it explicitly stated that post-treatment data has been analyses according to initial condition allocation\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP10. Between Group Analyses\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBetween-group statistical comparisons were reported for at least one outcome\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSame criteria as CogTale.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eP11. Statistics Reported\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eCogTale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eStudy reported both point measures and measures of variability for at least one outcome measure.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNeuroBITE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSame criteria as CogTale.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003e\u003cem\u003eNote.\u003c/em\u003e Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e provides a shorthand (e.g. P1. Eligibility Criteria) for each PEDro item and provides information on how CogTale and NeuroBITE each assess these items. Differences in construct definition between CogTale and NeuroBITE are highlighted in \u003cb\u003ebold.\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section3\"\u003e \u003ch2\u003e2.2.1 CogTale\u003c/h2\u003e \u003cp\u003eCogTale generated both PEDro and RoB scores through a systematic and algorithm-driven process. Data for trials included in this study was extracted by coders who were graduate students taking part in the CogTale Internship \u0026ndash; a program through which they received further training in understanding trial design, evaluation, and evidence synthesis. The CogTale data extraction form captures numerous design features necessary for quality assessment, but coders are unaware of or have access to the scoring algorithm, effectively blinding them to how their responses would impact the final scores. Further reducing bias, the lead investigators who developed CogTale were not involved in either the data extraction process or statistical analyses. The first author, who has no affiliation with CogTale, conducted an additional independent review on a subset of these extraction forms to ensure accuracy, and all data analysis was performed using anonymised datasets to prevent identification of specific studies during comparison.\u003c/p\u003e \u003cp\u003eCogTale's algorithm was then used to automatically calculate quality scores based on specific responses from the extraction form. For example, the PEDro item P1 (Eligibility Criteria) is determined by responses to two specific questions: \"\u003cem\u003eWas eligibility criteria specified?\u003c/em\u003e\" and \"\u003cem\u003eWas the source of participants specified?\u003c/em\u003e\". If both questions are answered \"yes,\" P1 receives a score of 1; otherwise, it scores 0. This systematic approach is applied consistently across all PEDro and RoB items, ensuring objectivity in the scoring process.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e \u003ch2\u003e2.2.2 NeuroBITE PEDro Scores\u003c/h2\u003e \u003cp\u003eNeuroBITE used the PEDro-P scale, which is a refinement of the original PEDro scale specifically designed to rate studies on NeuroBITE. The PEDro-P scale has adapted and re-worded the same 11 criteria from the original PEDro scale to ensure they apply unambiguously to trials on NeuroBITE. Reviewers are trained to complete manual PEDro ratings based on specific criteria for each item, as shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, though it is unclear how many raters scored the PEDro scale in NeuroBITE.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section3\"\u003e \u003ch2\u003e2.2.3 Cochrane Meta-Analyses RoB Scores\u003c/h2\u003e \u003cp\u003eStudies from Cochrane meta-analyses included in the current study were assessed using the Cochrane RoB 1 Tool (Higgins, 2011). The process involves a detailed manual evaluation of each study, where usually two trained reviewers independently assess and rate the risk of bias. After completing their individual assessments, the reviewers discuss any discrepancies to reach a consensus on the final scores, ensuring consistency and accuracy in the evaluation.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Statistical Analyses\u003c/h2\u003e \u003cp\u003eWe analysed the level of agreement between PEDro total scores using a Bland-Altman plot. This analysis provided critical insights into both the bias and variability between the two sets of scores. The average bias, represented by the mean difference, indicated the systematic difference between the two measurement tools, showing whether one consistently rated higher or lower than the other. The limits of agreement gave a measure of variability, showing the range within which most of the score differences lay. This is essential for understanding how much variation exists between the two scoring methods and assessing if the differences are clinically acceptable. Additionally, by plotting the differences against the average of the two scores, we were able to detect any potential trends, such as proportional bias, where the differences between the scores vary across the scale. This is crucial to determine whether there is consistent agreement across the full scoring range or if the agreement breaks down at higher or lower scores. Overall, the Bland-Altman plot offered a comprehensive understanding of both the extent of bias and the variability, allowing us to evaluate the level of agreement between the two PEDro total scores and their potential interchangeability.\u003c/p\u003e \u003cp\u003eTo assess the level of agreement between PEDro items, we used unweighted kappa, which measures the degree of agreement between two raters beyond what would be expected by chance. This statistic tells us how consistently the raters scored each item to determine whether there is meaningful agreement. However, due to the binary nature of the data, the kappa statistic is vulnerable to the kappa prevalence paradox, where high or low prevalence of one category can lead to a misleadingly low kappa value despite high observed agreement (Zec et al., \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). For example, if both raters consistently agree that an item is 'absent' for most cases, the kappa value may appear low, even though their agreement is strong.\u003c/p\u003e \u003cp\u003eTo address this limitation, we also employed Gwet\u0026rsquo;s AC1 and Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) as more robust metrics for assessing agreement. Gwet\u0026rsquo;s AC1 adjusts for the prevalence paradox by considering the probability of chance agreement differently, offering a more stable estimate of agreement across different prevalence rates (Wongpakaran et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). PABAK similarly adjusts for prevalence and bias by recalculating the kappa statistic to account for these factors, making it less sensitive to distributional imbalances in the data (Chen et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2009\u003c/span\u003e). Gwet\u0026rsquo;s AC1 and PABAK were reported as the primary statistics, as they offer a more accurate reflection of agreement in this context. Although Gwet\u0026rsquo;s AC1 and PABAK are not exact substitutes for unweighted kappa, they still provide valuable insights into the level of agreement between the two measures by offering complementary perspectives that mitigate the limitations of kappa. The inclusion of these metrics to accommodate for the kappa prevalence paradox is supported by their use in other studies (e.g, Boye et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), making this approach both robust and well-validated in agreement analyses.\u003c/p\u003e \u003cp\u003eTo assess the level of agreement between Risk of Bias items, we used weighted kappa since this provides harsher penalties for disagreements when studies were rated as either low or high risk of bias, as these discrepancies are more critical than those involving an unclear risk of bias. By using a squared weighting matrix, we assigned greater penalties to larger differences in ratings, ensuring that the level of disagreement between extreme ratings (low vs. high risk) was more heavily reflected in the final agreement statistic. This approach provided a more nuanced evaluation of agreement, considering the relative importance of different types of disagreements.\u003c/p\u003e \u003cp\u003eWe also reported Gwet\u0026rsquo;s AC1 to control for the kappa prevalence paradox, as the ordinal data only had three categories, making it vulnerable to this issue. Gwet\u0026rsquo;s AC1 used a similar weighting matrix as weighted kappa to reflect the importance of larger discrepancies. PABAK was not reported for Risk of Bias items, as it could not be weighted. Gwet\u0026rsquo;s AC1 will be reported as the primary statistic, providing a more stable estimate of agreement.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Level of Agreement Between PEDro Total Scores\u003c/h2\u003e \u003cp\u003eThe Bland-Altman plot (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) revealed the mean difference (bias) between the two measures was M\u0026thinsp;=\u0026thinsp;0.92 (95% CI: 0.36 to 1.38). The limits of agreement, defined as the mean difference\u0026thinsp;\u0026plusmn;\u0026thinsp;1.96 times the standard deviation of the differences, were \u003cb\u003e\u0026minus;\u003c/b\u003e\u0026thinsp;2.09 to 3.93. The plot also revealed greater variability at the extremes of the measurement range such that at both lower and higher means, the differences between the two measures are more pronounced and approach the limits of agreement. These results suggest that although the measures are generally in agreement, there is increased variability in the agreement at the extremes of the score range.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Level of Agreement Between PEDro Items\u003c/h2\u003e \u003cp\u003eRaw values for Gwet\u0026rsquo;s AC1 and PABAK for each of the 11 PEDro items are reported in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Overall, items P1, P2, and P11 demonstrated almost perfect agreement between raters, as indicated by Gwet\u0026rsquo;s AC1 values. Items P5, P6, P7, and P10 exhibited substantial agreement, while P3 and P4 showed moderate agreement. Additionally, item P8 had slight agreement, and P9 showed poor agreement, suggesting substantial variability in rater judgments for these items.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003ePEDro Item Level of Agreement Statistics\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePEDro Item\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePA\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUK\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGwet AC1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePABAK\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP1. Eligibility Criteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e84.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.16 (.280)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.81 (\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP2. Random Allocation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e94.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.77 (\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.93 (\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP3. Allocation Concealment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e70.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.32 (.040)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.48 (.003)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.41\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP4. Baseline Comparability\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e68.59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.22 (.112)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.46(.005)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.35\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP5. Participant Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e81.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0(1.000)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.77(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP6. Therapist Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e81.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-0.08(0.565)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.77(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP7. Assessor Blinding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e89.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.78(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.79(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.78\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP8. Retention Rate\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e56.76\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.13(.420)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.15 (.378)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.14\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP9. Intention to Treat\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e40.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.08 (.379)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-0.19 (.258)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-0.19\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP10. Between-Group Analyses\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e83.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.17(.295)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.80 (\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.67\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eP11. Statistics Reported\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e83.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-0.04(.689)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.81(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003e\u003cem\u003eNote.\u003c/em\u003e PA\u0026thinsp;=\u0026thinsp;Percentage Agreement; UK\u0026thinsp;=\u0026thinsp;Unweighted Kappa; PABAK\u0026thinsp;=\u0026thinsp;Prevalence Adjusted and Bias Adjusted Kappa. The table provides raw statistics for the level of agreement between CogTale and NeuroBITE PEDro items. Both UK and Gwet\u0026rsquo;s AC1 are presented with their corresponding coefficients and p-values.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Level of Agreement Between Risk of Bias Items\u003c/h2\u003e \u003cp\u003eRaw statistics for Gwet\u0026rsquo;s AC1 are reported in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. The results show that random sequence generation, allocation concealment, and detection bias had substantial levels of agreement, while performance bias, attrition bias, and reporting bias demonstrated fair levels of agreement.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eLevel of Agreement Between Risk of Bias Items for CogTale and Cochrane Meta-Analyses\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItem\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePA\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eWK\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGwet AC1\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRandom Sequence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e67.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.40(.011)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.69(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAllocation Concealment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e62.16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.422(.002)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.68(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDetection Bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e64.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.363(.008)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.68(\u0026lt;\u0026thinsp;.001)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePerformance Bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e29.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026minus;\u0026thinsp;.087(.302)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.35(.003)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAttrition Bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e21.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.082(.410)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.26(.002)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eReporting Bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e45.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026minus;\u0026thinsp;.020(.888)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.35(.022)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003e\u003cem\u003eNote.\u003c/em\u003e PA\u0026thinsp;=\u0026thinsp;Percentage Agreement; WK\u0026thinsp;=\u0026thinsp;Weighted Kappa. The table provides raw statistics for the level of agreement between CogTale and Cochrane Risk of Bias items. Both UK and Gwet\u0026rsquo;s AC1 are presented with their corresponding coefficients and p-values.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThe aim of this study was to investigate how well the CogTale PEDro and Risk of Bias scores align with manual scoring of these indices.\u003c/p\u003e \u003cp\u003eThe Bland-Altman analysis shows that, on average, CogTale produced slightly higher PEDro total scores compared to NeuroBITE. However, the relatively small bias suggests that the two tools are generally aligned when assessing PEDro scores. Despite this general alignment, there is substantial variability in the individual differences between the tools' scores, as indicated by the wide LoA. The variability is especially pronounced at both the lower and higher ends of the score range, where differences between CogTale and NeuroBITE are more substantial. In contrast, the middle range of scores shows tighter clustering of differences, suggesting better consistency between the tools in this range. This pattern of variability implies that while the tools show good agreement on average, they may not be interchangeable across the entire range of scores. Particularly, users should be cautious when interpreting CogTale scores at the extreme ends of the score spectrum, as the differences between the tools become more pronounced in these regions.\u003c/p\u003e \u003cp\u003eTo further investigate the underlying causes of these discrepancies, we conducted an item-level analysis to assess the level of agreement between CogTale and NeuroBITE for PEDro scores. While the majority of items showed substantial to almost perfect agreement, items P3, P4, P8, and P9 displayed only slight to moderate agreement, contributing significantly to the discrepancies in total scores.\u003c/p\u003e \u003cp\u003eA critical examination revealed that these discrepancies stem from how the CogTale algorithm translates abstract quality concepts into concrete decision rules, creating different operational definitions. For instance, in P9 (\"Intention to Treat\"), CogTale accepts either the use of intention to treat or evidence that the intervention was delivered as allocated, whereas NeuroBITE requires explicit mention of intention to treat or clear statement that post-treatment data was analysed according to initial allocations. This definitional difference results in CogTale providing more lenient ratings for P9, as our data confirms. Conversely, for P3 ('Allocation Concealment'), CogTale applies stricter standards, requiring that randomisation outcomes were concealed until intervention commencement, while NeuroBITE accepts studies where participants were assigned to groups after enrolment without explicit mention of concealment methods.\u003c/p\u003e \u003cp\u003eSimilar definitional differences exist in the algorithmic operationalisation of P4 ('Baseline Comparability') and P8 ('Retention Rate'), creating subtle differences in construct definitions that led to either more lenient or stricter CogTale ratings compared to NeuroBITE\u0026mdash;patterns consistently observed in our data. These inconsistencies have significant implications: either the CogTale algorithm needs revision to better align with standard PEDro definitions, or alternatively, CogTale's PEDro scores should be considered a modified version of the scale rather than directly interchangeable with NeuroBITE.\u003c/p\u003e \u003cp\u003eThe level of agreement was substantial on three Risk of Bias domains (e.g., Random Sequence, Allocation Concealment, and Detection Bias), but was only fair for the other three domains (e.g., performance, attrition, and reporting bias). These differences, however, are not necessarily indicative of invalid assessments by CogTale but are more likely to reflect conceptual differences in how the items are evaluated. CogTale's algorithm focuses on identifying whether certain risk of bias criteria is present in a study, while Cochrane meta-analyses take an additional step by evaluating the potential impact of these criteria on the study's outcomes to determine the risk of bias (Higgins, 2011). This added layer of subjective expert interpretation in Cochrane\u0026rsquo;s approach, absent in CogTale, likely accounts for the discrepancies between the two tools in risk of bias assessments.\u003c/p\u003e \u003cp\u003eFor example, when assessing performance bias, both Cochrane meta-analyses and CogTale examine whether the participants were blinded. However, Cochrane also considers whether the lack of blinding would likely bias the results, while CogTale does not assess the likelihood of this bias occurring, suggesting a systematic difference between the tools. In cognitive interventions, the blinding of participants may not always lead to performance bias if the intervention relies on self-driven cognitive tasks, where awareness of the intervention's purpose is less likely to influence outcomes. In contrast, blinding may be crucial in studies where participant expectations can strongly influence results, such as pharmacological interventions. These interpretation differences suggest that the poor agreement is likely due to systematic error arising from how each construct is assessed, rather than random error. Although the more nuanced approach required for the assessment of Risk of Bias in Cochrane Reviews is useful when it comes to drawing a distinction between threats to bias and actual bias, the approach requires greater subject matter expertise, and given the duplicate nature of RoB assessments in Cochrane Reviews, may also increase the frequency of inter-rater discrepancies, owing in part to differences in rater\u0026rsquo;s experience. The algorithm-based approach implemented in the CogTale platform, in contrast, is not influenced by subjective factors and in that regard is likely to yield more consistent scores.\u003c/p\u003e \u003cp\u003eImportantly, the algorithm-based approach to RoB ratings in the CogTale Platform performed better overall than current AI/NLP-based approaches such as RobotReviewer. While RobotReviewer showed moderate agreement with human raters for Random Sequence Generation and Blinding of Participants and Personnel, it exhibited poor agreement for Allocation Concealment and Blinding of Outcome Assessors (Tian et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Conversely, CogTale demonstrated higher agreement across these domains, performing especially well in areas where RobotReviewer encountered difficulties, such as Allocation Concealment. These performance differences likely stem from fundamental methodological distinctions between the systems. RobotReviewer employs a fully automated AI and machine learning approach that analyses full-text articles to generate risk classifications without human intervention (Tian et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). In contrast, CogTale uses a semi-automated approach where human reviewers manually extract data, which is then processed through structured questions and pre-defined decision rules and without incremental improvement/adaptation of the model. This human-in-the-loop design allows CogTale to capture key methodological elements more systematically than fully automated systems while potentially achieving greater consistency than fully manual human ratings. These results suggest that CogTale's algorithm-based system offers a more suitable alternative for accelerating the Risk of Bias rating process, addressing the shortcomings seen in previous AI-driven approaches like RobotReviewer.\u003c/p\u003e \u003cp\u003eThe findings of this study have important implications for the validity and broader application of CogTale\u0026rsquo;s algorithm based PEDro and Risk of Bias scores. While systematic differences in the assessment criteria between CogTale and manual scoring methods like NeuroBITE and Cochrane were observed, these discrepancies are primarily attributable to variations in evaluation approaches, rather than a lack of validity in the CogTale system. The overall consistency of scores, particularly for the PEDro scale, supports the accuracy and utility of CogTale\u0026rsquo;s automated assessments. Although certain items\u0026mdash;such as P3 (Allocation Concealment) and P9 (Intention to Treat)\u0026mdash;revealed divergent scoring tendencies due to differences in evaluation criteria, the observed patterns reflect expected systematic variations in criteria, rather than random error. Thus, while CogTale's scores should be regarded as a modified version of traditional manual assessments, the high level of agreement for most items reinforces the tool\u0026rsquo;s potential to serve as a valid and reliable alternative for methodological evaluation.\u003c/p\u003e \u003cp\u003eThese findings also highlight the potential of CogTale to substantially expedite the meta-analysis process by automating the traditionally time-consuming methodological assessment of studies. Manual scoring of PEDro and Risk of Bias items is often laborious, contributing to delays in the synthesis and dissemination of research findings. CogTale\u0026rsquo;s algorithm-based approach offers a streamlined solution, enabling faster generation of methodological scores without significant loss of validity. This increased efficiency is particularly relevant for systematic reviews and meta-analyses, where methodological assessments can represent a bottleneck. By reducing the time required for these evaluations, CogTale has the potential to accelerate evidence synthesis, ensuring that systematic reviews remain timely and that clinical guidelines can be updated more rapidly. Ultimately, the integration of CogTale into evidence synthesis workflows in the field of COTs in ageing and dementia, could help bridge the gap between research and practice, facilitating the timely implementation of evidence-based interventions in clinical settings and improving the overall responsiveness of healthcare to emerging research.\u003c/p\u003e \u003cp\u003eDespite the important findings of this study, several limitations must be considered. One limitation of this study is the relatively small sample size of studies included in the analysis, which may limit the generalisability of the findings. This limitation arises from the restricted availability of studies that are accessible with Risk of Bias and/or PEDro scores on both the CogTale and NeuroBITE or Cochrane platforms. As a result, the comparisons made between the scoring systems might not fully capture the broader range of possible variations across different types of studies. This could mean that the observed differences between the tools are more reflective of the specific set of studies analysed rather than being indicative of the tools\u0026rsquo; performance across all possible contexts. As evidence keeps being added to the CogTale platform, these analyses can be repeated with larger datasets.\u003c/p\u003e \u003cp\u003eAnother limitation of this study is its focus on score alignment without considering how discrepancies in scoring between CogTale and NeuroBITE might influence the outcomes of systematic reviews and meta-analyses. Since CogTale incorporates PEDro scores as part of its overall methodological grading, the slightly more lenient scoring by CogTale could potentially lead to more favourable evaluations of study quality in these analyses. This leniency might result in higher methodological ratings for some studies, affecting the subsequent grading of the evidence and conclusions drawn from meta-analyses. Work is currently underway to evaluate the extent to which meta-analytic studies conducted in CogTale broadly replicates the findings from published meta-analyses (Hiskens-Raven, \u003cem\u003eIn Preparation\u003c/em\u003e), and this will further clarify this issue. It is important to note, however, that PEDro scores are just one of several factors contributing to CogTale\u0026rsquo;s overall grading framework. Therefore, while CogTale may apply slightly more lenient scoring in this domain, the broader grading process remains comprehensive, reducing the likelihood that this leniency will substantially bias the overall quality assessments.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll data used during this study is included in the Supplementary Information files.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eABF, BMH, SB, and SSS are the scientific developers of the CogTale platform, which was evaluated in this study. The first author (OA) and other co-authors (CC, IB) have no competing interests to declare.\u0026quot;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFunding in support of this project was provided by NIH-NIA R35 Award (to Benjamin Hampstead).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOA led the analysis and manuscript preparation. CC assisted with analyses, writing the manuscript preparation. IS assisted with the acquisition of the data. \u0026nbsp;BMH, SB and SSS all contributed to the development of the CogTale database and manuscript preparation. ABF designed the study, led the development of the CogTale database, and contributed to manuscript preparation. All authors have reviewed and approved the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe CogTale team would like to acknowledge the contributions of a range of trainee coders through the CogTale internship who have assisted with establishing and growing the CogTale database.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eArmijo-Olivo S, Craig R, Campbell S. Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials. Res Synthesis Methods. 2020;11(3):484\u0026ndash;93. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/jrsm.1398\u003c/span\u003e\u003cspan address=\"10.1002/jrsm.1398\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBahar-Fuchs A, Martyr A, Goh AM, Sabates J, Clare L. Cognitive training for people with mild to moderate dementia. Cochrane Libr. 2019. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.cd013069.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.cd013069.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBoye KS, Matza LS, Currie BM, Coyne KS. Validity and analysis of the Diabetes Injection Device Preference Questionnaire (DID-PQ). J Patient-Reported Outcomes. 2020;4(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s41687-020-00266-x\u003c/span\u003e\u003cspan address=\"10.1186/s41687-020-00266-x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBurns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in Evidence-Based Medicine. Plast Reconstr Surg. 2011;128(1):305\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1097/prs.0b013e318219c171\u003c/span\u003e\u003cspan address=\"10.1097/prs.0b013e318219c171\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCashin AG, McAuley JH. Clinimetrics: Physiotherapy Evidence Database (PEDRO) scale. J Physiotherapy. 2019;66(1):59. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jphys.2019.08.005\u003c/span\u003e\u003cspan address=\"10.1016/j.jphys.2019.08.005\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol. 2009;9(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/1471-2288-9-5\u003c/span\u003e\u003cspan address=\"10.1186/1471-2288-9-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u003cem\u003eCochrane Evidence Synthesis and Methods\u003c/em\u003e. (2024). Cochrane. Retrieved September 21, 2024, from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cochrane.org/\u003c/span\u003e\u003cspan address=\"https://www.cochrane.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Morton NA. The PEDro scale is a valid measure of the methodological quality of clinical trials: a demographic study. Australian J Physiotherapy. 2009;55(2):129\u0026ndash;33. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/s0004-9514(09)70043-1\u003c/span\u003e\u003cspan address=\"10.1016/s0004-9514(09)70043-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGaneshkumar P, Gopalakrishnan S. Systematic reviews and meta-analysis: Understanding the best evidence in primary healthcare. J Family Med Prim Care. 2013;2(1):9. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.4103/2249-4863.109934\u003c/span\u003e\u003cspan address=\"10.4103/2249-4863.109934\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHiggins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne J. The Cochrane Collaboration\u0026rsquo;s tool for assessing risk of bias in randomised trials. BMJ. 2011;343(oct18 2):d5928. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmj.d5928\u003c/span\u003e\u003cspan address=\"10.1136/bmj.d5928\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. a. C.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKudlicka A, Martyr A, Bahar-Fuchs A, Sabates J, Woods B, Clare L. Cognitive rehabilitation for people with mild to moderate dementia. Cochrane Libr. 2023;2023(6). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.cd013388.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.cd013388.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMorris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510\u0026ndash;20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1258/jrsm.2011.110180\u003c/span\u003e\u003cspan address=\"10.1258/jrsm.2011.110180\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarshall IJ, Kuiper J, Banner E, Wallace BC. (2017, July). Automating biomedical evidence synthesis: RobotReviewer. \u003cem\u003eProceedings of the Conference of the Association for Computational Linguistics Meeting\u003c/em\u003e, 2017, 7\u0026ndash;12. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.18653/v1/P17-4002\u003c/span\u003e\u003cspan address=\"10.18653/v1/P17-4002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMyung S. How to review and assess a systematic review and meta-analysis article: a methodological study (secondary publication). J Educational Evaluation Health Professions. 2023;20:24. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3352/jeehp.2023.20.24\u003c/span\u003e\u003cspan address=\"10.3352/jeehp.2023.20.24\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u003cem\u003eNeuroRehab Evidence Resource\u003c/em\u003e. (2024, September 16). NeuroBITE. Retrieved September 21, 2024, from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://neurorehab-evidence.com/web/cms/content/home\u003c/span\u003e\u003cspan address=\"https://neurorehab-evidence.com/web/cms/content/home\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRasool Z, Kurniawan S, Balugo S, Barnett S, Vasa R, Chesser C, Bahar-Fuchs A. Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset. Nat Lang Process J. 2024;100083. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.nlp.2024.100083\u003c/span\u003e\u003cspan address=\"10.1016/j.nlp.2024.100083\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSabates J, Belleville S, Castellani M, Dwolatzky T, Hampstead BM, Lampit A, Simon S, Anstey K, Goodenough B, Mancuso S, Marques D, Sinnott R, Bahar-Fuchs A. CogTale: an online platform for the evaluation, synthesis, and dissemination of evidence from cognitive interventions studies. Syst Reviews. 2021;10(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13643-021-01787-2\u003c/span\u003e\u003cspan address=\"10.1186/s13643-021-01787-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTian Y, Yang X, Doi SA, Furuya-Kanamori L, Lin L, Kwong JS, Xu C. Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans. Res Synthesis Methods. 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/jrsm.1761\u003c/span\u003e\u003cspan address=\"10.1002/jrsm.1761\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eT\u0026oacute;th B, Berek L, Gul\u0026aacute;csi L, P\u0026eacute;ntek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Reviews. 2024;13(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13643-024-02592-3\u003c/span\u003e\u003cspan address=\"10.1186/s13643-024-02592-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen\u0026rsquo;s Kappa and Gwet\u0026rsquo;s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/1471-2288-13-61\u003c/span\u003e\u003cspan address=\"10.1186/1471-2288-13-61\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZec S, Soriani N, Comoretto R, Baldi I. High agreement and high prevalence: the paradox of Cohen\u0026rsquo;s Kappa. Open Nurs J. 2017;11(1):211\u0026ndash;8. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2174/1874434601711010211\u003c/span\u003e\u003cspan address=\"10.2174/1874434601711010211\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"systematic-reviews","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sysr","sideBox":"Learn more about [Systematic Reviews](http://systematicreviewsjournal.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/sysr/default.aspx","title":"Systematic Reviews","twitterHandle":"@MedicalEvidence","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Systematic reviews, meta-analysis, methodological quality, PEDro scale, Risk of Bias (RoB), algorithm-based scoring, CogTale platform","lastPublishedDoi":"10.21203/rs.3.rs-5486560/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5486560/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cb\u003eBackground\u003c/b\u003e\u003c/p\u003e \u003cp\u003eEvidence hierarchies guide evidence-based practice by ranking forms of evidence to support translation and clinical decision-making. Systematic reviews and meta-analyses (SRMAs) represent the highest form of evidence but are time and resource-intensive, contributing to the estimated 17-year lag in the translation of evidence into practice. Tools that automate aspects of the systematic review processes aim to shorten this gap. Specifically, algorithm-based evaluation of study quality, as performed in the CogTale evidence synthesis platform, accelerates such processes relative to manual methods, leading to a more rapid synthesis of the evidence. In this study, we assessed the agreement between CogTale\u0026rsquo;s algorithm-based scoring of the PEDro and RoB scales with manual scoring of these scales.\u003c/p\u003e\u003cp\u003e\u003cb\u003eMethods\u003c/b\u003e\u003c/p\u003e \u003cp\u003eWe selected 37 randomised controlled trials (RCTs) with PEDro scores available on the NeuroBITE Platform and 37 trials with Risk of Bias (RoB) scores available in Cochrane meta-analyses. Agreement for individual PEDro and RoB items was evaluated using Gwet\u0026rsquo;s AC1, while a Bland-Altman plot assessed total PEDro score agreement.\u003c/p\u003e\u003cp\u003e\u003cb\u003eResults\u003c/b\u003e\u003c/p\u003e \u003cp\u003eThe Bland-Altman analysis showed an average difference in PEDro scores of 0.92 between CogTale and NeuroBITE, with limits of agreement from \u0026minus;\u0026thinsp;2.09 to 3.93. Gwet's AC1 revealed almost perfect agreement for PEDro items P1, P2, and P11; substantial agreement for P5, P6, P7, and P10; moderate agreement for P3 and P4; slight agreement for P8; and poor agreement for P9. For RoB domains, substantial agreement was found for random sequence generation, allocation concealment, and detection bias, with fair agreement in other domains.\u003c/p\u003e\u003cp\u003e\u003cb\u003eConclusions\u003c/b\u003e\u003c/p\u003e \u003cp\u003eOverall, CogTale's algorithm-based PEDro and RoB scores align well with manual scores, despite some discrepancies in specific PEDro (P3, P8, P9) and RoB items, likely due to systematic scoring criteria variations. CogTale shows promise in automating quality assessments, potentially reducing time for evidence synthesis while maintaining accuracy. Future research should address key limitations by examining how scoring differences impact meta-analytic outcomes and evaluate CogTale's performance with larger datasets as more evidence accumulates on the platform.\u003c/p\u003e","manuscriptTitle":"The Comparability of Manual vs. Algorithm-Based Calculation of Clinical Trial Methodological Quality Indices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-15 10:37:16","doi":"10.21203/rs.3.rs-5486560/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2025-04-16T21:36:36+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-04-13T02:35:23+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-04-02T07:10:03+00:00","index":"","fulltext":""},{"type":"submitted","content":"Systematic Reviews","date":"2025-04-02T02:33:27+00:00","index":"","fulltext":""},{"type":"decision","content":"Minor revision","date":"2025-03-10T20:30:31+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"systematic-reviews","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sysr","sideBox":"Learn more about [Systematic Reviews](http://systematicreviewsjournal.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/sysr/default.aspx","title":"Systematic Reviews","twitterHandle":"@MedicalEvidence","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b88725b2-bbe3-4d1a-92ed-6ce393c9b495","owner":[],"postedDate":"April 15th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-05-07T09:26:26+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-15 10:37:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5486560","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5486560","identity":"rs-5486560","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}