Automated Evidence Synthesis of Cognitive Interventions in Older People using the CogTale Platform: A Validation Study

Automated Evidence Synthesis of Cognitive Interventions in Older People using the CogTale Platform: A Validation Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automated Evidence Synthesis of Cognitive Interventions in Older People using the CogTale Platform: A Validation Study Courtney Chesser, Storm Hiskens-Ravest, Isabelle J.M. Burke, Jamie Charlton, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7078540/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Advancing evidence-based medicine for aging-related interventions requires efficient and reliable methods for synthesising research findings. This study evaluates CogTale, an online platform designed to streamline evidence synthesis for cognition-oriented treatments (COTs) in older adults, by automating key aspects of meta-analysis. The platform’s performance was validated by replicating findings from three Cochrane Reviews of Cognition-oriented treatments (COTs) for older adults, cognitive training (CT), cognitive stimulation therapy (CST), and cognitive rehabilitation (CR). Key outcomes from the primary “Summary of Findings” tables were compared based on effect direction, confidence intervals, effect size magnitudes, and evidence certainty. Of the 72 studies analysed in the Cochrane Reviews, 61 were included in CogTale’s database. Using a lenient replication threshold, CogTale approximately replicated 16 of 18 outcomes (88.9%), while 11 outcomes (61.1%) met a more stringent threshold. Replication was most consistent for CST outcomes, with greater variability observed in CT and CR results due to data availability and methodological differences. These findings suggest that CogTale can approximately replicate high-quality SRMA results, particularly in CST, and demonstrate its potential as a scalable tool for aging research. CogTale demonstrates the potential to enhance efficiency and accessibility in evidence synthesis for aging-related interventions, offering researchers, clinicians, and policymakers a powerful tool for supporting evidence-based decision-making in dementia care. Further refinements are needed to optimise accuracy, particularly in addressing methodological discrepancies and ensuring data completeness. cognition-oriented treatments cognitive training evidence synthesis automation dementia meta-analysis Figures Figure 1 Figure 2 Figure 3 1. Introduction Cognitive decline is a hallmark of aging and poses a significant challenge to maintaining quality of life in older adults. Cognition-oriented treatments (COTs) address the cognitive and functional impairments commonly seen in aging populations. Interventions like cognitive stimulation therapy (CST), cognitive training (CT), and cognitive rehabilitation (CR) hold promise for mitigating age-related cognitive decline and enhancing functional capacity, making them critical areas of focus within older adult research—the interdisciplinary field exploring the biological mechanisms of aging and age-related diseases. CST is a structured and manualised intervention focused on general cognitive stimulation and discussion of various topics in the context of activities, usually in a group (Bahar-Fuchs et al., 2013 ; (Woods et al., 2023 ). CT focuses on the formal training of cognitive abilities (e.g., memory, attention) through practice on tasks (often game-like) or a strategy-based approach (Bahar-Fuchs et al., 2014 ; (Bahar-Fuchs et al., 2019 ). CR is a goal-oriented, enablement-based approach using various strategies to improve performance on personally relevant functional goals (Clare, 2017 ; (Clare & Woods, 2004 ; (Kudlicka et al., 2023 ). These interventions aim to improve cognitive performance and contribute to overall well-being. However, the effectiveness of these interventions often depends on synthesising evidence across diverse studies to identify robust patterns and guide clinical application. Systematic reviews and meta-analyses (SRMAs) serve this critical role. SRMAs represent the highest standard for synthesising evidence, forming the foundation of evidence-based medicine (EBM). EBM integrates clinical expertise with rigorous research to inform patient care decisions effectively (Evans, 2003 ; (Sackett, 1997 ). Organisations like the Cochrane Collaboration exemplify EBM principles by producing rigorously compiled SRMAs that support clinical decision-making (Tanjong-Ghogomu et al., 2009 ). SRMAs synthesise evidence in a highly structured manner, offering insights beyond individual studies, making them critical for clinical decision-making, guideline development, and evidence appraisal (García-Perdomo, 2016 ; (Paul & Leibovici, 2014 ). Despite their importance, SRMAs are complex, labour-intensive, and time-consuming (Marshall et al., 2018 ). Nearly a quarter of SRMAs become outdated within two years of publication (Shojania et al., 2007 ). Therefore, accelerating the SRMA production process via automation of key steps is essential to effectively maintain pace with the burgeoning body of academic literature (van Dinter et al., 2021 ). Christopoulou (Christopoulou, 2023 ) highlights areas of advancement in the automation of SRMAs, particularly in tasks such as study selection and data extraction, with tools like the Systematic Review Toolbox demonstrating effectiveness in supporting evidence synthesis (Johnson et al., 2022 ). However, fully automated meta-analysis tools, especially for data synthesis and interpretation, are still nascent and require further development and validation. One such automation effort is the development of CogTale, an online platform capable of streamlining the evidence synthesis process (Sabates et al., 2021 ). CogTale is a comprehensive database and repository of trial evidence on the effects of structured cognition-oriented treatments (COTs) for older adults, including cognitive stimulation therapy, cognitive training, and cognitive rehabilitation (Sabates et al., 2021 ). It aims to provide researchers, clinicians and consumers with clear and unbiased evidence-based information to help answer questions about COTs, including through evaluation of the methodology and findings from single studies, the production of plain language summaries of evidence (i.e., “Citizen Briefings”), and the rapid synthesis of the evidence through meta-analyses. Users can refine their searches of the database using various filters related to the intervention (e.g., specific type and ingredients, dose, duration, etc.), the population (e.g., older adults with dementia, mild cognitive impairment, etc.), study design (e.g., type of control, type of randomisation, outcomes assessed, etc.), analysis of results, and methodological quality scores among others, allowing for highly curated queries. The platform generates detailed reports sent to the user based on their meta-analytic query. Importantly, the meta-analytic functionality on CogTale awaits validation and the extent to which rapid meta-analytic queries produced on the CogTale platform can replicate the findings from rigorous and high-quality SMRAs, such as those published by the Cochrane Database of Systematic Reviews, remains to be established. Accordingly, the current study aims to evaluate the extent to which the CogTale platform can replicate the findings from three Cochrane Reviews focused on key COTs, namely, CST (Woods et al., 2023 ), CT (Bahar-Fuchs et al., 2019 ), and CR (Kudlicka et al., 2023 ). Table 1 details the methodological features and main findings of three recent Cochrane Reviews of CST CT, and CR. [Insert Table 1 ] In discussing issues related to scientific replication, Hedges (Hedges, 2019 ) and Hedges and Schauer (Hedges & Schauer, 2019 ) propose a distinction between an exact replication , in which findings (e.g., effect estimates) are identical in the replication studies, and an approximate replication , which is more lenient and allows for minor differences. They further distinguish formulating a null hypothesis according to which replication has not occurred (burden of proof is demonstrating replication) and formulating a null hypothesis according to which replication has occurred (burden of proof is demonstrating replication has not occurred) In attempting to replicate the findings from SMRAs, exact replication appears unrealistic when different methodologies are used, particularly when variations in the included set of studies or data availability are present. Accordingly, the null hypothesis that the replication was successful is appropriate in the current study, in which the goal is to assess whether the meta-analytic findings produced by CogTale are broadly consistent with those of published Cochrane Reviews. We expect the findings from CogTale to approximately replicate those of the Cochrane reviews. In practical terms, we hypothesise that the findings from CogTale will not be substantially different from those of recently published Cochrane Reviews of COTs. Specifically, concerning the key/main outcomes in each of the three reviews, we hypothesise that the direction (favouring experimental vs. control), strength (small, medium, large), and levels of certainty in the evidence (high, moderate, low, very low) in the effect estimates will be similar when evaluated on the CogTale platform. Furthermore, we hypothesise that the absolute value of effect estimates generated by the CogTale platform will not differ significantly from the absolute value of effect estimates found in the three Cochrane Reviews. 2. Methods 2.1. Replication Sample The replication sample comprised effect estimates reported in three Cochrane Reviews of COTs for older adults with dementia (Bahar-Fuchs et al., 2019 ; (Kudlicka et al., 2023 ; (Woods et al., 2023 ). These reviews were selected as they are each regarded as the most authoritative and methodologically rigorous SMRAs of the key COT approaches for people with dementia, and all primary studies included in these reviews were either already included or eligible for inclusion on the CogTale platform. Because of the large number of outcomes included in these reviews, the current replication effort focused on reproducing the results of the first "Summary of Findings" (SoF) table in each Cochrane review, comparing the relevant treatment (CST, CT, or CR) to control conditions on up to seven key treatment outcomes immediately at the end of the treatment period, and including medium-term outcomes for CR. For each outcome eligible for inclusion in the replication, we extracted the number of participants and studies, standardised mean differences (SMDs) and 95% confidence intervals (CIs) of SMD, study weight, and GRADE certainty in the evidence. SMDs were then converted to Hedges' g according to guidance provided by Hedges (Hedges, 1981 ). After extracting relevant outcomes and metrics, we mapped outcomes to predefined outcome categories within CogTale's classification system to determine whether the effect estimate could be produced on the CogTale platform. Importantly, the a priori classification system used in the CogTale platform ensures consistency in outcome classification, supporting systematic comparisons. 2.2. CogTale data processing flow and preparation for replication The data processing workflow on the CogTale platform involves manual and automated steps. Detailed descriptions of the CogTale information processing pipeline, statistical methodologies, and evidence grading procedures are described elsewhere (Sabates et al., 2021 ) and summarised in the Swim Lane Diagram in Fig. 1. Briefly, trained human coders manually completed data extraction from eligible studies using CogTale’s comprehensive data extraction form, which covers detailed information about the study design, including measures and outcomes, characteristics of experimental and control interventions, and study results. Methodological quality indices, including scores on the PEDro scale (Maher et al., 2003 ), the Jadad scale (Jadad et al., 1996 ), and the Cochrane Risk of Bias tool (ROB1.0) (Higgins et al., 2019 ; (Higgins et al., 2011 ) are then automatically computed based on a custom automated scoring algorithm. Where relevant raw data for an outcome are provided in manuscripts undergoing data extraction (e.g., sample size ( n ), means, and standard deviations (SDs)), the platform automatically calculates the Hedges’ g as a measure of effect estimate. When the meta-analysis function in the CogTale platform is used, variance-weighted estimates are pooled and corrected. A random-effects meta-analysis of the selected studies is carried out using restricted maximum likelihood (REML) (Langan et al., 2019 ) and the Hartung-Knapp-Sidik-Jonkman (HKSJ) method for CIs (Hartung & Knapp, 2001 ; (Sidik & Jonkman, 2007 ). Findings of meta-analytic queries are provided in reports generated via R Markdown, summarising for each broad and specific outcome the effect estimate and methodological quality of primary studies, the pooled treatment estimates with heterogeneity and significance testing, and the certainty of the evidence. [Insert Fig. 1] 2.3. Replication Anslysis The feasibility of replication within CogTale was assessed based on the results reported in each article and the existing outcome classification categories in CogTale. Replicability was determined by reviewing the individual studies included in the analyses for each outcome presented in the Cochrane Summary of Findings (SoF) tables. To be included in the replication, studies had to report raw values (Means, SDs) for the outcome of the experimental and control conditions at minimum at baseline and post-treatment. Studies were excluded from the replication analysis if they reported data unsupported by the CogTale platform, such as reporting change scores without baseline data or missing means and standard deviations. Once data extraction was completed, a meta-analysis of each included outcome using the CogTale platform was conducted. Key information was extracted, including the number of participants and studies, Hedges' g values and their CIs, study weights, and certainty in the evidence from the reports automatically generated by the platform. Comparatively, Cochrane reviews utilise the GRADE framework (Schünemann et al., 2019 ) to assess the certainty in the evidence and apply standardized tools such as RevMan and random-effects models to synthesize data, often incorporating methods like the HKSJ approach for handling between-study heterogeneity The replication analysis adhered to the approximate replication framework proposed by Hedges and Schauer (Hedges & Schauer, 2019 ), which assumes replication unless substantial evidence suggests otherwise. This framework recognised inherent differences between the Cochrane and CogTale methodologies, frameworks, and evidence grading systems, and does not aim for exact replication. Instead, it focused on assessing the degree of alignment between these systems, while also interpreting any observed differences. To operationalise this approach, we evaluated replication based on five predefined criteria, allowing for methodological flexibility to accommodate minor discrepancies: 1 Direction of the estimate . We compared the direction of each pair of effect estimates. Replication was assumed when the direction of the effect estimates was aligned. 2 Equivalence of point estimate . For each outcome, we calculated the standardized difference between each pair of effect estimates (Cochrane and CogTale) using z-tests, with the combined variance of the estimates derived from their respective standard errors. Replication was assumed when there was no significant standardized difference defined as p < 0.00278. The significance threshold was adjusted using the Bonferroni correction to account for multiple comparisons. The original significance level (α = 0.05) was divided by the number of outcomes (k = 18). 3 Overlapping 95% CIs . The 95% CIs for each pair of effect estimates were compared. Replication was assumed when there was an overlap of the 95% CIs. 4 Effect magnitude . The magnitudes of each pair of effect estimates were compared and interpreted according to a modified system informed by Cohen's (Cohen, 1969 ) and Brydges’ (Brydges, 2019 ) guidelines as small (≤ 0.3), medium ( > = 0.3 to < 0.8), or large (≥ 0.8). While these thresholds provide a general framework for categorization, we acknowledge that they should not be interpreted as rigid cut-offs. For example, differences near the boundaries (e.g., 0.29 vs. 0.31) are evaluated within the context of the broader replication criteria. Replication was assumed unless the magnitudes of the effect estimates differed substantially, defined as being two categories apart (e.g., small vs. large). A one-category difference was acceptable if all other criteria supported replication. 5 Certainty in the evidence . Certainty in the evidence was compared between Cochrane and CogTale. Cochrane's certainty in the evidence was assessed using the GRADE framework, which grades the certainty of evidence from very low to high by considering factors that may decrease or increase confidence (Schünemann et al., 2019 ). CogTale employs a custom evidence grading algorithm, combining information about the magnitude of the effect, mean methodological quality of the included studies, heterogeneity, and sample size, based on which ratings of low, moderate, or high certainty are produced. Replication was assumed unless substantial differences in certainty in the evidence ratings were found—defined as two levels apart. Despite these methodological differences, evaluating alignment between the systems was critical in understanding CogTale’s ability to approximately replicate Cochrane findings and identify areas for improvement. In evaluating the replicability of findings from the Cochrane Reviews using the CogTale platform, we employed two thresholds to interpret replication outcomes. The stringent threshold classified replication as unsuccessful if any of the five criteria were violated. In contrast, the lenient threshold classified replication as unsuccessful only when more than one criterion was violated. These thresholds were designed to explore varying levels of alignment between the two systems under different conditions. 3. Results Of the 72 studies included across the three Cochrane Reviews, 67 (93.05%) were included in the meta-analysis of at least one outcome, and 61 studies (84.72%) were included on the CogTale platform. Of the 10 studies that were not included, one (De Vreese & Neri, 1999) could not be sourced, and nine (Alvares-Pereira et al., 2021; (Beck et al., 1988; (Breuil et al., 1994; (Brueggen et al., 2017; (Goudour et al., 2011; (Heiss et al., 2008; (Kallio et al., 2018; (Marinho et al., 2021; (Rai et al., 2021) did not report raw values (means, SDs) for the outcome of the experimental and control conditions at baseline and post-treatment. Of the 21 effect estimates/outcomes reported in the SoF tables of the three Cochrane reviews, 18 were included in the replication study. Two outcomes concerning CT—Participant Burden (retention rates) and Change in Mood and Well-being (caregiver)—were excluded due to the absence of a corresponding outcome in CogTale for the former and only one study in the latter, below CogTale's threshold of three studies required for a meta-analysis. One outcome concerning CR, self-efficacy (participant self-report), was excluded as it included only two studies. Applying the stringent threshold, approximate replication was observed for 11 of 18 outcomes (61.11%), while 16 of the 18 outcomes (88.89%) met the criteria for replication under the lenient threshold. For 17 of the 18 outcomes (94.4%), the direction of the estimates aligned, and 95% confidence intervals overlapped, indicating replication. Additionally, no statistically significant differences were detected in any of the point effect estimates, further supporting the assumption of replication. Effect magnitudes were consistently classified for thirteen outcomes (72.22%), differed by one category for four outcomes (22.22%) and differed substantially for one outcome (5.56%). Certainty in the evidence was identical for seven outcomes (38.89%), varied by one category for seven outcomes (38.89%), and differed substantially for four outcomes (22.22%). Table 2 summarises the replication criteria and decisions concerning the 18 outcomes included in the replication. Table 3 shows the z-score values and comparison between Cochrane and Cogtale effect estimate magnitude and certainty in the evidence. Figure 2 provides a forest plot comparing Cochrane and CogTale effect size estimates with 95% CIs. [ Insert Table 2] [Insert Table 3] [ Insert Figure 2] We assessed seven outcomes concerning the effect of cognitive stimulation. Of the 34 studies that contributed to these outcomes in Woods et al. 1 , 29 (85.29%) were included in the CogTale sample. Approximate replication was achieved for all seven outcomes under our stringent and lenient thresholds. No significant differences were observed in the point effect estimates; all 95% CIs overlapped. Effect magnitudes were consistently classified for five outcomes (71.43%) and differed by one category for two outcomes (28.57%). Certainty in the evidence was identical for four outcomes (57.14%) and varied by one category for three outcomes (42.86%). 3.1. Cognitive Training We assessed five outcomes concerning the effect of cognitive training. Of the 32 studies that contributed to these outcomes in Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), 26 (81.25%) were included in the CogTale sample. Under the stringent threshold, approximate replication was achieved for two of the five outcomes (40%). In contrast, under the lenient threshold, approximate replication was achieved for all five outcomes (100%). For all outcomes the direction of the estimates aligned, no significant differences were observed in the point effect estimates and 95% CIs overlapped. Effect magnitudes were consistently classified for three outcomes (60%) and differed by one category for two outcomes (40%). Certainty in the evidence was identical for one outcome (20%), varied by one category for one outcome (20%), and differed substantially for three outcomes (60%). The three outcomes that were not replicated—Change in Capacity for Activities of Daily Living, Change in Delayed Memory, and Change in Participants' Mood — are further explored below. 3.1.1. Cognitive Training: Change in Capacity for Activities of Daily Living Of the ten studies (687 participants) that contributed to this outcome in Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), six (60%) were included in the CogTale sample, representing a total of 884 participants across all conditions. Differences in sample sizes owe to Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) reporting the sample size for only the intervention and control group, whereas CogTale reports the total sample size immediately post-intervention, across subgroups or conditions. Approximate replication of study-level effect estimates was observed in three (50%) studies (Bergamaschi et al., 2013; (Giuli et al., 2016; (Jelcic et al., 2012). However, discrepancies in effect size, direction, and magnitude indicated non-replication for specific studies. For example, Lee et al., (Lee et al., 2013), demonstrated overlapping 95% CIs but substantial differences in effect estimate magnitude (large vs. small) and misaligned effect estimate direction. Similarly, Galante et al., (Galante et al., 2007) and Amieva et al (Amieva et al., 2016)., showed overlapping 95% CIs but divergent effect directions. The analysis of the pooled estimates demonstrated small effect sizes with overlapping 95% CIs and aligned effect directions. However, substantial differences in certainty in the evidence were observed, with Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), reporting low certainty and CogTale reporting high certainty. 3.1.2. Cognitive Training: Change in Delayed Memory Of the 11 studies (543 participants) analysed by Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), nine (81.81%) were included in the CogTale sample, representing a total of 631 participants across all conditions. Approximate replication was observed in eight of the nine (88.89%) studies (Barban et al., 2016; (Boller et al., 2012; (Davis et al., 2001; (Jelcic et al., 2014; (Jelcic et al., 2012; (Mapelli et al., 2013; (Quayhagen et al., 2000; (Trebbastoni et al., 2018). Study-level discrepancies in the direction of effect estimates, combined with discrepancies in the certainty in the evidence ratings observed in the analysis of the pooled estimates, indicated nonreplication. For example, Cahn-Weiner (Cahn-Weiner et al., 2003) demonstrated effect estimates were classified within the same category with overlapping 95% CIs; however, the effect estimate direction was misaligned. The analysis of the pooled estimates demonstrated effect size differences of one category, with Cochrane indicating a large effect and CogTale reporting a medium effect. Both analyses demonstrated overlapping 95% CIs and aligned effect directions. However, substantial differences in certainty in the evidence were observed, with Cochrane indicating very low certainty and CogTale reporting moderate certainty. 3.1.3. Cognitive Training: Change in Participants' Mood Of the eight studies contributing to this outcome in Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) (577 participants), all eight (100%) were included in the CogTale sample representing 1,044 participants across all conditions. Approximate replication was observed in five (62.5%) studies (Amieva et al., 2016; (Bergamaschi et al., 2013; (Davis et al., 2001; (Galante et al., 2007; (Lee et al., 2013). However, discrepancies in effect magnitude, 95% CIs, and effect direction indicated non-replication in several cases. For instance, Galante (Galante et al., 2007) reported overlapping 95% CIs but demonstrated substantial differences in effect size magnitude (large vs. small) and misaligned effect directions. Similarly, Giuli (Giuli et al., 2016) demonstrated a significant z-score, no overlap in 95% CIs, divergent effect directions, and substantial differences in effect size magnitude (large vs. small). The analysis of the pooled estimate indicated that effect estimates were classified within the same category, with overlapping 95% CIs and aligned effect estimate direction. However, differences in certainty in the evidence were notable: Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) rated the certainty as very low, whereas CogTale reported moderate certainty. 3.2. Cognitive Rehabilitation All six studies included in Kudlicka et al. (Kudlicka et al., 2023) were also included in the CogTale sample (100%). Approximate replication was achieved for two outcomes (33.33%) when using the stringent threshold, and four outcomes (66.66) when using the lenient threshold. The direction of the estimates aligned for all outcomes except one (16.67%), Quality of life (participant self-report). No significant differences in point effect estimates were detected; however, one (16.67%) outcome—Functional ability in targeted activities: personal goals - performance (PR)—demonstrated a non-overlapping 95% CI. The 95% CIs overlapped for the remaining five outcomes (83.33%). Effect magnitudes were classified within the same category for five outcomes (83.33%) and differed substantially for one outcome (16.67%). Certainty in the evidence ratings were identical for one outcome (16.67%), differed by one category for three outcomes (50%), and differed substantially for two outcomes (33.33%). The four outcomes that were not replicated—Functional ability in targeted activities: personal goals - performance (PR), General functional ability (IR), Mood: depression (PT), and Quality of life (PR) — are further explored below. 3.2.1. Cognitive Rehabilitation: Functional Ability in Targeted Activities: Personal Goals - Performance (PR) Kudlicka et al. (Kudlicka et al., 2023) analysed three studies with 501 participants, and CogTale reviewed three with 531 participants across all conditions. Approximate replication was not achieved in two (66.6%) studies (Clare et al., 2010; (Hindle et al., 2018). Discrepancies in 95% CIs, significant differences in the point effect estimates, and certainty in the evidence indicated nonreplication. For example, Clare et al. (Clare et al., 2019)—despite demonstrating an aligned effect estimate direction, showed no overlap in 95% CIs and a significant z-score. The analysis of the pooled estimates results demonstrated overlapping 95% CIs and significant differences in the point effect estimates. Kudlicka et al. (Kudlicka et al., 2023) rated the certainty in the evidence as high, while CogTale rated it moderate. 3.2.2. Cognitive Rehabilitation: General functional ability (IR) Kudlicka et al. (Kudlicka et al., 2023) analysed three studies with 673 participants across intervention and control conditions, and CogTale reviewed three with 974 participants across all conditions. Approximate replication was achieved in all three studies (100%). All effect estimates were aligned, with overlapping 95% CIs; moreover, there were no significant z-scores. At the study level, Thivierge et al. (Thivierge et al., 2014) indicated a one-category difference in effect estimate magnitude. In the analysis of the pooled estimates, substantial differences in certainty in the evidence indicated nonreplication. Kudlicka et al. (Kudlicka et al., 2023) rated the certainty in the evidence as low, while CogTale rated it high. 3.2.3. Cognitive Rehabilitation: Mood: Depression (Participant Self-Report) Kudlicka et al. (Kudlicka et al., 2023) analysed three studies with 502 participants, and CogTale reviewed three with 543 participants across all conditions, representing 100% of the Cochrane sample. Approximate replication was achieved for one (33.33%) study (Clare et al., 2010). Differences in effect size magnitudes, 95% CIs, and certainty in the evidence contributed to non-replication. For example, Clare et al. (Clare et al., 2019) exhibited overlapping 95% CIs but misaligned effect directions, resulting in non-replication. Moreover, Hindle et al. (Hindle et al., 2018) demonstrated aligned effect directions but no overlap in 95% CIs, leading to non-replication. In the analysis of the pooled estimates, the results demonstrated overlapping 95% confidence intervals (CIs) and consistent alignment in the direction of the effects. However, there was a substantial difference in effect size magnitude (large vs. small) and certainty in the evidence with Kudlicka et al. (Kudlicka et al., 2023) reporting very low certainty and CogTale indicating moderate certainty. 3.2.4. Cognitive Rehabilitation: Quality of life (PR) Kudlicka et al. (Kudlicka et al., 2023) analysed five studies with 853 participants across intervention and control conditions, and CogTale reviewed four with 582 participants across all conditions, representing (80%) of the Cochrane sample. Approximate replication was only achieved in only one study (25%). Only one effect estimate was aligned; however, all 95% CIs overlapped. No significant z-scores were detected. All effect size magnitudes were classified as the same category. At the study level, Thivierge et al. (Thivierge et al., 2014) indicated a one-category difference in effect estimate magnitude. The analysis of the pooled estimated substantial differences in certainty in the evidence indicated nonreplication. Kudlicka et al. (Kudlicka et al., 2023) rated the certainty in the evidence as low, while CogTale rated it high. Table 3 shows a summary of the replication criteria and decisions in relation to outcomes that indicated nonreplication. Table 4 provides the z-score values and comparison between Cochrane and Cogtale effect estimate magnitudes. Figure 3 provides a forest plot comparing Cochrane and CogTale study level effect size estimates with 95% CIs for the identified outcomes that indicated nonreplication. [Insert Table 4] [ Insert Table 5] [ Insert Figure 3] 4. Discussion This study evaluated the extent to which the CogTale platform could replicate key findings from recent Cochrane Reviews of COTs for individuals with dementia. Specifically, the study examined whether CogTale could generate effect estimates aligned with Cochrane Reviews in terms of effect direction, absolute values, magnitude, and certainty in the evidence. By applying both stringent and lenient replication thresholds, this research not only validated aspects of CogTale's meta-analytic functionality but also highlighted its capacity to produce timely, high-quality evidence syntheses. These findings underscore CogTale's potential as a valuable tool for addressing the challenges of traditional systematic reviews and meta-analyses, particularly their resource intensity and susceptibility to rapid obsolescence in the context of the rapidly growing body of literature on aging-related interventions Depending on the strictness of the replication criteria applied, Cochrane findings were replicated on the CogTale platform to varying degrees. Under the lenient threshold, most outcomes—close to nine out of ten— were deemed replicated. In contrast, when a more stringent threshold was applied, approximately two-thirds of outcomes met the replication criteria. CST outcomes aligned closely with Cochrane results, whereas CT and CR outcomes showed more variability. Discrepancies were evident in CT and CR outcomes, where effect size magnitudes, direction, and certainty in the evidence ratings diverged. Notably, CogTale and Cochrane were aligned on the primary outcomes for CST and CT. For CST, the primary outcome "Cognition" was approximately replicated under the stringent threshold, demonstrating consistent findings between the two methodologies. Similarly, for CT, the primary outcome "Change in a global measure of cognition (composite)" was also approximately replicated under the stringent threshold. These results highlight CogTale's ability to align with Cochrane’s findings for the main outcomes of these interventions. However, for CR, the primary outcome "Functional ability in targeted activities: personal goals – performance (PR)" was not approximately replicated, as the 95% confidence intervals did not overlap. While some variability was observed for secondary outcomes, these measures often differ across studies and are not always the primary focus of the interventions. This suggests that the alignment of primary outcomes, such as cognition for CST and CT, and functional ability for CR, represents a meaningful and important consistency that warrants emphasis despite other divergences. These differences suggest that while CogTale partially aligns with Cochrane’s results, methodological variations, such as differences in evidence grading and included study samples, may influence replication accuracy. Nonreplication in this study likely arises from several factors. First, differences in the study samples included in CogTale and those included in Cochrane Reviews arose from select primary studies lacking the raw data (means, standard deviations) required by CogTale. Approximately one-fifth of studies for CT and just over one-seventh for CST were excluded, potentially contributing to variability in replicability. For CST, approximate replication was achieved for all seven assessed outcomes under both stringent and lenient thresholds, demonstrating strong alignment with Cochrane findings. In contrast, for CT, only two of five outcomes met the stringent replication threshold, though all five outcomes were replicated under the lenient threshold. Cognitive rehabilitation (CR), which included all six studies from Cochrane, demonstrated approximate replication for only two of six outcomes under the stringent threshold and four outcomes under the lenient threshold. These findings suggest that while data availability influenced replicability for CST and CT, methodological differences, such as evidence grading and analysis approaches, may have played a greater role in the variability observed for CR outcomes. Variations in evidence grading methodologies—Cochrane’s utilisation of GRADE versus CogTale’s custom evidence grading algorithm—could account for divergent certainty in the evidence ratings. Additionally, methodological differences, such as manual data coding in CogTale, increase variability and the potential for human error, although this process is not unique to CogTale. Similar manual data extraction methods are employed in Cochrane reviews, where review authors independently extract data using structured forms and resolve disagreements through consensus. While both approaches rely on human input, CogTale coders undergo comprehensive training and their performance is regularly monitored by platform admins, aiming to standardise data entry and minimise errors. The current findings suggest that while CogTale demonstrates promise as a tool for automated evidence synthesis in COTs, its approach differs from the more conservative and methodologically restrictive standards of Cochrane. Rather than holding CogTale to Cochrane’s stringent benchmarks, this study highlights the potential benefits of CogTale’s more adaptable framework, particularly its ability to provide timely, user-friendly, and scalable evidence synthesis that can accommodate emerging data and evolving methodologies. CST, which had the largest sample size, showed strong alignment with Cochrane findings, instilling confidence in CogTale’s potential to provide credible evidence in robust datasets. However, replication was mixed for some CT and CR outcomes, highlighting the need for caution when interpreting results in domains with limited data. CogTale offers a unique opportunity to both align with established evidence synthesis frameworks and highlight gaps in traditional approaches. Its flexibility, user-friendly design, and nimble ability to rapidly process data set it apart from established methods such as Cochrane, which can be constrained by rigid and time-intensive processes. By leveraging adaptive evidence grading, integrating diverse data sources, and embracing innovative methodologies, CogTale provides a dynamic platform that fosters confidence in findings—even in contexts where established systems may fall short, such as rapidly evolving fields, or areas requiring real-time updates and adaptability. Strengthening trust in the system can be further achieved by enhancing transparency in algorithms for effect size estimation and certainty ratings, including comprehensive raw data, and refining outcome classification schemes. These improvements would underscore the system’s potential to provide reliable and accessible insights for clinicians and researchers. Additionally, ongoing training, calibration, and quality-control checks for human coders remain essential to ensure fidelity and reduce variability in the data extraction process. Nevertheless, for clinicians and stakeholders, CogTale offers significant potential to streamline evidence synthesis and decision-making, provided its limitations are carefully considered. Our findings also underscore the potential benefits and current challenges of increasing automation in systematic meta-reviews. In a previous paper, Rasool et al. (Rasool et al., 2024 ) demonstrated the feasibility of using large language models (LLMs) to expedite data extraction, thereby automating and increasing the speed of the evidence synthesis pipeline. However, their findings were mixed, highlighting that while LLMs show promise in streamlining data gathering, reducing human error, and enabling rapid updates to evidence syntheses, further work is required to address inconsistencies and optimize their integration into SRMA processes. While the present study did not incorporate LLM-based extraction within CogTale, the principle stands: employing artificial intelligence and LLMs can streamline data gathering, reduce human error, and expedite the development of “living” evidence syntheses. By leveraging these technologies, platforms like CogTale can be iteratively refined, moving closer to near-real-time evidence updates. Such advancements could significantly improve the responsiveness of evidence-based recommendations in dementia care and beyond. Given their rigorous methodological standards, this study’s use of Cochrane reviews as a validation benchmark is a notable strength, particularly due to their systematic approach and high-quality synthesis of randomised controlled trials. However, it is also essential to recognise the limitations of the Cochrane approach, including their often-limited scope, time-intensive processes, and the potential for excluding important contextual or non-randomised evidence. These constraints highlight the need for complementary methods, such as those offered by CogTale, to address broader clinical questions, incorporate emerging data, and provide more accessible, real-time insights that extend beyond Cochrane’s traditionally narrow focus. CogTale’s ability to replicate findings across diverse COTs highlights its versatility and scalability, demonstrating robustness even when data was incomplete. However, the exclusion of slightly less than one-fifth of Cochrane studies due to data unavailability introduces potential bias, as the missing studies might alter overall interpretations. A further limitation is the focus on replicating only the first Summary of Findings (SoF) table from each Cochrane review rather than the full range of analyses, limiting the comprehensiveness of the validation. Additionally, reliance on manual data coding is time- and labour-intensive and introduces variability. Future research should prioritise several critical areas to enhance the reliability and utility of CogTale and similar platforms. Data completeness and transparency must be improved by encouraging open data practices ensuring the availability and standardisation of raw data in primary studies. Increased data availability will enhance CogTale’s capacity to replicate established findings accurately. The data and code used in this study to reproduce the findings are accessible in the public repository the Open Science Framework. Methodological harmonisation is another key area, with efforts needed to align CogTale’s evidence grading procedures more closely with widely accepted frameworks, such as GRADE. Incorporating automated quality assessment tools and consensus-driven approaches to evidence weighting could further reduce discrepancies in certainty in the evidence ratings. Moreover, the integration of advanced automation also requires attention, with future iterations of CogTale leveraging LLMs, machine learning algorithms, and other AI-driven tools for data extraction and synthesis. These technologies could accelerate the development of "living" SRMAs that are continuously updated in the context of new evidence. Additionally, user education is essential to enhance trust and utility. Clear communication about how CogTale’s results are generated, alongside training modules, online tutorials, and decision aids, can help clinicians, researchers, and policymakers interpret the platform's outputs appropriately. Given the existing and comprehensive eCourse provided in the training of human coders, important steps have already been made in this direction. Finally, ongoing validation efforts should aim to replicate a broader array of outcomes, including all Summary of Findings tables from the three Cochrane Reviews. Incorporating other high-quality systematic reviews and conducting independent validation studies will further strengthen CogTale’s credibility as a tool for evidence synthesis. This study demonstrated that CogTale, as an automated evidence synthesis platform, can approximate the rigorous findings of Cochrane Reviews for COTs in dementia, offering a promising tool for advancing older adult research. While some discrepancies were identified, they provide valuable insights into areas for methodological refinement, enhanced data availability, and the integration of more advanced automation tools. These findings underscore the platform’s potential to streamline evidence synthesis processes, enabling timely and reliable information to guide clinicians, researchers, and policymakers. By accelerating the synthesis of evidence and supporting dynamic guideline development, CogTale can contribute significantly to improving therapeutic strategies for age-related cognitive decline and enhancing the quality of care for older adults with dementia. Declarations Conflicts of Interest A.B.F, B.M.H, S.B, T.D and S.S.S are the scientific developers of the CogTale platform, which was evaluated in this study. The other co-authors (S.H.R, C.C, I.J.M.B, J.C) have no competing interests to declare. The data used to support the findings of this study are available on the open science framework Funding declaration This project was funded by National Institute on Aging R35AG072262 (awarded to B.M.H). Author Contribution Conceptualisation: A.B.F; R.H.S; C.C. Methodology: A.B.F; R.H.S; C.C. Data Curation: S.H.R; I.J.M.B; C.C; J.C. Formal Analysis: S.H.R. Funding Acquisition: A.B.F; B.M.H.. Project Administration: S.H.R; I.J.M.B; C.C. Supervision: A.B.F. Validation: S.H.R; A.B.F; C.C. Visualisation: S.H.R. Writing –Original Draft: S.H.R; C.C. Writing – Review & Editing: A.B.F; B.M.H; S.B; S.S.S; S.H.R; C.C. All authors approved the final submitted draft. Acknowledgement The authors wish to dedicate this paper to the memory of co-author Prof Tzvi Dwolatzky, who sadly passed away in 2024. Prof Dwolatzky was a founding member of the CIDER Research Working Group and was involved in the development of the CogTale Platform from inception. He was a passionate clinician and researcher, committed to scientific research and discovery, but first and foremost, devoted to his patients, students, and mentees. He was a generous and kind individual who leaves a strong legacy, and will be sorely missed. The CogTale team would also like to acknowledge the contributions of a range of trainee coders through the CogTale internship who have assisted with establishing and growing the CogTale database. Data Availability The data used to support the findings of this study are available on the open science framework https://osf.io/vrzqu/?view_only=cc8353b8a8f24e769d3bec4217a59d9d References Alvares-Pereira, G., V., S.-N. M.,and, & Spector, A. (2021). Validation of the cognitive stimulation therapy (CST) program for people with dementia in Portugal. Aging & Mental Health , 25 (6), 1019–1028. https://doi.org/10.1080/13607863.2020.1836473 Amieva, H., Robert, P. H., Grandoulier, A. S., Meillon, C., De Rotrou, J., Andrieu, S., Berr, C., Desgranges, B., Dubois, B., Girtanner, C., Joël, M. E., Lavallart, B., Nourhashemi, F., Pasquier, F., Rainfray, M., Touchon, J., Chêne, G., & Dartigues, J. F. (2016). Group and individual cognitive therapies in Alzheimer's disease: the ETNA3 randomized trial. International Psychogeriatrics , 28 (5), 707–717. https://doi.org/10.1017/S1041610215001830 Bahar-Fuchs, A., Clare, L., & Woods, B. (2013). Cognitive training and cognitive rehabilitation for mild to moderate Alzheimer's disease and vascular dementia. The Cochrane database of systematic reviews , 6 , CD003260. https://doi.org/10.1002/14651858.CD003260.pub2 Bahar-Fuchs, A., Hampstead, B. M., & Clare, L. (2014). Cognitive training for older adults with MCI and mild dementia: State of the science, central challenges and possible solutions. Alzheimer's & Dementia , 10 . https://doi.org/10.1016/j.jalz.2014.04.138 Bahar-Fuchs, A., Martyr, A., Goh, A., Sabatés, J., & Clare, L. (2019). Cognitive training for people with mild to moderate dementia. The Cochrane database of systematic reviews , 3 , CD013069. https://doi.org/10.1002/14651858.CD013069.pub2 Barban, F., Annicchiarico, R., Pantelopoulos, S., Federici, A., Perri, R., Fadda, L., Carlesimo, G. A., Ricci, C., Giuli, S., & Scalici, F. (2016). Protecting cognition from aging and Alzheimer's disease: a computerized cognitive training combined with reminiscence therapy. International Journal of Geriatric Psychiatry , 31 (4), 340–348. https://doi.org/10.1002/gps.4328 Beck, C., Heacock, P., Mercer, S., Thatcher, R., & Sparkman, C. (1988). The impact of cognitive skills remediation training on persons with Alzheimer's disease or mixed dementia. Journal of Geriatric Psychiatry , 21 (1), 73–88. Bergamaschi, S., Arcara, G., Calza, A., Villani, D., Orgeta, V., & Mondini, S. (2013). One-year repeated cycles of cognitive training (CT) for Alzheimer’s disease. Aging clinical and experimental research , 25 , 421–426. https://doi.org/10.1007/s40520-013-0065-2 Boller, B., Jennings, J. M., Dieudonné, B., Verny, M., & Ergis, A. M. (2012). Recollection training and transfer effects in Alzheimer’s disease: effectiveness of the repetition-lag procedure. Brain and Cognition , 78 (2), 169–177. https://doi.org/10.1016/j.bandc.2011.10.011 Breuil, V., De Rotrou, J., Forette, F., Tortrat, D., Ganansia-Ganem, A., Frambourt, A., Moulin, F., & Boller, F. (1994). Cognitive stimulation of patients with dementia: Preliminary results. International Journal of Geriatric Psychiatry , 9 (3), 211–217. https://doi.org/10.1002/gps.930090306 Brueggen, K., Kasper, E., Ochmann, S., Pfaff, H., Webel, S., Schneider, W., & Teipel, S. (2017). Cognitive Rehabilitation in Alzheimer’s Disease: A Controlled Intervention Trial. Journal of Alzheimer’s Disease , 57 (4), 1315–1324. https://doi.org/10.3233/jad-160771 Brydges, C. R. (2019). Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology. Innovation in Aging , 3 (4). https://doi.org/10.1093/geroni/igz036 Cahn-Weiner, D. A., F., M. P., W., R. G.,and, & Ott, B. R. (2003). Results of a Randomized Placebo-Controlled Study of Memory Training for Mildly Impaired Alzheimer's Disease Patients. Applied Neuropsychology , 10 (4), 215–223. https://doi.org/10.1207/s15324826an1004_3 Christopoulou, S. C. (2023). Towards Automated Meta-Analysis of Clinical Trials: An Overview. BioMedInformatics , 3 (1), 115–140. https://mdpi-res.com/d_attachment/biomedinformatics/biomedinformatics-03-00009/article_deploy/biomedinformatics-03-00009.pdf?version=1675239592 Clare, L. (2017). Rehabilitation for people living with dementia: A practical framework of positive support. PLoS Medicine , 14 . https://doi.org/10.1371/journal.pmed.1002245 Clare, L., Kudlicka, A., Oyebode, J. R., Jones, R. W., Bayer, A., Leroi, I., Kopelman, M., James, I. A., Culverwell, A., Pool, J., Brand, A., Henderson, C., Hoare, Z., Knapp, M., & Woods, B. (2019). Individual goal-oriented cognitive rehabilitation to improve everyday functioning for people with early-stage dementia: A multicentre randomised controlled trial (the GREAT trial). International Journal of Geriatric Psychiatry , 34 (5), 709–721. https://doi.org/10.1002/gps.5076 Clare, L., Linden, D. E. J., Woods, R. T., Whitaker, R., Evans, S. J., Parkinson, C. H., van Paasschen, J., Nelis, S. M., Hoare, Z., Yuen, K. S. L., & Rugg, M. D. (2010). Goal-Oriented Cognitive Rehabilitation for People With Early-Stage Alzheimer Disease: A Single-Blind Randomized Controlled Trial of Clinical Efficacy. The American Journal of Geriatric Psychiatry , 18 (10), 928–939. https://doi.org/10.1097/JGP.0b013e3181d5792a Clare, L., & Woods, R. T. (2004). Cognitive training and cognitive rehabilitation for people with early-stage Alzheimer's disease: A review. Neuropsychological Rehabilitation , 14 , 385–401. https://doi.org/10.1080/09602010443000074 Cohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences. The SAGE Encyclopedia of Research Design . Davis, R. N., Massman, P. J., & Doody, R. S. (2001). Cognitive intervention in Alzheimer disease: a randomized placebo-controlled study. Alzheimer Disease & Associated Disorders , 15 (1), 1–9. De Vreese, L., & Neri, M. (1999). Ecological impact of combined cognitive training programs and drug treatment in Alzheimer’s disease. International Psychogeriatrics , 11 , S187. Evans, D. (2003). Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing , 12 (1), 77–84. https://doi.org/10.1046/j.1365-2702.2003.00662.x Galante, E., Venturini, G., & Fiaccadori, C. (2007). Computer-based cognitive intervention for dementia: preliminary results of a randomized clinical trial. G Ital Med Lav Ergon , 29 (3 Suppl B), B26–32. García-Perdomo, H. A. (2016). Evidence synthesis and meta-analysis: a practical approach. International Journal of Urological Nursing , 10 (1), 30–36. https://doi.org/10.1111/ijun.12087 Giuli, C., Papa, R., Lattanzio, F., & Postacchini, D. (2016). The Effects of Cognitive Training for Elderly: Results from My Mind Project. Rejuvenation Research , 19 (6), 485–494. https://doi.org/10.1089/rej.2015.1791 Goudour, A., Samson, S., Bakchine, S., & Ehrlé, N. (2011). [Semantic memory training in Alzheimer's disease]. Geriatrie et psychologie neuropsychiatrie du vieillissement , 9 2 , 237–247. https://doi.org/10.1684/pnv.2011.0272 Hartung, J., & Knapp, G. (2001). A refined method for the meta-analysis of controlled clinical trials with binary outcome. Statistics in Medicine , 20 (24), 3875–3889. https://doi.org/10.1002/sim.1009 Hedges, L. V. (1981). Distribution Theory for Glass's Estimator of Effect Size and Related Estimators. Journal of Educational Statistics , 6 (2), 107–128. https://doi.org/10.2307/1164588 Hedges, L. V. (2019). The Statistics of Replication. Methodology , 15 (Supplement 1), 3–14. https://doi.org/10.1027/1614-2241/a000173 Hedges, L. V., & Schauer, J. M. (2019). Statistical analyses for studying replication: Meta-analytic perspectives. Psychological Methods , 24 (5), 557–570. https://doi.org/10.1037/met0000189 Heiss, W. D., Kessler, J., Mielke, R., Szelies, B., & Herholz, K. (2008). Long-Term Effects of Phosphatidylserine, Pyritinol, and Cognitive Training in Alzheimer's Disease: A Neuropsychological, EEG, and PET Investigation. Dementia 5 (2), 88–98. https://doi.org/10.1159/000106702 Higgins, J. P., Savović, J., Page, M. J., Elbers, R. G., & Sterne, J. A. (2019). Assessing risk of bias in a randomized trial. In Cochrane Handbook for Systematic Reviews of Interventions (pp. 205–228). https://doi.org/10.1002/9781119536604.ch8 Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., Savović, J., Schulz, K. F., Weeks, L., & Sterne, J. A. C. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj , 343 , d5928. https://doi.org/10.1136/bmj.d5928 Hindle, J. V., Watermeyer, T. J., Roberts, J., Brand, A., Hoare, Z., Martyr, A., & Clare, L. (2018). Goal-orientated cognitive rehabilitation for dementias associated with Parkinson's disease―A pilot randomised controlled trial. International Journal of Geriatric Psychiatry , 33 (5), 718–728. https://doi.org/10.1002/gps.4845 Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials , 17 (1), 1–12. https://doi.org/10.1016/0197-2456(95)00134-4 Jelcic, N., Agostini, M., Meneghello, F., Bussè, C., Parise, S., Galano, A., Tonin, P., Dam, M., & Cagnin, A. (2014). Feasibility and efficacy of cognitive telerehabilitation in early Alzheimer’s disease: a pilot study. Clinical Interventions in Aging , 1605–1611. https://doi.org/10.2147/CIA.S68145 Jelcic, N., Cagnin, A., Meneghello, F., Turolla, A., Ermani, M., & Dam, M. (2012). Effects of lexical–semantic treatment on memory in early Alzheimer disease: An observer-blinded randomized controlled trial. Neurorehabilitation and neural repair , 26 (8), 949–956. https://doi.org/10.1177/1545968312440146 Johnson, E. E., O’Keefe, H., Sutton, A., & Marshall, C. (2022). The Systematic Review Toolbox: keeping up to date with tools to support evidence synthesis. Systematic Reviews , 11 (1), 258. https://doi.org/10.1186/s13643-022-02122-z Kallio, E. L., Öhman, H., Hietanen, M., Soini, H., Strandberg, T. E., Kautiainen, H., & Pitkälä, K. H. (2018). Effects of Cognitive Training on Cognition and Quality of Life of Older Persons with Dementia. Journal of the American Geriatrics Society , 66 (4), 664–670. https://doi.org/10.1111/jgs.15196 Kudlicka, A., Martyr, A., Bahar-Fuchs, A., Sabatés, J., Woods, B., & Clare, L. (2023). Cognitive rehabilitation for people with mild to moderate dementia. The Cochrane database of systematic reviews , 6 , CD013388. https://doi.org/10.1002/14651858.CD013388.pub2 Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019). A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Research Synthesis Methods , 10 (1), 83–98. https://doi.org/10.1002/jrsm.1316 Lee, G. Y., Yip, C. C., Yu, E. C., & Man, D. W. (2013). Evaluation of a computer-assisted errorless learning-based memory training program for patients with early Alzheimer’s disease in Hong Kong: a pilot study. Clinical Interventions in Aging , 623–633. https://doi.org/10.2147/CIA.S45726 Maher, C. G., Sherrington, C., Herbert, R. D., Moseley, A. M., & Elkins, M. (2003). Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials. Physical Therapy , 83 (8), 713–721. https://doi.org/10.1093/ptj/83.8.713 Mapelli, D., Di Rosa, E., Nocita, R., & Sava, D. (2013). Cognitive stimulation in patients with dementia: randomized controlled trial. Dementia and geriatric cognitive disorders extra , 3 (1), 263–271. https://doi.org/10.1159/000353457 Marinho, V., Bertrand, E., Naylor, R., Bomilcar, I., Laks, J., Spector, A., & Mograbi, D. C. (2021). Cognitive stimulation therapy for people with dementia in Brazil (CST-Brasil): Results from a single blind randomized controlled trial. International Journal of Geriatric Psychiatry , 36 (2), 286–293. https://doi.org/10.1002/gps.5421 Marshall, C., Kitchenham, B. A., & Brereton, P. (2018). Tool Features to Support Systematic Reviews in Software Engineering ? A Cross Domain Study. e Informatica Softw Eng J , 12 , 79–115. Paul, M., & Leibovici, L. (2014). Systematic review or meta-analysis? Their place in the evidence hierarchy. Clinical Microbiology and Infection , 20 (2), 97–100. https://doi.org/10.1111/1469-0691.12489 Quayhagen, M. P., Quayhagen, M., Corbeil, R. R., Hendrix, R. C., Jackson, J. E., Snyder, L., & Bower, D. (2000). Coping with dementia: evaluation of four nonpharmacologic interventions. International Psychogeriatrics , 12 (2), 249–265. https://doi.org/10.1017/S1041610200006360 Rai, H. K., Schneider, J., & Orrell, M. (2021). An Individual Cognitive Stimulation Therapy App for People with Dementia and Carers: Results from a Feasibility Randomized Controlled Trial (RCT). Clinical Interventions in Aging , 16 , 2079–2094. https://doi.org/10.2147/CIA.S323994 Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A. (2024). Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset. Natural Language Processing Journal , 8 , 100083. https://doi.org/10.1016/j.nlp.2024.100083 Sabates, J., Belleville, S., Castellani, M., Dwolatzky, T., Hampstead, B. M., Lampit, A., Simon, S., Anstey, K., Goodenough, B., Mancuso, S., Marques, D., Sinnott, R., & Bahar-Fuchs, A. (2021). CogTale: an online platform for the evaluation, synthesis, and dissemination of evidence from cognitive interventions studies. Systematic Reviews , 10 (1), 236. https://doi.org/10.1186/s13643-021-01787-2 Sackett, D. L. (1997). Evidence-based medicine. Seminars in Perinatology , 21 (1), 3–5. https://doi.org/10.1016/S0146-0005(97)80013-4 Schünemann, H. J., Higgins, J. P., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., & Guyatt, G. H. (2019). Group, o. b. o. t. C. G. M., & Group, t. C. S. M. Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In Cochrane Handbook for Systematic Reviews of Interventions (pp. 375–402). https://doi.org/10.1002/9781119536604.ch14 Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007). How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. Annals of Internal Medicine , 147 (4), 224–233. https://doi.org/10.7326/0003-4819-147-4-200708210-00179 Sidik, K., & Jonkman, J. N. (2007). A comparison of heterogeneity variance estimators in combining results of studies. Statistics in Medicine , 26 (9), 1964–1981. https://doi.org/10.1002/sim.2688 Tanjong-Ghogomu, E., Tugwell, P., & Welch, V. A. (2009). Evidence-based medicine and the Cochrane Collaboration. Bulletin of the NYU hospital for joint diseases , 67 2 , 198–205. Thivierge, S., Jean, L., & Simard, M. (2014). A randomized cross-over controlled study on cognitive rehabilitation of instrumental activities of daily living in Alzheimer disease. The American journal of geriatric psychiatry: official journal of the American Association for Geriatric Psychiatry , 22 (11), 1188–1199. https://doi.org/10.1016/j.jagp.2013.03.008 Trebbastoni, A., Imbriano, L., Podda, L., Rendace, L., Sacchetti, M. L., Campanelli, A., D'Antonio, F., & De Lena, C. (2018). Cognitive training in patients with Alzheimer's disease: findings of a 12-month randomized controlled trial. Current Alzheimer Research , 15 (5), 452–461. https://doi.org/10.2174/1567205014666171113105044 van Dinter, R., Tekinerdogan, B., & Catal, C. (2021). Automation of systematic literature reviews: A systematic literature review. Information and Software Technology , 136 , 106589. https://doi.org/10.1016/j.infsof.2021.106589 Woods, B., Rai, H. K., Elliott, E., Aguirre, E., Orrell, M., & Spector, A. (2023). Cognitive stimulation to improve cognitive functioning in people with dementia. The Cochrane database of systematic reviews , 1 , CD005562. https://doi.org/10.1002/14651858.CD005562.pub3 Tables Table 1 to 5 are available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files Table1.docx Table2.docx Table3.docx Table4.docx Table5.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7078540","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":489384799,"identity":"62307e4d-6a84-42db-9e8d-b6dc713a6d1a","order_by":0,"name":"Courtney Chesser","email":"","orcid":"","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Courtney","middleName":"","lastName":"Chesser","suffix":""},{"id":489384800,"identity":"8af91512-ee81-4cd3-bd31-c82aae978685","order_by":1,"name":"Storm Hiskens-Ravest","email":"","orcid":"","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Storm","middleName":"","lastName":"Hiskens-Ravest","suffix":""},{"id":489384801,"identity":"be54cbf7-cafe-4218-a5b1-b8999a2eb913","order_by":2,"name":"Isabelle J.M. Burke","email":"","orcid":"","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Isabelle","middleName":"J.M.","lastName":"Burke","suffix":""},{"id":489384804,"identity":"854efb0d-3722-46a2-bb33-ff84ef87e05f","order_by":3,"name":"Jamie Charlton","email":"","orcid":"","institution":"Deakin University","correspondingAuthor":false,"prefix":"","firstName":"Jamie","middleName":"","lastName":"Charlton","suffix":""},{"id":489384806,"identity":"f2178eaa-5cfe-493d-9473-d7ad4cd2dae1","order_by":4,"name":"Benjamin M. Hampstead","email":"","orcid":"","institution":"University of Michigan, VA Ann Arbor Healthcare System","correspondingAuthor":false,"prefix":"","firstName":"Benjamin","middleName":"M.","lastName":"Hampstead","suffix":""},{"id":489384808,"identity":"652fa984-2410-4885-be38-aba16ae2f8f2","order_by":5,"name":"Sylvie Belleville","email":"","orcid":"","institution":"Université de Montréal","correspondingAuthor":false,"prefix":"","firstName":"Sylvie","middleName":"","lastName":"Belleville","suffix":""},{"id":489384811,"identity":"01f68429-3fdb-4bb1-850a-23cde3967556","order_by":6,"name":"Sharon Sanz-Simon","email":"","orcid":"","institution":"Rutgers University","correspondingAuthor":false,"prefix":"","firstName":"Sharon","middleName":"","lastName":"Sanz-Simon","suffix":""},{"id":489384814,"identity":"44576e90-ee6e-4f46-a81e-33d82b8c7548","order_by":7,"name":"Tzvi Dwolatsky","email":"","orcid":"","institution":"Rambam Health Care","correspondingAuthor":false,"prefix":"","firstName":"Tzvi","middleName":"","lastName":"Dwolatsky","suffix":""},{"id":489384816,"identity":"5af1f62a-0cc0-44ae-87f6-b32d2874ff70","order_by":8,"name":"Alex Bahar-Fuchs","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEklEQVRIie3SMUvDQBTA8RcOMr0k64VA8hUuBALSql/lJBCXKkIhk0hBiEt1jhT8LFcOzHLGtSBIP0IgIAXRmmROKDo53H843ht+wx0HoNP9yxBIezrdALUJQLsVzMPEXbSDUfyaEOwIHCDOwlo3V9fvlJUvsplm08C9swTUmQSn4IOECjvxiuc5Zeoy9WZVGq6IzY2ikkA3wwQEMg9NfsM2GJOLXBpPBBmxcgkwQgKB0Sd+c9qSqDnK96c9+WpJMEKYwNiz8p4wz8jF2aojRkvYCAmlnU6sB05dNYvdZZUkj7fI1svqHEO1HSR+eS/f8INTu1RRvctOjotXFW532cT3y5Hrk8E3gf4z6HQ6ne6v/QC4p1bHBk0ZNwAAAABJRU5ErkJggg==","orcid":"","institution":"Deakin University","correspondingAuthor":true,"prefix":"","firstName":"Alex","middleName":"","lastName":"Bahar-Fuchs","suffix":""}],"badges":[],"createdAt":"2025-07-09 00:53:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7078540/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7078540/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87421949,"identity":"a1aacb13-f00e-4f54-a99e-90c6410f9f3c","added_by":"auto","created_at":"2025-07-23 15:40:51","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":149098,"visible":true,"origin":"","legend":"\u003cp\u003eCogtale Process Swim Lane Diagram\u003c/p\u003e\n\u003cp\u003eMongoDB is a database used for storing and managing data within the CogTale platform. R is a statistical programming language utilised for data analysis and report generation, including meta-analytic computations and visualisations. JavaScript is a programming language employed for developing the platform’s interactive web interface. The Admin role is responsible for overseeing the platform's overall functionality, managing user permissions, and maintaining data integrity. The Coder role involves extracting, entering, and verifying data from studies into the platform using predefined protocols to ensure consistency and accuracy.\u003c/p\u003e","description":"","filename":"Figure1CogtaleProcessSwimLaneDiagram.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/6746f923a4eab01fa62ee19c.jpg"},{"id":87420451,"identity":"fb12b4f7-f3af-4671-aaef-4f0e744474c2","added_by":"auto","created_at":"2025-07-23 15:24:51","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":3753504,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of Effect Sizes and 95% Confidence Intervals\u003c/p\u003e\n\u003cp\u003e‡ Substantial differences in effect size magnitude of two or more categories (e.g., small vs. large). \u003cbr\u003e\n† Substantial differences in certainty in the evidence ratings of two or more categories (e.g., very low vs. moderate). CI = Confidence Interval. PR = Participant self-report. IR = Informant report of participant.\u003c/p\u003e","description":"","filename":"Figure2ComparisonofEffectSizesandConfidenceIntervals.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/9c55a915bc7300be55ea14c2.jpg"},{"id":87421950,"identity":"b074f77d-372a-4969-9104-de0d59d48fdf","added_by":"auto","created_at":"2025-07-23 15:40:51","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":2212257,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of Study Level Effect Sizes and 95% CIs for Nonreplicated Outcomes\u003c/p\u003e\n\u003cp\u003e‡ Substantial differences in effect size magnitude of two or more categories (e.g., small vs. large). \u003cbr\u003e\n***p \u0026lt; .001, **p \u0026lt; .01, *p \u0026lt; .05. CI = Confidence Interval. PR = Participant self-report. IR = Informant report of participant.\u003c/p\u003e","description":"","filename":"Figure3EffectSizesand95CIsNonreplicatedOutcomes.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/2aae2574725e00c2279e8db5.jpg"},{"id":87422369,"identity":"ee27b726-adb6-40e1-8ea2-f27f738e3ec7","added_by":"auto","created_at":"2025-07-23 15:48:54","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":6964336,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/48844757-7c4d-4014-93bf-921ad77c82a8.pdf"},{"id":87420454,"identity":"e84d0b37-eb5d-4ae7-9a74-af84d0a9dbfc","added_by":"auto","created_at":"2025-07-23 15:24:51","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":15976,"visible":true,"origin":"","legend":"","description":"","filename":"Table1.docx","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/b93bbaa4bdeecdca0b760b4c.docx"},{"id":87420990,"identity":"26ebb341-dc8c-479a-a031-536a91a2ff46","added_by":"auto","created_at":"2025-07-23 15:32:51","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":26394,"visible":true,"origin":"","legend":"","description":"","filename":"Table2.docx","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/5223748542926b56e2303589.docx"},{"id":87421951,"identity":"54349867-f051-4606-8660-757845abbcba","added_by":"auto","created_at":"2025-07-23 15:40:51","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":26034,"visible":true,"origin":"","legend":"","description":"","filename":"Table3.docx","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/df884759e1b368d5a9c3ac86.docx"},{"id":87420460,"identity":"b2e669c1-0b01-4c20-8f3b-51664483610c","added_by":"auto","created_at":"2025-07-23 15:24:51","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":30064,"visible":true,"origin":"","legend":"","description":"","filename":"Table4.docx","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/f99981b48e86f7b9dc52d510.docx"},{"id":87420993,"identity":"677dc6d9-85c3-497e-ae40-803bc1d1d486","added_by":"auto","created_at":"2025-07-23 15:32:51","extension":"docx","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":27723,"visible":true,"origin":"","legend":"","description":"","filename":"Table5.docx","url":"https://assets-eu.researchsquare.com/files/rs-7078540/v1/977ceb76f07ed8bc7b59d8e1.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Automated Evidence Synthesis of Cognitive Interventions in Older People using the CogTale Platform: A Validation Study","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eCognitive decline is a hallmark of aging and poses a significant challenge to maintaining quality of life in older adults. Cognition-oriented treatments (COTs) address the cognitive and functional impairments commonly seen in aging populations. Interventions like cognitive stimulation therapy (CST), cognitive training (CT), and cognitive rehabilitation (CR) hold promise for mitigating age-related cognitive decline and enhancing functional capacity, making them critical areas of focus within older adult research\u0026mdash;the interdisciplinary field exploring the biological mechanisms of aging and age-related diseases. CST is a structured and manualised intervention focused on general cognitive stimulation and discussion of various topics in the context of activities, usually in a group (Bahar-Fuchs et al., \u003cspan class=\"CitationRef\"\u003e2013\u003c/span\u003e; (Woods et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). CT focuses on the formal training of cognitive abilities (e.g., memory, attention) through practice on tasks (often game-like) or a strategy-based approach (Bahar-Fuchs et al., \u003cspan class=\"CitationRef\"\u003e2014\u003c/span\u003e; (Bahar-Fuchs et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). CR is a goal-oriented, enablement-based approach using various strategies to improve performance on personally relevant functional goals (Clare, \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e; (Clare \u0026amp; Woods, \u003cspan class=\"CitationRef\"\u003e2004\u003c/span\u003e; (Kudlicka et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). These interventions aim to improve cognitive performance and contribute to overall well-being. However, the effectiveness of these interventions often depends on synthesising evidence across diverse studies to identify robust patterns and guide clinical application. Systematic reviews and meta-analyses (SRMAs) serve this critical role.\u003c/p\u003e\n\u003cp\u003eSRMAs represent the highest standard for synthesising evidence, forming the foundation of evidence-based medicine (EBM). EBM integrates clinical expertise with rigorous research to inform patient care decisions effectively (Evans, \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e; (Sackett, \u003cspan class=\"CitationRef\"\u003e1997\u003c/span\u003e). Organisations like the Cochrane Collaboration exemplify EBM principles by producing rigorously compiled SRMAs that support clinical decision-making (Tanjong-Ghogomu et al., \u003cspan class=\"CitationRef\"\u003e2009\u003c/span\u003e). SRMAs synthesise evidence in a highly structured manner, offering insights beyond individual studies, making them critical for clinical decision-making, guideline development, and evidence appraisal (Garc\u0026iacute;a-Perdomo, \u003cspan class=\"CitationRef\"\u003e2016\u003c/span\u003e; (Paul \u0026amp; Leibovici, \u003cspan class=\"CitationRef\"\u003e2014\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eDespite their importance, SRMAs are complex, labour-intensive, and time-consuming (Marshall et al., \u003cspan class=\"CitationRef\"\u003e2018\u003c/span\u003e). Nearly a quarter of SRMAs become outdated within two years of publication (Shojania et al., \u003cspan class=\"CitationRef\"\u003e2007\u003c/span\u003e). Therefore, accelerating the SRMA production process via automation of key steps is essential to effectively maintain pace with the burgeoning body of academic literature (van Dinter et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e). Christopoulou (Christopoulou, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e) highlights areas of advancement in the automation of SRMAs, particularly in tasks such as study selection and data extraction, with tools like the Systematic Review Toolbox demonstrating effectiveness in supporting evidence synthesis (Johnson et al., \u003cspan class=\"CitationRef\"\u003e2022\u003c/span\u003e). However, fully automated meta-analysis tools, especially for data synthesis and interpretation, are still nascent and require further development and validation. One such automation effort is the development of CogTale, an online platform capable of streamlining the evidence synthesis process (Sabates et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eCogTale is a comprehensive database and repository of trial evidence on the effects of structured cognition-oriented treatments (COTs) for older adults, including cognitive stimulation therapy, cognitive training, and cognitive rehabilitation (Sabates et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e). It aims to provide researchers, clinicians and consumers with clear and unbiased evidence-based information to help answer questions about COTs, including through evaluation of the methodology and findings from single studies, the production of plain language summaries of evidence (i.e., \u0026ldquo;Citizen Briefings\u0026rdquo;), and the rapid synthesis of the evidence through meta-analyses. Users can refine their searches of the database using various filters related to the intervention (e.g., specific type and ingredients, dose, duration, etc.), the population (e.g., older adults with dementia, mild cognitive impairment, etc.), study design (e.g., type of control, type of randomisation, outcomes assessed, etc.), analysis of results, and methodological quality scores among others, allowing for highly curated queries. The platform generates detailed reports sent to the user based on their meta-analytic query. Importantly, the meta-analytic functionality on CogTale awaits validation and the extent to which rapid meta-analytic queries produced on the CogTale platform can replicate the findings from rigorous and high-quality SMRAs, such as those published by the Cochrane Database of Systematic Reviews, remains to be established.\u003c/p\u003e\n\u003cp\u003eAccordingly, the current study aims to evaluate the extent to which the CogTale platform can replicate the findings from three Cochrane Reviews focused on key COTs, namely, CST (Woods et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e), CT (Bahar-Fuchs et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e), and CR (Kudlicka et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e details the methodological features and main findings of three recent Cochrane Reviews of CST CT, and CR.\u003c/p\u003e\n\u003cp\u003e[Insert Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e\n\u003cp\u003eIn discussing issues related to scientific replication, Hedges (Hedges, \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e) and Hedges and Schauer (Hedges \u0026amp; Schauer, \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e) propose a distinction between an \u003cem\u003eexact replication\u003c/em\u003e, in which findings (e.g., effect estimates) are identical in the replication studies, and an \u003cem\u003eapproximate replication\u003c/em\u003e, which is more lenient and allows for minor differences. They further distinguish formulating a null hypothesis according to which replication \u003cem\u003ehas not\u003c/em\u003e occurred (burden of proof is demonstrating replication) and formulating a null hypothesis according to which replication \u003cem\u003ehas occurred\u003c/em\u003e (burden of proof is demonstrating replication has \u003cem\u003enot\u003c/em\u003e occurred)\u003c/p\u003e\n\u003cp\u003eIn attempting to replicate the findings from SMRAs, exact replication appears unrealistic when different methodologies are used, particularly when variations in the included set of studies or data availability are present. Accordingly, the null hypothesis that the replication was successful is appropriate in the current study, in which the goal is to assess whether the meta-analytic findings produced by CogTale are broadly consistent with those of published Cochrane Reviews.\u003c/p\u003e\n\u003cp\u003eWe expect the findings from CogTale to approximately replicate those of the Cochrane reviews. In practical terms, we hypothesise that the findings from CogTale will not be substantially different from those of recently published Cochrane Reviews of COTs. Specifically, concerning the key/main outcomes in each of the three reviews, we hypothesise that the direction (favouring experimental vs. control), strength (small, medium, large), and levels of certainty in the evidence (high, moderate, low, very low) in the effect estimates will be similar when evaluated on the CogTale platform. Furthermore, we hypothesise that the absolute value of effect estimates generated by the CogTale platform will not differ significantly from the absolute value of effect estimates found in the three Cochrane Reviews.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\n \u003ch2\u003e2.1. Replication Sample\u003c/h2\u003e\n \u003cp\u003eThe replication sample comprised effect estimates reported in three Cochrane Reviews of COTs for older adults with dementia (Bahar-Fuchs et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e; (Kudlicka et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e; (Woods et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). These reviews were selected as they are each regarded as the most authoritative and methodologically rigorous SMRAs of the key COT approaches for people with dementia, and all primary studies included in these reviews were either already included or eligible for inclusion on the CogTale platform. Because of the large number of outcomes included in these reviews, the current replication effort focused on reproducing the results of the first \u0026quot;Summary of Findings\u0026quot; (SoF) table in each Cochrane review, comparing the relevant treatment (CST, CT, or CR) to control conditions on up to seven key treatment outcomes immediately at the end of the treatment period, and including medium-term outcomes for CR.\u003c/p\u003e\n \u003cp\u003eFor each outcome eligible for inclusion in the replication, we extracted the number of participants and studies, standardised mean differences (SMDs) and 95% confidence intervals (CIs) of SMD, study weight, and GRADE certainty in the evidence. SMDs were then converted to Hedges\u0026apos; g according to guidance provided by Hedges (Hedges, \u003cspan class=\"CitationRef\"\u003e1981\u003c/span\u003e). After extracting relevant outcomes and metrics, we mapped outcomes to predefined outcome categories within CogTale\u0026apos;s classification system to determine whether the effect estimate could be produced on the CogTale platform. Importantly, the a priori classification system used in the CogTale platform ensures consistency in outcome classification, supporting systematic comparisons.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\n \u003ch2\u003e2.2. CogTale data processing flow and preparation for replication\u003c/h2\u003e\n \u003cp\u003eThe data processing workflow on the CogTale platform involves manual and automated steps. Detailed descriptions of the CogTale information processing pipeline, statistical methodologies, and evidence grading procedures are described elsewhere (Sabates et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e) and summarised in the Swim Lane Diagram in Fig. 1. Briefly, trained human coders manually completed data extraction from eligible studies using CogTale\u0026rsquo;s comprehensive data extraction form, which covers detailed information about the study design, including measures and outcomes, characteristics of experimental and control interventions, and study results. Methodological quality indices, including scores on the PEDro scale (Maher et al., \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e), the Jadad scale (Jadad et al., \u003cspan class=\"CitationRef\"\u003e1996\u003c/span\u003e), and the Cochrane Risk of Bias tool (ROB1.0) (Higgins et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e; (Higgins et al., \u003cspan class=\"CitationRef\"\u003e2011\u003c/span\u003e) are then automatically computed based on a custom automated scoring algorithm.\u003c/p\u003e\n \u003cp\u003eWhere relevant raw data for an outcome are provided in manuscripts undergoing data extraction (e.g., sample size (\u003cem\u003en\u003c/em\u003e), means, and standard deviations (SDs)), the platform automatically calculates the Hedges\u0026rsquo; g as a measure of effect estimate. When the meta-analysis function in the CogTale platform is used, variance-weighted estimates are pooled and corrected. A random-effects meta-analysis of the selected studies is carried out using restricted maximum likelihood (REML) (Langan et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e) and the Hartung-Knapp-Sidik-Jonkman (HKSJ) method for CIs (Hartung \u0026amp; Knapp, \u003cspan class=\"CitationRef\"\u003e2001\u003c/span\u003e; (Sidik \u0026amp; Jonkman, \u003cspan class=\"CitationRef\"\u003e2007\u003c/span\u003e). Findings of meta-analytic queries are provided in reports generated via R Markdown, summarising for each broad and specific outcome the effect estimate and methodological quality of primary studies, the pooled treatment estimates with heterogeneity and significance testing, and the certainty of the evidence.\u003c/p\u003e\n \u003cp\u003e[Insert Fig. 1]\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\n \u003ch2\u003e2.3. Replication Anslysis\u003c/h2\u003e\n \u003cp\u003eThe feasibility of replication within CogTale was assessed based on the results reported in each article and the existing outcome classification categories in CogTale. Replicability was determined by reviewing the individual studies included in the analyses for each outcome presented in the Cochrane Summary of Findings (SoF) tables. To be included in the replication, studies had to report raw values (Means, SDs) for the outcome of the experimental and control conditions at minimum at baseline and post-treatment. Studies were excluded from the replication analysis if they reported data unsupported by the CogTale platform, such as reporting change scores without baseline data or missing means and standard deviations. Once data extraction was completed, a meta-analysis of each included outcome using the CogTale platform was conducted. Key information was extracted, including the number of participants and studies, Hedges\u0026apos; g values and their CIs, study weights, and certainty in the evidence from the reports automatically generated by the platform. Comparatively, Cochrane reviews utilise the GRADE framework (Sch\u0026uuml;nemann et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e) to assess the certainty in the evidence and apply standardized tools such as RevMan and random-effects models to synthesize data, often incorporating methods like the HKSJ approach for handling between-study heterogeneity\u003c/p\u003e\n \u003cp\u003eThe replication analysis adhered to the \u003cem\u003eapproximate replication\u003c/em\u003e framework proposed by Hedges and Schauer (Hedges \u0026amp; Schauer, \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e), which assumes replication unless substantial evidence suggests otherwise. This framework recognised inherent differences between the Cochrane and CogTale methodologies, frameworks, and evidence grading systems, and does not aim for \u003cem\u003eexact\u003c/em\u003e replication. Instead, it focused on assessing the degree of alignment between these systems, while also interpreting any observed differences. To operationalise this approach, we evaluated replication based on five predefined criteria, allowing for methodological flexibility to accommodate minor discrepancies:\u003c/p\u003e\u003cspan\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e1 Direction of the estimate\u003c/span\u003e. We compared the direction of each pair of effect estimates. Replication was assumed when the direction of the effect estimates was aligned.\u003cbr\u003e\u003c/span\u003e\u003cspan\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e2 Equivalence of point estimate\u003c/span\u003e. For each outcome, we calculated the standardized difference between each pair of effect estimates (Cochrane and CogTale) using z-tests, with the combined variance of the estimates derived from their respective standard errors. Replication was assumed when there was no significant standardized difference defined as p\u0026thinsp;\u0026lt;\u0026thinsp;0.00278. The significance threshold was adjusted using the Bonferroni correction to account for multiple comparisons. The original significance level (\u0026alpha;\u0026thinsp;=\u0026thinsp;0.05) was divided by the number of outcomes (k\u0026thinsp;=\u0026thinsp;18).\u003cbr\u003e\u003c/span\u003e\u003cspan\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e3 Overlapping 95% CIs\u003c/span\u003e. The 95% CIs for each pair of effect estimates were compared. Replication was assumed when there was an overlap of the 95% CIs.\u003cbr\u003e\u003c/span\u003e\u003cspan\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e4 Effect magnitude\u003c/span\u003e. The magnitudes of each pair of effect estimates were compared and interpreted according to a modified system informed by Cohen\u0026apos;s (Cohen, \u003cspan class=\"CitationRef\"\u003e1969\u003c/span\u003e) and Brydges\u0026rsquo; (Brydges, \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e) guidelines as small (\u0026le;\u0026thinsp;0.3), medium (\u0026thinsp;\u0026gt;\u0026thinsp;=\u0026thinsp;0.3 to \u0026lt;\u0026thinsp;0.8), or large (\u0026ge;\u0026thinsp;0.8). While these thresholds provide a general framework for categorization, we acknowledge that they should not be interpreted as rigid cut-offs. For example, differences near the boundaries (e.g., 0.29 vs. 0.31) are evaluated within the context of the broader replication criteria. Replication was assumed unless the magnitudes of the effect estimates differed substantially, defined as being two categories apart (e.g., small vs. large). A one-category difference was acceptable if all other criteria supported replication.\u003cbr\u003e\u003c/span\u003e\u003cspan\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e5 Certainty in the evidence\u003c/span\u003e. Certainty in the evidence was compared between Cochrane and CogTale. Cochrane\u0026apos;s certainty in the evidence was assessed using the GRADE framework, which grades the certainty of evidence from very low to high by considering factors that may decrease or increase confidence (Sch\u0026uuml;nemann et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). CogTale employs a custom evidence grading algorithm, combining information about the magnitude of the effect, mean methodological quality of the included studies, heterogeneity, and sample size, based on which ratings of low, moderate, or high certainty are produced. Replication was assumed unless substantial differences in certainty in the evidence ratings were found\u0026mdash;defined as two levels apart. Despite these methodological differences, evaluating alignment between the systems was critical in understanding CogTale\u0026rsquo;s ability to approximately replicate Cochrane findings and identify areas for improvement.\u003cbr\u003e\u003c/span\u003e\n \u003cp\u003eIn evaluating the replicability of findings from the Cochrane Reviews using the CogTale platform, we employed two thresholds to interpret replication outcomes. The \u003cem\u003estringent threshold\u003c/em\u003e classified replication as unsuccessful if \u003cem\u003eany\u003c/em\u003e of the five criteria were violated. In contrast, the \u003cem\u003elenient threshold\u003c/em\u003e classified replication as unsuccessful only when \u003cem\u003emore than one\u003c/em\u003e criterion was violated. These thresholds were designed to explore varying levels of alignment between the two systems under different conditions.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"3. Results","content":"\u003cp\u003eOf the 72 studies included across the three Cochrane Reviews, 67 (93.05%) were included in the meta-analysis of at least one outcome, and 61 studies (84.72%) were included on the CogTale platform. Of the 10 studies that were not included, one (De Vreese \u0026amp; Neri, 1999) could not be sourced, and nine (Alvares-Pereira et al., 2021; (Beck et al., 1988; (Breuil et al., 1994; (Brueggen et al., 2017; (Goudour et al., 2011; (Heiss et al., 2008; (Kallio et al., 2018; (Marinho et al., 2021; (Rai et al., 2021) \u0026nbsp;did not report raw values (means, SDs) for the outcome of the experimental and control conditions at baseline and post-treatment.\u003c/p\u003e\n\u003cp\u003eOf the 21 effect estimates/outcomes reported in the SoF tables of the three Cochrane reviews, 18 were included in the replication study. Two outcomes concerning CT\u0026mdash;Participant Burden (retention rates) and Change in Mood and Well-being (caregiver)\u0026mdash;were excluded due to the absence of a corresponding outcome in CogTale for the former and only one study in the latter, below CogTale\u0026apos;s threshold of three studies required for a meta-analysis. One outcome concerning CR, self-efficacy (participant self-report), was excluded as it included only two studies.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eApplying the stringent threshold, approximate replication was observed for 11 of 18 outcomes (61.11%), while 16 of the 18 outcomes (88.89%) met the criteria for replication under the lenient threshold. For 17 of the 18 outcomes (94.4%), the direction of the estimates aligned, and 95% confidence intervals overlapped, indicating replication. Additionally, no statistically significant differences were detected in any of the point effect estimates, further supporting the assumption of replication. Effect magnitudes were consistently classified for thirteen outcomes (72.22%), differed by one category for four outcomes (22.22%) and differed substantially for one outcome (5.56%). Certainty in the evidence was identical for seven outcomes (38.89%), varied by one category for seven outcomes (38.89%), and differed substantially for four outcomes (22.22%). Table 2 summarises the replication criteria and decisions concerning the 18 outcomes included in the replication. Table 3 shows the z-score values and comparison between Cochrane and Cogtale effect estimate magnitude and certainty in the evidence. Figure 2 provides a forest plot comparing Cochrane and CogTale effect size estimates with 95% CIs.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e[ Insert Table 2]\u003c/p\u003e\n\u003cp\u003e[Insert Table 3]\u003c/p\u003e\n\u003cp\u003e[ Insert Figure 2]\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe assessed\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eseven outcomes concerning the effect of cognitive stimulation. Of the 34 studies that contributed to these outcomes in Woods et al.\u003csup\u003e1\u003c/sup\u003e, 29 (85.29%) were included in the CogTale sample. Approximate replication was achieved for all seven outcomes under our stringent and lenient thresholds. No significant differences were observed in the point effect estimates; all 95% CIs overlapped. Effect magnitudes were consistently classified for five outcomes (71.43%) and differed by one category for two outcomes (28.57%). Certainty in the evidence was identical for four outcomes (57.14%) and varied by one category for three outcomes (42.86%).\u0026nbsp;\u003c/p\u003e\n\u003ch3\u003e3.1. \u0026nbsp;Cognitive Training\u003c/h3\u003e\n\u003cp\u003eWe assessed five outcomes concerning the effect of cognitive training. Of the 32 studies that contributed to these outcomes in Bahar-Fuchs et al.\u0026nbsp;(Bahar-Fuchs et al., 2019), 26 (81.25%) were included in the CogTale sample. Under the stringent threshold, approximate replication was achieved for two of the five outcomes (40%). In contrast, under the lenient threshold, approximate replication was achieved for all five outcomes (100%). For all outcomes the direction of the estimates aligned, no significant differences were observed in the point effect estimates and 95% CIs overlapped. Effect magnitudes were consistently classified for three outcomes (60%) and differed by one category for two outcomes (40%). Certainty in the evidence was identical for one outcome (20%), varied by one category for one outcome (20%), and differed substantially for three outcomes (60%). The three outcomes that were not replicated\u0026mdash;Change in Capacity for Activities of Daily Living, Change in Delayed Memory, and Change in Participants\u0026apos; Mood\u003cstrong\u003e\u003cem\u003e\u0026mdash;\u003c/em\u003e\u003c/strong\u003eare further explored below.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.1.1. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Training: Change in Capacity for Activities of Daily Living\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOf the ten studies (687 participants) that contributed to this outcome in Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), six (60%) were included in the CogTale sample, representing a total of 884 participants across all conditions. Differences in sample sizes owe to Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) reporting the sample size for only the intervention and control group, whereas CogTale reports the total sample size immediately post-intervention, across subgroups or conditions. Approximate replication of study-level effect estimates was observed in three (50%) studies (Bergamaschi et al., 2013; (Giuli et al., 2016; (Jelcic et al., 2012). However, discrepancies in effect size, direction, and magnitude indicated non-replication for specific studies. For example, Lee et al., (Lee et al., 2013), demonstrated overlapping 95% CIs but substantial differences in effect estimate magnitude (large vs. small) and misaligned effect estimate direction. Similarly, Galante et al., (Galante et al., 2007) and Amieva et al (Amieva et al., 2016)., showed overlapping 95% CIs but divergent effect directions. The analysis of the pooled estimates demonstrated small effect sizes with overlapping 95% CIs and aligned effect directions. However, substantial differences in certainty in the evidence were observed, with Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), reporting low certainty and CogTale reporting high certainty.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.1.2. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Training: Change in Delayed Memory\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOf the 11 studies (543 participants) analysed by Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019), nine (81.81%) were included in the CogTale sample, representing a total of 631 participants across all conditions. Approximate replication was observed in eight of the nine (88.89%) studies (Barban et al., 2016; (Boller et al., 2012; (Davis et al., 2001; (Jelcic et al., 2014; (Jelcic et al., 2012; (Mapelli et al., 2013; (Quayhagen et al., 2000; (Trebbastoni et al., 2018). Study-level discrepancies in the direction of effect estimates, combined with discrepancies in the certainty in the evidence ratings observed in the analysis of the pooled estimates, indicated nonreplication. For example, Cahn-Weiner (Cahn-Weiner et al., 2003) demonstrated effect estimates were classified within the same category with overlapping 95% CIs; however, the effect estimate direction was misaligned. The analysis of the pooled estimates demonstrated effect size differences of one category, with Cochrane indicating a large effect and CogTale reporting a medium effect. Both analyses demonstrated overlapping 95% CIs and aligned effect directions. However, substantial differences in certainty in the evidence were observed, with Cochrane indicating very low certainty and CogTale reporting moderate certainty.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.1.3. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Training: Change in Participants\u0026apos; Mood\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOf the eight studies contributing to this outcome in Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) \u0026nbsp;(577 participants), all eight (100%) were included in the CogTale sample representing 1,044 participants across all conditions. Approximate replication was observed in five (62.5%) studies (Amieva et al., 2016; (Bergamaschi et al., 2013; (Davis et al., 2001; (Galante et al., 2007; (Lee et al., 2013). However, discrepancies in effect magnitude, 95% CIs, and effect direction indicated non-replication in several cases. For instance, Galante (Galante et al., 2007) reported overlapping 95% CIs but demonstrated substantial differences in effect size magnitude (large vs. small) and misaligned effect directions. Similarly, Giuli (Giuli et al., 2016) demonstrated a significant z-score, no overlap in 95% CIs, divergent effect directions, and substantial differences in effect size magnitude (large vs. small). The analysis of the pooled estimate indicated that effect estimates were classified within the same category, with overlapping 95% CIs and aligned effect estimate direction. However, differences in certainty in the evidence were notable: Bahar-Fuchs et al. (Bahar-Fuchs et al., 2019) rated the certainty as very low, whereas CogTale reported moderate certainty.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003e3.2. \u0026nbsp;\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eCognitive Rehabilitation\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll six studies included in Kudlicka et al.\u0026nbsp;(Kudlicka et al., 2023) were also included in the CogTale sample (100%). Approximate replication was achieved for two outcomes (33.33%) when using the stringent threshold, and four outcomes (66.66) when using the lenient threshold. The direction of the estimates aligned for all outcomes except one (16.67%), Quality of life (participant self-report). No significant differences in point effect estimates were detected; however, one (16.67%) outcome\u0026mdash;Functional ability in targeted activities: personal goals - performance (PR)\u0026mdash;demonstrated a non-overlapping 95% CI. The 95% CIs overlapped for the remaining five outcomes (83.33%). Effect magnitudes were classified within the same category for five outcomes (83.33%) and differed substantially for one outcome (16.67%). Certainty in the evidence ratings were identical for one outcome (16.67%), differed by one category for three outcomes (50%), and differed substantially for two outcomes (33.33%). The four outcomes that were not replicated\u0026mdash;Functional ability in targeted activities: personal goals - performance (PR), General functional ability (IR), Mood: depression (PT), and Quality of life (PR)\u003cstrong\u003e\u003cem\u003e\u0026mdash;\u003c/em\u003e\u003c/strong\u003eare further explored below.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2.1. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Rehabilitation: Functional Ability in Targeted Activities: Personal Goals - Performance (PR)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eKudlicka et al. (Kudlicka et al., 2023) analysed three studies with 501 participants, and CogTale reviewed three with 531 participants across all conditions. Approximate replication was not achieved in two (66.6%) studies (Clare et al., 2010; (Hindle et al., 2018). Discrepancies in 95% CIs, significant differences in the point effect estimates, and certainty in the evidence indicated nonreplication. For example, Clare et al. (Clare et al., 2019)\u0026mdash;despite demonstrating an aligned effect estimate direction, showed no overlap in 95% CIs and a significant z-score. The analysis of the pooled estimates results demonstrated overlapping 95% CIs and significant differences in the point effect estimates. Kudlicka et al. (Kudlicka et al., 2023) rated the certainty in the evidence as high, while CogTale rated it moderate.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2.2. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Rehabilitation: General functional ability (IR)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eKudlicka et al. (Kudlicka et al., 2023) analysed three studies with 673 participants across intervention and control conditions, and CogTale reviewed three with 974 participants across all conditions. Approximate replication was achieved in all three studies (100%). All effect estimates were aligned, with overlapping 95% CIs; moreover, there were no significant z-scores. At the study level, Thivierge et al. (Thivierge et al., 2014) indicated a one-category difference in effect estimate magnitude. In the analysis of the pooled estimates, substantial differences in certainty in the evidence indicated nonreplication. Kudlicka et al. (Kudlicka et al., 2023) rated the certainty in the evidence as low, while CogTale rated it high.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2.3. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Rehabilitation: Mood: Depression (Participant Self-Report)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eKudlicka et al. (Kudlicka et al., 2023) analysed three studies with 502 participants, and CogTale reviewed three with 543 participants across all conditions, representing 100% of the Cochrane sample. Approximate replication was achieved for one (33.33%) study (Clare et al., 2010). Differences in effect size magnitudes, 95% CIs, and certainty in the evidence contributed to non-replication. For example, Clare et al. (Clare et al., 2019) exhibited overlapping 95% CIs but misaligned effect directions, resulting in non-replication. Moreover, Hindle et al. (Hindle et al., 2018) demonstrated aligned effect directions but no overlap in 95% CIs, leading to non-replication. In the analysis of the pooled estimates, the results demonstrated overlapping 95% confidence intervals (CIs) and consistent alignment in the direction of the effects. However, there was a substantial difference in effect size magnitude (large vs. small) and certainty in the evidence with Kudlicka et al. (Kudlicka et al., 2023) reporting very low certainty and CogTale indicating moderate certainty.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2.4. \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCognitive Rehabilitation: Quality\u0026nbsp;of\u0026nbsp;life\u0026nbsp;(PR)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eKudlicka et al.\u0026nbsp;(Kudlicka et al., 2023) analysed five studies with 853 participants across intervention and control conditions, and CogTale reviewed four with 582 participants across all conditions, representing (80%) of the Cochrane sample. Approximate replication was only achieved in only one study (25%). Only one effect estimate was aligned; however, all 95% CIs overlapped. No significant z-scores were detected. All effect size magnitudes were classified as the same category. At the study level, Thivierge et al.\u0026nbsp;(Thivierge et al., 2014) indicated a one-category difference in effect estimate magnitude. The analysis of the pooled estimated substantial differences in certainty in the evidence indicated nonreplication. Kudlicka et al.\u0026nbsp;(Kudlicka et al., 2023) rated the certainty in the evidence as low, while CogTale rated it high.\u003c/p\u003e\n\u003cp\u003eTable 3 shows a summary of the replication criteria and decisions in relation to outcomes that indicated nonreplication. Table 4 provides the z-score values and comparison between Cochrane and Cogtale effect estimate magnitudes. Figure 3 provides a forest plot comparing Cochrane and CogTale study level effect size estimates with 95% CIs for the identified outcomes that indicated nonreplication.\u003c/p\u003e\n\u003cp\u003e[Insert Table 4]\u003c/p\u003e\n\u003cp\u003e[ Insert Table 5]\u003c/p\u003e\n\u003cp\u003e[ Insert Figure 3]\u003c/p\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThis study evaluated the extent to which the CogTale platform could replicate key findings from recent Cochrane Reviews of COTs for individuals with dementia. Specifically, the study examined whether CogTale could generate effect estimates aligned with Cochrane Reviews in terms of effect direction, absolute values, magnitude, and certainty in the evidence. By applying both stringent and lenient replication thresholds, this research not only validated aspects of CogTale's meta-analytic functionality but also highlighted its capacity to produce timely, high-quality evidence syntheses. These findings underscore CogTale's potential as a valuable tool for addressing the challenges of traditional systematic reviews and meta-analyses, particularly their resource intensity and susceptibility to rapid obsolescence in the context of the rapidly growing body of literature on aging-related interventions\u003c/p\u003e\u003cp\u003eDepending on the strictness of the replication criteria applied, Cochrane findings were replicated on the CogTale platform to varying degrees. Under the lenient threshold, most outcomes\u0026mdash;close to nine out of ten\u0026mdash; were deemed replicated. In contrast, when a more stringent threshold was applied, approximately two-thirds of outcomes met the replication criteria. CST outcomes aligned closely with Cochrane results, whereas CT and CR outcomes showed more variability. Discrepancies were evident in CT and CR outcomes, where effect size magnitudes, direction, and certainty in the evidence ratings diverged.\u003c/p\u003e\u003cp\u003eNotably, CogTale and Cochrane were aligned on the primary outcomes for CST and CT. For CST, the primary outcome \"Cognition\" was approximately replicated under the stringent threshold, demonstrating consistent findings between the two methodologies. Similarly, for CT, the primary outcome \"Change in a global measure of cognition (composite)\" was also approximately replicated under the stringent threshold. These results highlight CogTale's ability to align with Cochrane\u0026rsquo;s findings for the main outcomes of these interventions. However, for CR, the primary outcome \"Functional ability in targeted activities: personal goals \u0026ndash; performance (PR)\" was not approximately replicated, as the 95% confidence intervals did not overlap.\u003c/p\u003e\u003cp\u003eWhile some variability was observed for secondary outcomes, these measures often differ across studies and are not always the primary focus of the interventions. This suggests that the alignment of primary outcomes, such as cognition for CST and CT, and functional ability for CR, represents a meaningful and important consistency that warrants emphasis despite other divergences. These differences suggest that while CogTale partially aligns with Cochrane\u0026rsquo;s results, methodological variations, such as differences in evidence grading and included study samples, may influence replication accuracy.\u003c/p\u003e\u003cp\u003eNonreplication in this study likely arises from several factors. First, differences in the study samples included in CogTale and those included in Cochrane Reviews arose from select primary studies lacking the raw data (means, standard deviations) required by CogTale. Approximately one-fifth of studies for CT and just over one-seventh for CST were excluded, potentially contributing to variability in replicability. For CST, approximate replication was achieved for all seven assessed outcomes under both stringent and lenient thresholds, demonstrating strong alignment with Cochrane findings. In contrast, for CT, only two of five outcomes met the stringent replication threshold, though all five outcomes were replicated under the lenient threshold. Cognitive rehabilitation (CR), which included all six studies from Cochrane, demonstrated approximate replication for only two of six outcomes under the stringent threshold and four outcomes under the lenient threshold.\u003c/p\u003e\u003cp\u003eThese findings suggest that while data availability influenced replicability for CST and CT, methodological differences, such as evidence grading and analysis approaches, may have played a greater role in the variability observed for CR outcomes. Variations in evidence grading methodologies\u0026mdash;Cochrane\u0026rsquo;s utilisation of GRADE versus CogTale\u0026rsquo;s custom evidence grading algorithm\u0026mdash;could account for divergent certainty in the evidence ratings. Additionally, methodological differences, such as manual data coding in CogTale, increase variability and the potential for human error, although this process is not unique to CogTale. Similar manual data extraction methods are employed in Cochrane reviews, where review authors independently extract data using structured forms and resolve disagreements through consensus. While both approaches rely on human input, CogTale coders undergo comprehensive training and their performance is regularly monitored by platform admins, aiming to standardise data entry and minimise errors.\u003c/p\u003e\u003cp\u003eThe current findings suggest that while CogTale demonstrates promise as a tool for automated evidence synthesis in COTs, its approach differs from the more conservative and methodologically restrictive standards of Cochrane. Rather than holding CogTale to Cochrane\u0026rsquo;s stringent benchmarks, this study highlights the potential benefits of CogTale\u0026rsquo;s more adaptable framework, particularly its ability to provide timely, user-friendly, and scalable evidence synthesis that can accommodate emerging data and evolving methodologies. CST, which had the largest sample size, showed strong alignment with Cochrane findings, instilling confidence in CogTale\u0026rsquo;s potential to provide credible evidence in robust datasets. However, replication was mixed for some CT and CR outcomes, highlighting the need for caution when interpreting results in domains with limited data.\u003c/p\u003e\u003cp\u003eCogTale offers a unique opportunity to both align with established evidence synthesis frameworks and highlight gaps in traditional approaches. Its flexibility, user-friendly design, and nimble ability to rapidly process data set it apart from established methods such as Cochrane, which can be constrained by rigid and time-intensive processes. By leveraging adaptive evidence grading, integrating diverse data sources, and embracing innovative methodologies, CogTale provides a dynamic platform that fosters confidence in findings\u0026mdash;even in contexts where established systems may fall short, such as rapidly evolving fields, or areas requiring real-time updates and adaptability. Strengthening trust in the system can be further achieved by enhancing transparency in algorithms for effect size estimation and certainty ratings, including comprehensive raw data, and refining outcome classification schemes. These improvements would underscore the system\u0026rsquo;s potential to provide reliable and accessible insights for clinicians and researchers. Additionally, ongoing training, calibration, and quality-control checks for human coders remain essential to ensure fidelity and reduce variability in the data extraction process. Nevertheless, for clinicians and stakeholders, CogTale offers significant potential to streamline evidence synthesis and decision-making, provided its limitations are carefully considered.\u003c/p\u003e\u003cp\u003eOur findings also underscore the potential benefits and current challenges of increasing automation in systematic meta-reviews. In a previous paper, Rasool et al. (Rasool et al., \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) demonstrated the feasibility of using large language models (LLMs) to expedite data extraction, thereby automating and increasing the speed of the evidence synthesis pipeline. However, their findings were mixed, highlighting that while LLMs show promise in streamlining data gathering, reducing human error, and enabling rapid updates to evidence syntheses, further work is required to address inconsistencies and optimize their integration into SRMA processes. While the present study did not incorporate LLM-based extraction within CogTale, the principle stands: employing artificial intelligence and LLMs can streamline data gathering, reduce human error, and expedite the development of \u0026ldquo;living\u0026rdquo; evidence syntheses. By leveraging these technologies, platforms like CogTale can be iteratively refined, moving closer to near-real-time evidence updates. Such advancements could significantly improve the responsiveness of evidence-based recommendations in dementia care and beyond.\u003c/p\u003e\u003cp\u003eGiven their rigorous methodological standards, this study\u0026rsquo;s use of Cochrane reviews as a validation benchmark is a notable strength, particularly due to their systematic approach and high-quality synthesis of randomised controlled trials. However, it is also essential to recognise the limitations of the Cochrane approach, including their often-limited scope, time-intensive processes, and the potential for excluding important contextual or non-randomised evidence. These constraints highlight the need for complementary methods, such as those offered by CogTale, to address broader clinical questions, incorporate emerging data, and provide more accessible, real-time insights that extend beyond Cochrane\u0026rsquo;s traditionally narrow focus. CogTale\u0026rsquo;s ability to replicate findings across diverse COTs highlights its versatility and scalability, demonstrating robustness even when data was incomplete. However, the exclusion of slightly less than one-fifth of Cochrane studies due to data unavailability introduces potential bias, as the missing studies might alter overall interpretations. A further limitation is the focus on replicating only the first Summary of Findings (SoF) table from each Cochrane review rather than the full range of analyses, limiting the comprehensiveness of the validation. Additionally, reliance on manual data coding is time- and labour-intensive and introduces variability.\u003c/p\u003e\u003cp\u003eFuture research should prioritise several critical areas to enhance the reliability and utility of CogTale and similar platforms. Data completeness and transparency must be improved by encouraging open data practices ensuring the availability and standardisation of raw data in primary studies. Increased data availability will enhance CogTale\u0026rsquo;s capacity to replicate established findings accurately. The data and code used in this study to reproduce the findings are accessible in the public repository the Open Science Framework. Methodological harmonisation is another key area, with efforts needed to align CogTale\u0026rsquo;s evidence grading procedures more closely with widely accepted frameworks, such as GRADE. Incorporating automated quality assessment tools and consensus-driven approaches to evidence weighting could further reduce discrepancies in certainty in the evidence ratings. Moreover, the integration of advanced automation also requires attention, with future iterations of CogTale leveraging LLMs, machine learning algorithms, and other AI-driven tools for data extraction and synthesis. These technologies could accelerate the development of \"living\" SRMAs that are continuously updated in the context of new evidence. Additionally, user education is essential to enhance trust and utility. Clear communication about how CogTale\u0026rsquo;s results are generated, alongside training modules, online tutorials, and decision aids, can help clinicians, researchers, and policymakers interpret the platform's outputs appropriately. Given the existing and comprehensive eCourse provided in the training of human coders, important steps have already been made in this direction. Finally, ongoing validation efforts should aim to replicate a broader array of outcomes, including all Summary of Findings tables from the three Cochrane Reviews. Incorporating other high-quality systematic reviews and conducting independent validation studies will further strengthen CogTale\u0026rsquo;s credibility as a tool for evidence synthesis.\u003c/p\u003e\u003cp\u003eThis study demonstrated that CogTale, as an automated evidence synthesis platform, can approximate the rigorous findings of Cochrane Reviews for COTs in dementia, offering a promising tool for advancing older adult research. While some discrepancies were identified, they provide valuable insights into areas for methodological refinement, enhanced data availability, and the integration of more advanced automation tools. These findings underscore the platform\u0026rsquo;s potential to streamline evidence synthesis processes, enabling timely and reliable information to guide clinicians, researchers, and policymakers. By accelerating the synthesis of evidence and supporting dynamic guideline development, CogTale can contribute significantly to improving therapeutic strategies for age-related cognitive decline and enhancing the quality of care for older adults with dementia.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eConflicts of Interest\u003c/h2\u003e\n\u003cp\u003eA.B.F, B.M.H, S.B, T.D and S.S.S are the scientific developers of the CogTale platform, which was evaluated in this study. The other co-authors (S.H.R, C.C, I.J.M.B, J.C) have no competing interests to declare. The data used to support the findings of this study are available on the open science framework\u003c/p\u003e\n\u003ch2\u003eFunding declaration\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThis project was funded by National Institute on Aging R35AG072262 (awarded to B.M.H).\u003c/p\u003e\n\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\n\u003cp\u003eConceptualisation: A.B.F; R.H.S; C.C. Methodology: A.B.F; R.H.S; C.C. Data Curation: S.H.R; I.J.M.B; C.C; J.C. Formal Analysis: S.H.R. Funding Acquisition: A.B.F; B.M.H.. Project Administration: S.H.R; I.J.M.B; C.C. Supervision: A.B.F. Validation: S.H.R; A.B.F; C.C. Visualisation: S.H.R. Writing \u0026ndash;Original Draft: S.H.R; C.C. Writing \u0026ndash; Review \u0026amp; Editing: A.B.F; B.M.H; S.B; S.S.S; S.H.R; C.C. All authors approved the final submitted draft.\u003c/p\u003e\n\u003ch2\u003eAcknowledgement\u003c/h2\u003e\n\u003cp\u003eThe authors wish to dedicate this paper to the memory of co-author Prof Tzvi Dwolatzky, who sadly passed away in 2024. Prof Dwolatzky was a founding member of the CIDER Research Working Group and was involved in the development of the CogTale Platform from inception. He was a passionate clinician and researcher, committed to scientific research and discovery, but first and foremost, devoted to his patients, students, and mentees. He was a generous and kind individual who leaves a strong legacy, and will be sorely missed. The CogTale team would also like to acknowledge the contributions of a range of trainee coders through the CogTale internship who have assisted with establishing and growing the CogTale database.\u003c/p\u003e\n\u003ch2\u003eData Availability\u003c/h2\u003e\n\u003cp\u003eThe data used to support the findings of this study are available on the open science framework https://osf.io/vrzqu/?view_only=cc8353b8a8f24e769d3bec4217a59d9d\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAlvares-Pereira, G., V., S.-N. M.,and, \u0026amp; Spector, A. (2021). Validation of the cognitive stimulation therapy (CST) program for people with dementia in Portugal. \u003cem\u003eAging \u0026amp; Mental Health\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(6), 1019\u0026ndash;1028. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/13607863.2020.1836473\u003c/span\u003e\u003cspan address=\"10.1080/13607863.2020.1836473\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAmieva, H., Robert, P. H., Grandoulier, A. S., Meillon, C., De Rotrou, J., Andrieu, S., Berr, C., Desgranges, B., Dubois, B., Girtanner, C., Jo\u0026euml;l, M. E., Lavallart, B., Nourhashemi, F., Pasquier, F., Rainfray, M., Touchon, J., Ch\u0026ecirc;ne, G., \u0026amp; Dartigues, J. F. (2016). Group and individual cognitive therapies in Alzheimer's disease: the ETNA3 randomized trial. \u003cem\u003eInternational Psychogeriatrics\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(5), 707\u0026ndash;717. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1017/S1041610215001830\u003c/span\u003e\u003cspan address=\"10.1017/S1041610215001830\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBahar-Fuchs, A., Clare, L., \u0026amp; Woods, B. (2013). Cognitive training and cognitive rehabilitation for mild to moderate Alzheimer's disease and vascular dementia. \u003cem\u003eThe Cochrane database of systematic reviews\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e, CD003260. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.CD003260.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.CD003260.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBahar-Fuchs, A., Hampstead, B. M., \u0026amp; Clare, L. (2014). Cognitive training for older adults with MCI and mild dementia: State of the science, central challenges and possible solutions. \u003cem\u003eAlzheimer's \u0026amp; Dementia\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jalz.2014.04.138\u003c/span\u003e\u003cspan address=\"10.1016/j.jalz.2014.04.138\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBahar-Fuchs, A., Martyr, A., Goh, A., Sabat\u0026eacute;s, J., \u0026amp; Clare, L. (2019). Cognitive training for people with mild to moderate dementia. \u003cem\u003eThe Cochrane database of systematic reviews\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e, CD013069. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.CD013069.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.CD013069.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBarban, F., Annicchiarico, R., Pantelopoulos, S., Federici, A., Perri, R., Fadda, L., Carlesimo, G. A., Ricci, C., Giuli, S., \u0026amp; Scalici, F. (2016). Protecting cognition from aging and Alzheimer's disease: a computerized cognitive training combined with reminiscence therapy. \u003cem\u003eInternational Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e31\u003c/em\u003e(4), 340\u0026ndash;348. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/gps.4328\u003c/span\u003e\u003cspan address=\"10.1002/gps.4328\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBeck, C., Heacock, P., Mercer, S., Thatcher, R., \u0026amp; Sparkman, C. (1988). The impact of cognitive skills remediation training on persons with Alzheimer's disease or mixed dementia. \u003cem\u003eJournal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(1), 73\u0026ndash;88.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBergamaschi, S., Arcara, G., Calza, A., Villani, D., Orgeta, V., \u0026amp; Mondini, S. (2013). One-year repeated cycles of cognitive training (CT) for Alzheimer\u0026rsquo;s disease. \u003cem\u003eAging clinical and experimental research\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e, 421\u0026ndash;426. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s40520-013-0065-2\u003c/span\u003e\u003cspan address=\"10.1007/s40520-013-0065-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBoller, B., Jennings, J. M., Dieudonn\u0026eacute;, B., Verny, M., \u0026amp; Ergis, A. M. (2012). Recollection training and transfer effects in Alzheimer\u0026rsquo;s disease: effectiveness of the repetition-lag procedure. \u003cem\u003eBrain and Cognition\u003c/em\u003e, \u003cem\u003e78\u003c/em\u003e(2), 169\u0026ndash;177. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.bandc.2011.10.011\u003c/span\u003e\u003cspan address=\"10.1016/j.bandc.2011.10.011\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBreuil, V., De Rotrou, J., Forette, F., Tortrat, D., Ganansia-Ganem, A., Frambourt, A., Moulin, F., \u0026amp; Boller, F. (1994). Cognitive stimulation of patients with dementia: Preliminary results. \u003cem\u003eInternational Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e(3), 211\u0026ndash;217. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/gps.930090306\u003c/span\u003e\u003cspan address=\"10.1002/gps.930090306\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBrueggen, K., Kasper, E., Ochmann, S., Pfaff, H., Webel, S., Schneider, W., \u0026amp; Teipel, S. (2017). Cognitive Rehabilitation in Alzheimer\u0026rsquo;s Disease: A Controlled Intervention Trial. \u003cem\u003eJournal of Alzheimer\u0026rsquo;s Disease\u003c/em\u003e, \u003cem\u003e57\u003c/em\u003e(4), 1315\u0026ndash;1324. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3233/jad-160771\u003c/span\u003e\u003cspan address=\"10.3233/jad-160771\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBrydges, C. R. (2019). Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology. \u003cem\u003eInnovation in Aging\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(4). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/geroni/igz036\u003c/span\u003e\u003cspan address=\"10.1093/geroni/igz036\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCahn-Weiner, D. A., F., M. P., W., R. G.,and, \u0026amp; Ott, B. R. (2003). Results of a Randomized Placebo-Controlled Study of Memory Training for Mildly Impaired Alzheimer's Disease Patients. \u003cem\u003eApplied Neuropsychology\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(4), 215\u0026ndash;223. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1207/s15324826an1004_3\u003c/span\u003e\u003cspan address=\"10.1207/s15324826an1004_3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eChristopoulou, S. C. (2023). Towards Automated Meta-Analysis of Clinical Trials: An Overview. \u003cem\u003eBioMedInformatics\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(1), 115\u0026ndash;140. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://mdpi-res.com/d_attachment/biomedinformatics/biomedinformatics-03-00009/article_deploy/biomedinformatics-03-00009.pdf?version=1675239592\u003c/span\u003e\u003cspan address=\"https://mdpi-res.com/d_attachment/biomedinformatics/biomedinformatics-03-00009/article_deploy/biomedinformatics-03-00009.pdf?version=1675239592\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eClare, L. (2017). Rehabilitation for people living with dementia: A practical framework of positive support. \u003cem\u003ePLoS Medicine\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pmed.1002245\u003c/span\u003e\u003cspan address=\"10.1371/journal.pmed.1002245\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eClare, L., Kudlicka, A., Oyebode, J. R., Jones, R. W., Bayer, A., Leroi, I., Kopelman, M., James, I. A., Culverwell, A., Pool, J., Brand, A., Henderson, C., Hoare, Z., Knapp, M., \u0026amp; Woods, B. (2019). Individual goal-oriented cognitive rehabilitation to improve everyday functioning for people with early-stage dementia: A multicentre randomised controlled trial (the GREAT trial). \u003cem\u003eInternational Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e(5), 709\u0026ndash;721. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/gps.5076\u003c/span\u003e\u003cspan address=\"10.1002/gps.5076\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eClare, L., Linden, D. E. J., Woods, R. T., Whitaker, R., Evans, S. J., Parkinson, C. H., van Paasschen, J., Nelis, S. M., Hoare, Z., Yuen, K. S. L., \u0026amp; Rugg, M. D. (2010). Goal-Oriented Cognitive Rehabilitation for People With Early-Stage Alzheimer Disease: A Single-Blind Randomized Controlled Trial of Clinical Efficacy. \u003cem\u003eThe American Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e18\u003c/em\u003e(10), 928\u0026ndash;939. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1097/JGP.0b013e3181d5792a\u003c/span\u003e\u003cspan address=\"10.1097/JGP.0b013e3181d5792a\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eClare, L., \u0026amp; Woods, R. T. (2004). Cognitive training and cognitive rehabilitation for people with early-stage Alzheimer's disease: A review. \u003cem\u003eNeuropsychological Rehabilitation\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e, 385\u0026ndash;401. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/09602010443000074\u003c/span\u003e\u003cspan address=\"10.1080/09602010443000074\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences. \u003cem\u003eThe SAGE Encyclopedia of Research Design\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDavis, R. N., Massman, P. J., \u0026amp; Doody, R. S. (2001). Cognitive intervention in Alzheimer disease: a randomized placebo-controlled study. \u003cem\u003eAlzheimer Disease \u0026amp; Associated Disorders\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(1), 1\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDe Vreese, L., \u0026amp; Neri, M. (1999). Ecological impact of combined cognitive training programs and drug treatment in Alzheimer\u0026rsquo;s disease. \u003cem\u003eInternational Psychogeriatrics\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e, S187.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eEvans, D. (2003). Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. \u003cem\u003eJournal of Clinical Nursing\u003c/em\u003e, \u003cem\u003e12\u003c/em\u003e(1), 77\u0026ndash;84. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1046/j.1365-2702.2003.00662.x\u003c/span\u003e\u003cspan address=\"10.1046/j.1365-2702.2003.00662.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGalante, E., Venturini, G., \u0026amp; Fiaccadori, C. (2007). Computer-based cognitive intervention for dementia: preliminary results of a randomized clinical trial. \u003cem\u003eG Ital Med Lav Ergon\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(3 Suppl B), B26\u0026ndash;32.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGarc\u0026iacute;a-Perdomo, H. A. (2016). Evidence synthesis and meta-analysis: a practical approach. \u003cem\u003eInternational Journal of Urological Nursing\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(1), 30\u0026ndash;36. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/ijun.12087\u003c/span\u003e\u003cspan address=\"10.1111/ijun.12087\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGiuli, C., Papa, R., Lattanzio, F., \u0026amp; Postacchini, D. (2016). The Effects of Cognitive Training for Elderly: Results from My Mind Project. \u003cem\u003eRejuvenation Research\u003c/em\u003e, \u003cem\u003e19\u003c/em\u003e(6), 485\u0026ndash;494. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1089/rej.2015.1791\u003c/span\u003e\u003cspan address=\"10.1089/rej.2015.1791\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGoudour, A., Samson, S., Bakchine, S., \u0026amp; Ehrl\u0026eacute;, N. (2011). [Semantic memory training in Alzheimer's disease]. \u003cem\u003eGeriatrie et psychologie neuropsychiatrie du vieillissement\u003c/em\u003e, \u003cem\u003e9 2\u003c/em\u003e, 237\u0026ndash;247. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1684/pnv.2011.0272\u003c/span\u003e\u003cspan address=\"10.1684/pnv.2011.0272\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHartung, J., \u0026amp; Knapp, G. (2001). A refined method for the meta-analysis of controlled clinical trials with binary outcome. \u003cem\u003eStatistics in Medicine\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e(24), 3875\u0026ndash;3889. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/sim.1009\u003c/span\u003e\u003cspan address=\"10.1002/sim.1009\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHedges, L. V. (1981). Distribution Theory for Glass's Estimator of Effect Size and Related Estimators. \u003cem\u003eJournal of Educational Statistics\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(2), 107\u0026ndash;128. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/1164588\u003c/span\u003e\u003cspan address=\"10.2307/1164588\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHedges, L. V. (2019). The Statistics of Replication. \u003cem\u003eMethodology\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(Supplement 1), 3\u0026ndash;14. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1027/1614-2241/a000173\u003c/span\u003e\u003cspan address=\"10.1027/1614-2241/a000173\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHedges, L. V., \u0026amp; Schauer, J. M. (2019). Statistical analyses for studying replication: Meta-analytic perspectives. \u003cem\u003ePsychological Methods\u003c/em\u003e, \u003cem\u003e24\u003c/em\u003e(5), 557\u0026ndash;570. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/met0000189\u003c/span\u003e\u003cspan address=\"10.1037/met0000189\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHeiss, W. D., Kessler, J., Mielke, R., Szelies, B., \u0026amp; Herholz, K. (2008). Long-Term Effects of Phosphatidylserine, Pyritinol, and Cognitive Training in Alzheimer's Disease: A Neuropsychological, EEG, and PET Investigation. \u003cem\u003eDementia 5\u003c/em\u003e(2), 88\u0026ndash;98. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1159/000106702\u003c/span\u003e\u003cspan address=\"10.1159/000106702\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHiggins, J. P., Savović, J., Page, M. J., Elbers, R. G., \u0026amp; Sterne, J. A. (2019). Assessing risk of bias in a randomized trial. In \u003cem\u003eCochrane Handbook for Systematic Reviews of Interventions\u003c/em\u003e (pp. 205\u0026ndash;228). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/9781119536604.ch8\u003c/span\u003e\u003cspan address=\"10.1002/9781119536604.ch8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHiggins, J. P. T., Altman, D. G., G\u0026oslash;tzsche, P. C., J\u0026uuml;ni, P., Moher, D., Oxman, A. D., Savović, J., Schulz, K. F., Weeks, L., \u0026amp; Sterne, J. A. C. (2011). The Cochrane Collaboration\u0026rsquo;s tool for assessing risk of bias in randomised trials. \u003cem\u003eBmj\u003c/em\u003e, \u003cem\u003e343\u003c/em\u003e, d5928. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmj.d5928\u003c/span\u003e\u003cspan address=\"10.1136/bmj.d5928\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHindle, J. V., Watermeyer, T. J., Roberts, J., Brand, A., Hoare, Z., Martyr, A., \u0026amp; Clare, L. (2018). Goal-orientated cognitive rehabilitation for dementias associated with Parkinson's disease―A pilot randomised controlled trial. \u003cem\u003eInternational Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e33\u003c/em\u003e(5), 718\u0026ndash;728. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/gps.4845\u003c/span\u003e\u003cspan address=\"10.1002/gps.4845\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., \u0026amp; McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? \u003cem\u003eControlled Clinical Trials\u003c/em\u003e, \u003cem\u003e17\u003c/em\u003e(1), 1\u0026ndash;12. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/0197-2456(95)00134-4\u003c/span\u003e\u003cspan address=\"10.1016/0197-2456(95)00134-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJelcic, N., Agostini, M., Meneghello, F., Buss\u0026egrave;, C., Parise, S., Galano, A., Tonin, P., Dam, M., \u0026amp; Cagnin, A. (2014). Feasibility and efficacy of cognitive telerehabilitation in early Alzheimer\u0026rsquo;s disease: a pilot study. \u003cem\u003eClinical Interventions in Aging\u003c/em\u003e, 1605\u0026ndash;1611. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2147/CIA.S68145\u003c/span\u003e\u003cspan address=\"10.2147/CIA.S68145\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJelcic, N., Cagnin, A., Meneghello, F., Turolla, A., Ermani, M., \u0026amp; Dam, M. (2012). Effects of lexical\u0026ndash;semantic treatment on memory in early Alzheimer disease: An observer-blinded randomized controlled trial. \u003cem\u003eNeurorehabilitation and neural repair\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(8), 949\u0026ndash;956. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/1545968312440146\u003c/span\u003e\u003cspan address=\"10.1177/1545968312440146\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJohnson, E. E., O\u0026rsquo;Keefe, H., Sutton, A., \u0026amp; Marshall, C. (2022). The Systematic Review Toolbox: keeping up to date with tools to support evidence synthesis. \u003cem\u003eSystematic Reviews\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 258. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13643-022-02122-z\u003c/span\u003e\u003cspan address=\"10.1186/s13643-022-02122-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKallio, E. L., \u0026Ouml;hman, H., Hietanen, M., Soini, H., Strandberg, T. E., Kautiainen, H., \u0026amp; Pitk\u0026auml;l\u0026auml;, K. H. (2018). Effects of Cognitive Training on Cognition and Quality of Life of Older Persons with Dementia. \u003cem\u003eJournal of the American Geriatrics Society\u003c/em\u003e, \u003cem\u003e66\u003c/em\u003e(4), 664\u0026ndash;670. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jgs.15196\u003c/span\u003e\u003cspan address=\"10.1111/jgs.15196\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKudlicka, A., Martyr, A., Bahar-Fuchs, A., Sabat\u0026eacute;s, J., Woods, B., \u0026amp; Clare, L. (2023). Cognitive rehabilitation for people with mild to moderate dementia. \u003cem\u003eThe Cochrane database of systematic reviews\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e, CD013388. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.CD013388.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.CD013388.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLangan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., \u0026amp; Simmonds, M. (2019). A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. \u003cem\u003eResearch Synthesis Methods\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(1), 83\u0026ndash;98. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/jrsm.1316\u003c/span\u003e\u003cspan address=\"10.1002/jrsm.1316\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLee, G. Y., Yip, C. C., Yu, E. C., \u0026amp; Man, D. W. (2013). Evaluation of a computer-assisted errorless learning-based memory training program for patients with early Alzheimer\u0026rsquo;s disease in Hong Kong: a pilot study. \u003cem\u003eClinical Interventions in Aging\u003c/em\u003e, 623\u0026ndash;633. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2147/CIA.S45726\u003c/span\u003e\u003cspan address=\"10.2147/CIA.S45726\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMaher, C. G., Sherrington, C., Herbert, R. D., Moseley, A. M., \u0026amp; Elkins, M. (2003). Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials. \u003cem\u003ePhysical Therapy\u003c/em\u003e, \u003cem\u003e83\u003c/em\u003e(8), 713\u0026ndash;721. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/ptj/83.8.713\u003c/span\u003e\u003cspan address=\"10.1093/ptj/83.8.713\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMapelli, D., Di Rosa, E., Nocita, R., \u0026amp; Sava, D. (2013). Cognitive stimulation in patients with dementia: randomized controlled trial. \u003cem\u003eDementia and geriatric cognitive disorders extra\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(1), 263\u0026ndash;271. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1159/000353457\u003c/span\u003e\u003cspan address=\"10.1159/000353457\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMarinho, V., Bertrand, E., Naylor, R., Bomilcar, I., Laks, J., Spector, A., \u0026amp; Mograbi, D. C. (2021). Cognitive stimulation therapy for people with dementia in Brazil (CST-Brasil): Results from a single blind randomized controlled trial. \u003cem\u003eInternational Journal of Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e36\u003c/em\u003e(2), 286\u0026ndash;293. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/gps.5421\u003c/span\u003e\u003cspan address=\"10.1002/gps.5421\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMarshall, C., Kitchenham, B. A., \u0026amp; Brereton, P. (2018). Tool Features to Support Systematic Reviews in Software Engineering ? A Cross Domain Study. \u003cem\u003ee Informatica Softw Eng J\u003c/em\u003e, \u003cem\u003e12\u003c/em\u003e, 79\u0026ndash;115.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003ePaul, M., \u0026amp; Leibovici, L. (2014). Systematic review or meta-analysis? Their place in the evidence hierarchy. \u003cem\u003eClinical Microbiology and Infection\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e(2), 97\u0026ndash;100. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/1469-0691.12489\u003c/span\u003e\u003cspan address=\"10.1111/1469-0691.12489\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eQuayhagen, M. P., Quayhagen, M., Corbeil, R. R., Hendrix, R. C., Jackson, J. E., Snyder, L., \u0026amp; Bower, D. (2000). Coping with dementia: evaluation of four nonpharmacologic interventions. \u003cem\u003eInternational Psychogeriatrics\u003c/em\u003e, \u003cem\u003e12\u003c/em\u003e(2), 249\u0026ndash;265. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1017/S1041610200006360\u003c/span\u003e\u003cspan address=\"10.1017/S1041610200006360\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRai, H. K., Schneider, J., \u0026amp; Orrell, M. (2021). An Individual Cognitive Stimulation Therapy App for People with Dementia and Carers: Results from a Feasibility Randomized Controlled Trial (RCT). \u003cem\u003eClinical Interventions in Aging\u003c/em\u003e, \u003cem\u003e16\u003c/em\u003e, 2079\u0026ndash;2094. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2147/CIA.S323994\u003c/span\u003e\u003cspan address=\"10.2147/CIA.S323994\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., \u0026amp; Bahar-Fuchs, A. (2024). Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset. \u003cem\u003eNatural Language Processing Journal\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e, 100083. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.nlp.2024.100083\u003c/span\u003e\u003cspan address=\"10.1016/j.nlp.2024.100083\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSabates, J., Belleville, S., Castellani, M., Dwolatzky, T., Hampstead, B. M., Lampit, A., Simon, S., Anstey, K., Goodenough, B., Mancuso, S., Marques, D., Sinnott, R., \u0026amp; Bahar-Fuchs, A. (2021). CogTale: an online platform for the evaluation, synthesis, and dissemination of evidence from cognitive interventions studies. \u003cem\u003eSystematic Reviews\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(1), 236. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13643-021-01787-2\u003c/span\u003e\u003cspan address=\"10.1186/s13643-021-01787-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSackett, D. L. (1997). Evidence-based medicine. \u003cem\u003eSeminars in Perinatology\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(1), 3\u0026ndash;5. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S0146-0005(97)80013-4\u003c/span\u003e\u003cspan address=\"10.1016/S0146-0005(97)80013-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSch\u0026uuml;nemann, H. J., Higgins, J. P., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., \u0026amp; Guyatt, G. H. (2019). Group, o. b. o. t. C. G. M., \u0026amp; Group, t. C. S. M. Completing \u0026lsquo;Summary of findings\u0026rsquo; tables and grading the certainty of the evidence. In \u003cem\u003eCochrane Handbook for Systematic Reviews of Interventions\u003c/em\u003e (pp. 375\u0026ndash;402). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/9781119536604.ch14\u003c/span\u003e\u003cspan address=\"10.1002/9781119536604.ch14\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., \u0026amp; Moher, D. (2007). How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. \u003cem\u003eAnnals of Internal Medicine\u003c/em\u003e, \u003cem\u003e147\u003c/em\u003e(4), 224\u0026ndash;233. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.7326/0003-4819-147-4-200708210-00179\u003c/span\u003e\u003cspan address=\"10.7326/0003-4819-147-4-200708210-00179\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSidik, K., \u0026amp; Jonkman, J. N. (2007). A comparison of heterogeneity variance estimators in combining results of studies. \u003cem\u003eStatistics in Medicine\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(9), 1964\u0026ndash;1981. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/sim.2688\u003c/span\u003e\u003cspan address=\"10.1002/sim.2688\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTanjong-Ghogomu, E., Tugwell, P., \u0026amp; Welch, V. A. (2009). Evidence-based medicine and the Cochrane Collaboration. \u003cem\u003eBulletin of the NYU hospital for joint diseases\u003c/em\u003e, \u003cem\u003e67 2\u003c/em\u003e, 198\u0026ndash;205.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eThivierge, S., Jean, L., \u0026amp; Simard, M. (2014). A randomized cross-over controlled study on cognitive rehabilitation of instrumental activities of daily living in Alzheimer disease. \u003cem\u003eThe American journal of geriatric psychiatry: official journal of the American Association for Geriatric Psychiatry\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(11), 1188\u0026ndash;1199. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jagp.2013.03.008\u003c/span\u003e\u003cspan address=\"10.1016/j.jagp.2013.03.008\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTrebbastoni, A., Imbriano, L., Podda, L., Rendace, L., Sacchetti, M. L., Campanelli, A., D'Antonio, F., \u0026amp; De Lena, C. (2018). Cognitive training in patients with Alzheimer's disease: findings of a 12-month randomized controlled trial. \u003cem\u003eCurrent Alzheimer Research\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(5), 452\u0026ndash;461. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2174/1567205014666171113105044\u003c/span\u003e\u003cspan address=\"10.2174/1567205014666171113105044\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003evan Dinter, R., Tekinerdogan, B., \u0026amp; Catal, C. (2021). Automation of systematic literature reviews: A systematic literature review. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e136\u003c/em\u003e, 106589. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2021.106589\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2021.106589\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWoods, B., Rai, H. K., Elliott, E., Aguirre, E., Orrell, M., \u0026amp; Spector, A. (2023). Cognitive stimulation to improve cognitive functioning in people with dementia. \u003cem\u003eThe Cochrane database of systematic reviews\u003c/em\u003e, \u003cem\u003e1\u003c/em\u003e, CD005562. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/14651858.CD005562.pub3\u003c/span\u003e\u003cspan address=\"10.1002/14651858.CD005562.pub3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable 1 to 5 are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"cognition-oriented treatments, cognitive training, evidence synthesis, automation, dementia, meta-analysis","lastPublishedDoi":"10.21203/rs.3.rs-7078540/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7078540/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAdvancing evidence-based medicine for aging-related interventions requires efficient and reliable methods for synthesising research findings. This study evaluates CogTale, an online platform designed to streamline evidence synthesis for cognition-oriented treatments (COTs) in older adults, by automating key aspects of meta-analysis. The platform\u0026rsquo;s performance was validated by replicating findings from three Cochrane Reviews of Cognition-oriented treatments (COTs) for older adults, cognitive training (CT), cognitive stimulation therapy (CST), and cognitive rehabilitation (CR). Key outcomes from the primary \u0026ldquo;Summary of Findings\u0026rdquo; tables were compared based on effect direction, confidence intervals, effect size magnitudes, and evidence certainty. Of the 72 studies analysed in the Cochrane Reviews, 61 were included in CogTale\u0026rsquo;s database. Using a lenient replication threshold, CogTale approximately replicated 16 of 18 outcomes (88.9%), while 11 outcomes (61.1%) met a more stringent threshold. Replication was most consistent for CST outcomes, with greater variability observed in CT and CR results due to data availability and methodological differences. These findings suggest that CogTale can approximately replicate high-quality SRMA results, particularly in CST, and demonstrate its potential as a scalable tool for aging research. CogTale demonstrates the potential to enhance efficiency and accessibility in evidence synthesis for aging-related interventions, offering researchers, clinicians, and policymakers a powerful tool for supporting evidence-based decision-making in dementia care. Further refinements are needed to optimise accuracy, particularly in addressing methodological discrepancies and ensuring data completeness.\u003c/p\u003e","manuscriptTitle":"Automated Evidence Synthesis of Cognitive Interventions in Older People using the CogTale Platform: A Validation Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-23 15:24:47","doi":"10.21203/rs.3.rs-7078540/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b88725b2-bbe3-4d1a-92ed-6ce393c9b495","owner":[],"postedDate":"July 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-02T23:08:16+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-23 15:24:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7078540","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7078540","identity":"rs-7078540","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}