{"paper_id":"3eb4c615-4d12-4b48-8e33-c1a6763d5ccc","body_text":"Evaluating the use of Artificial Intelligence (AI) in Systematic Review Abstract Screening: A Comparative Study of AI-aided Tools | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluating the use of Artificial Intelligence (AI) in Systematic Review Abstract Screening: A Comparative Study of AI-aided Tools Selin Akaraci, Sophie Marie Jones, Christopher Tate, Niamh O'Kane, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7915337/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Background: Manual abstract screening in systematic reviews is a time-consuming and labour-intensive task. With the rise of artificial intelligence (AI), the number of published articles has grown substantially, adding to the workload of review studies that rely on robust and timely evidence synthesis. At the same time, AI-aided screening tools have been developed to accelerate this process. While previous studies have demonstrated the efficiency of such tools, ongoing technological advances necessitate updated evaluations, particularly for tools that are freely accessible. In review types such as umbrella reviews, where both the topic area and study design are central to eligibility decisions, the performance of AI-aided tools remains underexplored. Methods: We conducted a comparative evaluation of six freely available AI-aided abstract screening tools: Rayyan, RobotAnalyst, PICO Portal, Abstrackr, ASReview, and Colandr using a previously completed umbrella review of interdisciplinary urban planning and public health studies. We assessed (1) early recall performance (i.e., identification of included studies within the first 25% of screening), (2) feature availability and depth, and (3) user experience. This Study WIthin a Review (SWAR) was registered in the SWAR repository as SWAR 25. Results: All evaluated tools supported the review process by facilitating screening and offering features such as prioritization and keyword highlighting. However, none identified more than 50% of the previously included studies within the first 25% of screening. Feature analysis and user feedback suggested that Rayyan and PICO Portal provided the most useful functionality for our interdisciplinary umbrella review context, although limitations were noted in duplicate removal and in recognising the importance of study design in eligibility decisions. Conclusions: Although a growing number of AI-assisted abstract screening tools are publicly and freely available, their accuracy, usability, and adaptability to different review designs remain limited. Enhanced support for duplicate detection and integration of study design considerations could improve their utility in umbrella reviews and other complex evidence syntheses. Continued evaluation and user training may support broader adoption across diverse research contexts. Figures Figure 1 Figure 2 1. Introduction Systematic reviews (SRs) are pivotal for informing evidence-based practice, consensus building, and the identification of research gaps. Timely evidence synthesis is essential for supporting decision-making, yet the traditional review process can delay the delivery of actionable insights[ 1 – 3 ]. The rapid growth of scientific literature, partly driven by advances in artificial intelligence (AI) and digital technologies[ 4 , 5 ] has further increased the volume of published articles, intensifying the workload for systematic reviewers. Therefore, accelerating the screening process without compromising the accuracy of manual review could enhance the efficiency of evidence generation and deliver timely results. To address this, a growing number of artificial intelligence (AI)-aided software tools have been developed to improve the speed and consistency of screening [ 6 – 8 ]. These tools vary in accessibility and functionality, ranging from freely available applications that provide core screening features to subscription-based systems offering more advanced capabilities such as automated duplicate removal, predictive prioritization, and workflow management. Some tools incorporate machine learning algorithms that continuously learn from reviewer decisions to prioritize unscreened records according to their likelihood of inclusion, while others rely primarily on manual review with optional AI support. Some AI-aided tools offer support across multiple stages of the review process (Rayyan, PICO Portal), while others focus on specific tasks such as abstract screening (Abstrackr), which is among the most well-supported, widely implemented, and technically advanced features [ 9 , 10 ]. However, the initial adoption of AI-aided tools in SRs was relatively slow, as evidenced in a 2018 study that surveyed systematic reviewers from the field of biomedical sciences[ 11 ]. Several factors contributed to this, including licensing costs, steep learning curves, and limited support for some freely available tools. Additionally, a study published in 2018 reported that a lack of awareness contributed to low uptake of AI-aided screening tools [ 12 ]. However, a more recent survey (published in 2021) conducted among reviewers involved in evidence synthesis, health technology assessment, and guideline development, who all had conducted interventions, scoping, accuracy, and prognostic reviews, found that 79% reported using automation tools during the screening phase [ 13 ]. A growing body of literature, including several reviews, has explored the landscape of AI-aided screening tools, revealing key trends in focus, functionality, and limitations [ 9 , 10 , 14 – 18 ], with a common focus on feature identification and comparison. Some reviews [ 11 ] [ 9 , 15 ] have systematically catalogued the technical capabilities of existing screening platforms, particularly in relation to dual screening, reference management, and machine learning integration. These studies often relied on structured frameworks (e.g., DESMET) [ 19 ]or weighted scoring systems to evaluate tool functionality [ 14 , 18 ]. Other features of freely available AI-aided screening tools, such as usability and accessibility, have been evaluated, particularly for non-technical users[ 10 , 15 ]. Marshall et al. (2019)[ 10 ] conducted a descriptive study introducing several tools for screening tasks, including Abstrackr[ 8 ], EPPI-Reviewer [ 20 ], RobotAnalyst [ 21 ], SWIFT-Review [ 22 ], Colandr [ 23 ], and Rayyan [ 24 ], with a focus on their applicability for researchers without programming expertise. Harrison et al. (2020) [ 15 ] emphasized user friendliness and interface design, identifying Covidence and Rayyan as the most popular tools due to their ease of use. Similarly, two other studies [ 17 , 25 ] conducted scoping reviews to map the overall landscape of automation in screening. Blaziot’s study included 12 systematic reviews in the health sciences that employed some form of AI method. Among these, eight reviews used AI during the title and abstract screening stages and included evaluations conducted by the authors of each review. The findings revealed time savings and workload reductions associated with the use of AI. The 47 studies included in Khalil et al (2022) scoping review examined tools for the semi-automation of various stages of systematic reviews, encompassing both freely available and commercially licensed software (e.g., DistillerSR [ 26 ]); the authors noted that the screening stage has reached a relatively high level of maturity, although many tools remain insufficiently validated, are often domain-specific, and may present usability or accessibility challenges. Other studies have aimed to evaluate performance and real-world applications of AI-aided screening tools, though such work remains relatively limited. Reis et al. (2023), for example, assessed three screening tools – Rayyan, Abstrackr, and Colandr – based on speed, accuracy, and perceived usefulness, incorporating both objective outcomes and user perspectives. Their findings suggested that Rayyan software offered the best user experience and achieved the highest true-positive rate, meaning it correctly identified the largest number of relevant studies[ 18 ]. All three tools demonstrated comparable performance in identifying true negatives, as well as in measures of proportion missed, workload reduction, and time savings[ 18 ]. Similarly, Kohl et al. (2018)[ 27 ] compared several AI-aided tools used at different stages of the review process, including screening, highlighting their respective features and usability. Despite evaluating only a small number of tools, they identified CADIMA[ 27 ] as the most favourable tool.. Furthermore, some studies evaluated AI-aided tool performance only within a single domain such as health care [ 9 , 17 ]. Taken together, these studies demonstrate a growing understanding and application of AI-aided screening tools, particularly in terms of available features and general usability. However, several key gaps remain. First, there is a need for studies that systematically re-evaluate AI-aided screening tools with respect to their updated feature sets and performance improvements. Second, existing evaluations have not assessed these tools in the context of umbrella reviews-studies that synthesize evidence from multiple systematic reviews rather than primary studies-which are becoming increasingly common alongside AI development [ 28 ]. Umbrella reviews present additional challenges, as automation tools must not only assess topical relevance and adhere to pre-defined eligibility criteria, but also accurately identify study designs by distinguishing between systematic reviews and primary research. Third, there is a need to evaluate these tools in multidomain or interdisciplinary contexts, as performance may vary when applied to reviews spanning diverse fields. In addition, there is a broader methodological gap regarding the transparent and standardized comparison of available screening tools. This has been underscored in a recent study [ 16 ] that highlighted the need for larger evaluation datasets from completed reviews, to enable more robust comparisons of tool performance and generalizability across review types and disciplines. To address these gaps, we aimed to systematically catalogue existing tools that automate abstract screening and conduct a comparative study of web-based, freely available AI-aided abstract screening tools by evaluating their feature sets, screening performance, and user experience. We focused on tools that are freely accessible or offer at least a trial version to ensure that all evaluated methods could be readily accessed, tested, and reproduced by other researchers without subscription barriers. Using a previously completed umbrella review focusing on the impacts of urban environments on cognitive health as a case study, we performed three complementary analyses in this Study Within a Review (SWAR)[ 29 ]. These were (1) a feature analysis based on established frameworks; (2) a performance assessment to assess the ability of tools to retrieve relevant studies within an initial screening subset; and (3) a user survey capturing subjective perceptions of learnability, interface intuitiveness, and proficiency time. By situating this evaluation within an umbrella review context, our study provides novel insights into how current freely available AI-aided tools perform under more demanding methodological requirements, and highlights avenues for future development in AI-aided abstract screening. 2. Methods This SWAR was registered in the SWAR repository as SWAR 25[ 30 ]. 2.1 Identification of AI-Aided Abstract Screening Tools We identified freely available AI-aided abstract screening tools through a comprehensive search strategy utilising multiple sources, including systematic electronic database searches to capture relevant published studies. University library webpages specifically detailing systematic review methodologies were also consulted, as these resources often curate and recommend evidence synthesis tools. The Systematic Review Toolbox [ 31 ], an online platform providing a comprehensive repository of software tools designed to support various stages of systematic reviews and evidence synthesis, was temporarily unavailable during our search period due to ongoing maintenance. However, we contacted the Systematic Review Toolbox team directly and received written guidance regarding currently available and emerging tools suitable for abstract screening. Additionally, we conducted extensive web searches and participated in webinars, including those hosted by academic institutions such as King’s College London (e.g., https://libguides.kcl.ac.uk/systematicreview/ai ), to understand the latest developments in AI-aided screening tools. Finally, we employed Research Rabbit ( https://www.researchrabbit.ai/ ), an AI-powered research discovery platform, to identify additional relevant literature and tools, and to sense check if any pertinent studies were overlooked in the above process. To map the evidence on the evaluation of AI-aided abstract screening tools, we constructed a matrix with identified tools listed in the rows and individual studies in the columns (see Table S1). This approach allowed us to visualize the extent and distribution of tool evaluations, identify which tools had been widely studied versus those with limited or no evaluation, and to detect potential redundancies or gaps in the literature. An evaluation framework was developed to assess the identified AI-aided abstract screening tools (see Table S2) and was composed of three sections: (1) performance metrics, (2) feature analysis (see Table S3), and (3) user survey (see file S1). This framework was based on previous studies that evaluated AI-aided screening tools at various stages of systematic reviews [ 14 , 18 ]. Four team members (SA, SMJ, CT, NOK) were assigned to test the tools, with each tool evaluated by two randomly selected raters. All raters received training based on our recently completed umbrella review synthesising evidence on the association between urban environment features and cognitive health. Subsequently, each rater independently evaluated the tools. 2.1.1 Database Search Searches were conducted in Embase, MEDLINE, and the Cochrane Library to identify studies using AI-aided abstract screening tools across various evidence synthesis methodologies. The search covered publications from January 2021 to April 2024. This date range was chosen to build upon the comprehensive systematic review previously conducted by Blaziot et al. [ 17 ], which covered AI-aided abstract screening tools up to 2021, allowing our search to focus on capturing recent developments and publications from 2021 onward. The initial search was conducted in November 2024 to ensure inclusion of the most recent studies. The search strategy was developed collaboratively by two members of the research team (SJ and SA), adapting search terms from Blaziot et al.[ 17 ] and Khalil et al.[ 25 ]. The strategy was piloted in MEDLINE to ensure optimal balance between comprehensiveness and precision. The finalised search strings were subsequently adapted for the Cochrane Library. The full search strategy is provided in Table S4. 2.1.2 AI-aided Screening Tool Identification Screening was conducted in two phases using Covidence. Titles and abstracts were independently screened by two team members (SA and SMJ), followed by full-text screening. Any disagreements were resolved through discussion. We included systematic reviews, rapid reviews, umbrella reviews, evidence gap maps, evidence mapping, scoping reviews, and similar review types that employed AI-aided abstract screening tools with clearly reported AI methods. Methodology papers introducing novel AI models or screening software were also included. Only articles written in English were included. Priority was given to tools designed for interdisciplinary searches, although healthcare-specific tools were included. Review protocols, conference proceedings, and studies lacking full-text availability were excluded to ensure the inclusion of complete and accessible evidence. Following the identification of eligible studies, additional screening criteria were applied to determine which tools to evaluate in detail. Specifically, the additional criteria included tools that were currently functioning, did not require coding, free to use, and not limited to a specific subject domain. 2.2 Assessment of AI-aided Abstract Screening Tools All activities related to testing the identified AI-aided tools in this study were guided by a previously conducted umbrella review, with pilot testing of the inclusion/exclusion criteria among the research team included as one of these preparatory tasks. Following the selection of AI-aided tools, we tested their usability and performance using a set of 2,212 abstracts identified in the umbrella review. The review applied the search strategy provided in Table S5 to PubMed, Embase, and PsycINFO, without restrictions on publication date, and examined the effects of urban environments on cognitive health. Before assigning AI tools for evaluation, we assessed the consistency of each reviewer’s screening judgments to confirm that all reviewers could reliably apply the same eligibility criteria. Establishing this baseline consistency ensured that any observed differences in screening outcomes during AI-assisted evaluations could be attributed to tool performance rather than variability in human judgment. To achieve this, we conducted a calibration exercise using a 10% random sample of abstracts from the umbrella review dataset (n = 250), stratified by researchers not involved in the screening to include a representative distribution of previously included and excluded studies. The original screening decisions, used as the reference standard, were made by a different team of experienced systematic reviewers following dual independent screening and resolution of conflicts through consensus. Each of the three current reviewers (SJ, CT, and NOK) independently screened the calibration sample while blinded to the original decisions. Their inclusion and exclusion judgments were compared to the reference standard by one of the authors of the present study (SA). Conflicts among screening decisions were discussed and resolved through consensus. Raw rates of agreement exceeded 95% across all reviewers, and Cohen’s kappa values ranged from 0.900 to 0.986, indicating a high degree of agreement according to the Landis and Koch interpretation scale [REF]. These results provided strong evidence that current reviewers could consistently and accurately apply the previously defined eligibility criteria. Subsequently, AI-aided tools were allocated to reviewers, with each tool being tested by two reviewers. To mitigate against any potential bias, the distribution of tools was designed to ensure that reviewers had either no prior experience or only minimal experience with the tools. 2.2.1 Feature Analysis The feature analysis aimed to systematically identify and compare the functionalities embedded within each tested tool, providing potential users with insights to select tools that best align with their specific needs and preferences. The feature set used in this study was adapted from previously published frameworks[ 14 , 18 ] and was refined through internal discussion among the research team. These included, for example, the ability to import and allocate references, remove duplicates, export screening results, and support distinct title/abstract and full-text screening phases. Additional features such as conflict resolution tools, keyword highlighting, project progress tracking, and the ability to attach comments or PDFs to references were also evaluated. Active user support availability was also considered as a potential facilitator of tool adoption. The final set of features assessed is provided in Table S3. Two types of features were included in the analysis. Binary features captured whether a given functionality was present or absent, while compound features assessed the degree or quality of support, typically structured around multiple pre-defined levels of implementation. For each tool, it was first determined whether a particular feature was available. If so, the feature's implementation was then compared to conformance criteria as defined in Table S2. Feature analysis was carried out by three researchers (CT, SJ, and NOK), with each tool independently rated by two of the three researchers as described in Section 2.2 . All assessments were subsequently reviewed and verified by a single team member (SA) to ensure consistency. The user survey and performance evaluation were conducted by four researchers (CT, NOK, SA, and SJ), with each tool evaluated by two raters. Where applicable, intra-observer and inter-rater agreement were considered during the evaluation process, following approaches used in previous studies as done in a previously published study [ 9 ]. 2.2.2 Performance Metric Our primary performance metric was recall, defined as the proportion of relevant studies identified within a given portion of the screening queue. We selected recall as the key outcome because, in evidence synthesis, the ability to adequately identify relevant studies early in the screening process is essential to reducing workload while maintaining review quality. This was particularly critical in the context of our umbrella review, where missing relevant systematic reviews could compromise the comprehensiveness and validity of the synthesis. Initially, we adopted an exploratory approach in which screening continued until reviewers judged that the remaining unscreened abstracts were likely to be irrelevant. However, this strategy proved inconsistent across tools. In some cases, particularly with Abstrackr, no clear point of convergence was reached, and screening needed to continue through nearly the full dataset to ensure all relevant studies were captured. To overcome this variability and allow for standardised comparison, we implemented a fixed evaluation threshold: we screened the first 25% of records prioritised by each tool and calculated the recall achieved within that subset. This approach reflects a commonly used heuristic in the AI-aided screening literature, where early recall performance is used as a proxy for workload reduction potential[ 32 ]. While this threshold is not intended as a stopping rule, it provides a pragmatic benchmark to assess whether a tool surfaces a substantial proportion of relevant studies early in the screening process. 2.2.3 User Survey To evaluate the perceived usability and functionality of the AI-aided abstract screening tools, we conducted a post-use user experience survey. The objective of the survey was to capture subjective impressions of tool usability, ease of learning, and feature adequacy as perceived by the reviewers involved in tool testing. The survey explored several key dimensions. These included the ease with which reviewers were able to learn how to use each tool, their perceptions of user-friendliness and interface intuitiveness, and the amount of time required to gain proficiency in the tool. In addition, reviewers were asked whether they identified any missing or insufficient features and were invited to suggest potential improvements. The full list of survey questions is provided in File S1. The instrument was adapted from the previously published framework [ 18 ], ensuring methodological rigor and comparability with prior studies of AI-aided evidence synthesis tools. The questionnaire included both closed- and open-ended questions to allow for structured feedback as well as exploratory input. All reviewers who participated in the tool performance evaluation completed the survey immediately following their use of each assigned tool. Survey responses were analysed descriptively to identify common usability patterns and inform future tool development recommendations. 3. Results 3.1 Identification of AI-aided Abstract Screening Tools In total, we identified 43 tools (see File S3), eight of which were deemed suitable for analysis: AbstrackR, CADIMA, Colandr, Rayyan, PICO Portal [ 29 ], RobotAnalyst, and ASReview. An overview of all identified tools is provided in Supplementary Table S3. Of these eight tools, six-AbstrackR, ASReview, Colandr, PICO Portal, Rayyan, and RobotAnalyst-were included in the evaluation, as they offered AI-based prioritization and/or active learning functionalities at the time of assessment. CADIMA and LitReview were excluded because they did not support AI-assisted title and abstract screening during the study period. 3.2 Feature Analysis The feature analysis revealed that all seven AI-aided abstract screening tools included in this study offered core functionalities necessary for basic screening workflows. Results of the feature analysis are shown in Table 1 . Common features across tools included reference importing, inclusion and exclusion marking, and the ability to export results - functions that are foundational to any systematic review process. All tools supported formal export in standard formats and allowed some form of structured screening. Some tools also offered visual progress tracking (n = 4) and keyword highlighting (n = 4), aiding usability and navigation during screening. Despite this shared baseline, tools varied in the level of support for collaboration, reviewer management, and workflow customisation. However, as our evaluation focused on independent, single-user screening, these features were assessed based on their documentation and availability in the interface, rather than hands-on multi-user testing. In tools where such features were present, they typically appeared in pre-defined forms (e.g., fixed roles or basic multi-user settings), but advanced customisation options were not commonly observed. This interpretation should be treated with caution, as we did not assess collaborative functionality through live, team-based screening scenarios. One notable distinction across tools was in the handling of duplicate references. PICO Portal [ 33 ] was the only tool in our evaluation that fully supported automated duplicate removal, offering a streamlined process for detecting and removing duplicates without manual intervention. In contrast, while other tools provided partial support, such as identifying or flagging potential duplicates, they typically did not offer complete or automatic removal and required users to manage duplicates manually. Similarly, project auditing (i.e., tracking reviewer actions and decisions over time) and PRISMA diagram generation were unsupported in all the tools except for PICO Portal, despite their importance for transparency and reporting in systematic reviews. Support for multilingual interfaces and non-Latin characters appeared limited across most tools based on available documentation and interface settings. While some platforms indicated partial support for other languages or character sets, our evaluation was restricted to English-language articles and interfaces, and we did not directly test multilingual functionality. As such, findings in this area should be interpreted with caution. Some tools demonstrated unique strengths. For instance, PICO Portal, Colandr and Abstrackr offered highly intuitive, visually user-friendly interfaces that flattened the learning curve for new users. These tools also integrated direct customer support and detailed help documentation, which may facilitate adoption among less experienced reviewers. In contrast, others prioritised streamlined workflows for single reviewers or incorporated customisable tagging systems, which can support internal classification schemes during screening. The total scores of support levels across features for each tool are presented in Fig. 1 . According to these results, PICO Portal achieved the highest average score, reflecting consistent support across a wide range of features, including collaboration, usability, and data handling. Colandr, Rayyan and AbstrackR also performed well, particularly in user-focused aspects such as tagging, keyword highlighting, and progress tracking. In contrast, tools like RobotAnalyst and ASReview demonstrated more limited functionality overall, with lower average scores, especially in collaboration and discrepancy resolution. These findings suggest that while most tools support essential screening activities, only a few provide the comprehensive feature sets required for complex, team-based systematic review workflows. 3.3 Performance Our findings indicated that none of the tools retrieved more than 50% of relevant studies within the first 25% of screened records. 3.4 User Survey Table 2 Reviewer scores for each tool on easiness to learn, user-friendliness, and time to learn the tool. Survey Questions Reviewer no. Easiness to learn User-friendliness Time to learn AbstracKR R2 5 2.5 5 R3 7.5 2.5 5 AsReview R1 7.5 5 7.5 R2 7.5 7.5 7.5 Colandr R1 5 5 5 R3 7.5 5 5 PICO Portal R1 7.5 5 7.5 R4 7.5 7.5 10 Rayyan R1 7.5 7.5 10 R4 7.5 7.5 7.5 RobotAnalyst R3 7.5 7.5 7.5 R4 7.5 7.5 10 Table 2 presents the individual reviewer ratings for easiness to learn, user-friendliness, and time to learn the tool across the evaluated tools, while Fig. 2 illustrates the average scores for each question per tool in a stacked bar graph format. Across all tools, higher scores reflected a more favourable user experience. Several tools stood out positively in these domains: Rayyan, RobotAnalyst, and PICO Portal received the highest average scores for user-friendliness and time efficiency, indicating that users found them intuitive and quick to learn. PICO Portal was also noted for its visually accessible interface, which contributed to an efficient onboarding process. ASReview was also rated positively for ease of learning and required minimal time to reach proficiency, suggesting a generally favourable learning experience. While tools varied in design and user interaction, those with clear visual layouts, guided workflows, and accessible support documentation tended to receive higher usability ratings. These findings underscore the value of intuitive interface design and streamlined access to enhance the user experience during AI-aided abstract screening. In addition to survey feedback, practical observations during the evaluation process provided further context regarding tool access and setup. RobotAnalyst required a trial account to access the platform. A three-month license was granted to the authors for this evaluation, with no restriction on the number of abstracts uploaded. PICO Portal similarly offered a free trial with full feature access, allowing for uninterrupted use during the review period. CADIMA was accessed through a time-limited training account, which expired after three months. Unlike other tools (i.e., PICO Portal, Rayyan, AsReview, Colandr and Abstrackr) CADIMA did not support RIS file uploads. Instead, it offered an internal search function, which during testing did not yield usable references. This limited the team’s ability to proceed with screening through the platform. Colandr presented some access challenges for team members, specifically related to password resets and account recovery. These issues delayed initial use for some reviewers. However, the tool did provide user guides on its website, which helped address questions related to tool functions once access was obtained. These observations reflect tool configurations and access experiences at the time of evaluation. While functionality and support may change over time, such practical considerations remain important for teams planning to adopt AI-assisted tools in time-sensitive review contexts. Reviewers were asked to identify any features they found to be missing or insufficient in the tools, as well as to suggest improvements that could enhance the user experience. Several common themes emerged across responses, including requests for clearer feedback on AI behaviour, automated deduplication, improved filtering and tagging functions, and enhanced documentation or user support. A frequently mentioned concern was the lack of clarity around AI functionality and prioritisation mechanisms. Reviewers using tools with prioritisation features, such as PICO Portal and Rayyan, noted uncertainty regarding how the software reordered records based on relevance during screening. Reviewers suggested that visual indicators such as confidence scores or AI progress feedback could help clarify when it may be reasonable to stop screening. Similarly, in PICO Portal, reviewers expressed a desire to better understand how the \"priority screening algorithm\" was functioning in real-time. Another recurring theme from the user survey was the need for improved duplicate handling. Several tools either lacked automated deduplication entirely or offered only limited manual support. Reviewers highlighted that integrating automatic duplicate removal, particularly in large-scale reviews, would substantially reduce screening burden. This was noted specifically in relation to Abstrackr and RobotAnalyst. Documentation and onboarding support were also identified as areas for improvement. One reviewer mentioned difficulty locating official documentation for RobotAnalyst, which led to a trial-and-error learning approach despite the tool being otherwise well-received. Similarly, Colandr users experienced issues with logging in and noted challenges in using filtering features, including occasional script errors and difficulty locating tags for studies, even after setup. Reviewers also recommended enhancing filtering, tagging, and study categorisation options, particularly for Colandr and ASReview. While ASReview allowed note attachment, users suggested the addition of highlighting functionality, as well as clarity on whether multi-user collaboration was supported. In some cases, visual or interface limitations (e.g., image rendering errors in Abstrackr) were also noted as affecting usability. Finally, reviewers emphasised the value of progress tracking features. Tools that lacked a visible indication of screening completion (e.g., percentage completed) were perceived as less transparent. Reviewers recommended that tools include more robust project progress displays to support workflow management and planning. 4. Discussion 4.1 Summary of Findings Our study evaluated the performance, features, and user experience of six freely available AI-aided abstract screening tools using an umbrella review as a case study. While performance differences between tools were not substantial, none were able to retrieve more than 50% of the included studies after screening 25% of abstracts, indicating limitations in early recall. Nevertheless, most tools adequately supported core screening tasks, and several demonstrated additional strengths in usability, tagging, and exporting functions. A likely reason for this performance could be the added complexity of our case study, and that the tools had to do more than judge topical relevance because they also needed to distinguish systematic reviews from primary research. This two-tiered classification task is inherently more challenging than typical screening, which usually focuses solely on content relevance. Furthermore, except for PICO Portal, none of the freely available tools automatically removed duplicates. This limitation can substantially increase screening burden and reduce efficiency in large-scale reviews. Despite these challenges, several tools were still able to surface a subset of pertinent studies early in the process, indicating they could help lighten reviewer workload. Although complete manual screening remains necessary in most cases, these platforms can serve as valuable assistants by streamlining the initial screening stages. PICO Portal consistently emerged as the highest-performing tool across multiple dimensions, offering strong support for collaboration, usability, and progress tracking. Colandr and Rayyan also performed well, especially in user-facing features like article tagging, keyword highlighting, and ease of navigation, though they lacked advanced collaborative functions such as project auditing and user-role customisation. Our survey results showed generally positive user experiences but also highlighted gaps in functionality and accessibility. Importantly, many free tools restrict essential features such as duplicate removal or allocation to paid plans or require time intensive account setup, limiting their appeal for time-constrained or resource-limited researchers. 4.2 Comparison with Existing Evidence Previous evaluations of AI-aided screening tools have consistently reported substantial workload reductions, which aligns with our findings [ 18 ] [ 9 , 14 , 15 ]. However, to our knowledge, no prior study has evaluated these tools specifically in the context of umbrella reviews or systematically assessed recall rates at early screening stages, making direct comparisons with our results challenging. Similarly, while some studies analysed tools that require paid subscriptions [ 14 ], these are not directly comparable to our study as we focused only on tools that are freely available for at least one case study. Among the available tools, Rayyan has been highlighted in the literature as a favourable option. For example, a recent study [ 18 ] reported that Rayyan correctly identified nearly 80% of relevant records, recommending it as the most effective software in their evaluation. In our study, Rayyan demonstrated strengths in user experience and feature availability, but our performance analysis did not reproduce the high recall rates reported previously, underscoring the potential influence of review context and dataset characteristics. PICO Portal has received limited formal assessment, likely because it was only introduced in 2020. One study [ 9 ] included PICO Portal in a broad feature analysis that focused on functionalities across all stages of the systematic review process rather than on screening performance specifically. The study evaluated the approach feature of screening, which referred to the type of AI methodology used to prioritise records during title/abstract screening, but it did not assess other aspects such as usability, efficiency, or recall performance. An earlier evaluation [ 34 ] highlighted PICO Portal’s PICO-based keyword highlighting and AI-driven prioritisation as key strengths, while noting a steep learning curve and usability challenges for new users. In contrast, our team found PICO Portal intuitive and user-friendly, suggesting potential improvements in the interface since its initial evaluation. On the other hand, while Abstrackr has been positively reviewed in prior studies [ 18 , 35 ] for its screening efficiency, our team comprising researchers with over five years of experience and involvement in multiple reviews experienced difficulties using it effectively for title/abstract screening. Such discrepancies between previous evaluations and our experience are unsurprising given the rapidly evolving nature of AI-aided review tools and differences in study designs, datasets, and reviewer experience. 4.3 Strengths, Limitations and Implications A key strength of this study is its use of a real-world umbrella review on an interdisciplinary topic that comprised more than 2,000 abstracts. Using this previous study as a case study allowed us to evaluate tool performance across a broad and complex abstract set. Another strength is the multi-dimensional evaluation framework, which included not only performance and feature analysis but also direct user experience through structured survey data. However, several limitations must be noted. First, our findings are based on a single case study, which limits generalisability. Future research should examine these tools across different domains to determine whether results are consistent. Additionally, while we documented the presence of features such as multi-user support and non-Latin character handling, we did not test their real-world performance and thus cannot comment on their practical utility. 4.4 Implications and Future Directions Our findings suggest that AI-aided screening tools are promising but remain underused in evidence synthesis. A significant barrier is the lack of awareness, training, and user support, particularly for researchers without computational backgrounds. Institutions can help close this gap by offering regular training workshops and integrating AI tools into research methodology curricula to increase both exposure and user confidence. Developers also play a critical role in facilitating adoption. Usability and onboarding support should be prioritised to lower the entry barrier for new users and flatten the learning curve. Our experience with the PICO Portal team illustrates the positive impact of proactive engagement: the developers provided onboarding materials, live demonstrations, and responsive assistance, all of which helped address the steep learning curve and can encourage broader use. Such direct interaction between developers and end-users can significantly improve adoption and satisfaction, particularly for researchers unfamiliar with AI technologies. From a development standpoint, there is clear room for improvement. Key areas include enhancing duplicate detection, implementing reliable study design recognition (e.g., distinguishing reviews from primary research), and offering transparent performance tracking. Additionally, improving multi-language support and non-Latin character recognition would expand the utility of these tools in global and multilingual research contexts. Authors of research studies themselves can also facilitate automation by clearly reporting study design and methodology in abstracts. This practice would help both human reviewers and AI tools accurately classify studies during the screening phase, which is particularly important in umbrella reviews where distinguishing between primary studies and reviews is essential but often requires full-text access. Finally, while generative AI and no-code tools offer new opportunities for automation, free platforms still fall short of offering fully comprehensive solutions for complex, multidisciplinary reviews. Tools such as PICO Portal are moving in the right direction by incorporating progress tracking and detailed performance reports, but broader functionality without reliance on paywalls is essential to ensure equity, scalability, and wider adoption of AI-aided screening in systematic review practice. 5. Conclusions This study evaluated six freely available AI-aided abstract screening tools using an umbrella review as a test case. Our findings suggest that these tools are useful and increasingly accessible for supporting systematic reviews, even for researchers without programming experience. While most tools performed adequately in core screening tasks, limitations remain, particularly in early recall, duplicate removal, study design recognition, and multi-user functionality. Despite these challenges, AI-aided screening tools hold strong potential to enhance the efficiency and consistency of evidence synthesis. For broader adoption, however, improvements in usability, feature completeness, and institutional training support are needed. As the field evolves, developers, researchers, and institutions all have a role to play in advancing and adapting these tools to better meet the practical demands of diverse review contexts. Declarations 6.1 Ethics approval and consent to participate Not applicable 6.2 Competing interest The authors declare that they have no competing interests 6.3 Consent for Publication Not applicable 6.4 Funding This work was supported by the Evidence Synthesis Ireland SWAR Award. 6.5 Author Contributions SA, RFH, LG, and MC contributed to the conceptualization of the study. SA, SMJ, CT, and NOK contributed to the methodology and to data analysis and interpretation. SA drafted the original manuscript. SA, SMJ, CT, NOK, RFH, LG, MC and MD reviewed and edited the manuscript. All authors read and approved the final version of the manuscript. 6.6 Acknowledgements We thank Dr Morteza Namvar for his valuable contributions to the discussions that informed the development of this study. 6.7 Availability of data and material All data generated or analysed during this study are included in this published article [and its supplementary information files] References Beller EM, et al. Are systematic reviews up-to-date at the time of publication? Syst reviews. 2013;2(1):36. Sampson M, et al. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61(6):531–6. Allers K, et al. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol. 2018;95:102–10. Hao Q et al. AI Expands Scientists' Impact but Contracts Science's Focus. arXiv preprint arXiv:2412.07727, 2024. Porto JR, et al. Quantifying the Scope of Artificial Intelligence–Assisted Writing in Orthopaedic Medical Literature: An Analysis of Prevalence and Validation of AI-Detection Software. JAAOS-Journal of the American Academy of Orthopaedic Surgeons; 2022. p. 105435. van Dijk SH, et al. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ open. 2023;13(7):e072254. Fabiano N, et al. How to optimize the systematic review process using AI tools. JCPP Adv. 2024;4(2):e12234. Wallace BC et al. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr . in Proceedings of the 2nd ACM SIGHIT international health informatics symposium . 2012. Cowie K, et al. Web-based software tools for systematic literature review in medicine: systematic search and feature analysis. JMIR Med Inf. 2022;10(5):e33219. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst reviews. 2019;8(1):163. Van Altena A, Spijker R, Olabarriaga S. Usage of automation tools in systematic reviews. Res synthesis methods. 2019;10(1):72–82. O’Connor AM, et al. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst reviews. 2018;7(1):3. Scott AM, et al. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol. 2021;138:80–94. Van der Mierden S, et al. Software tools for literature screening in systematic reviews in biomedical research. Altex. 2019;36(3):508–17. Harrison H, et al. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med Res Methodol. 2020;20(1):7. Schmidt L et al. A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, 2023. 181: pp. 65–75. Blaizot A, et al. Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synthesis Methods. 2022;13(3):353–62. Dos Reis AHS, et al. Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study. Syst reviews. 2023;12(1):68. Kitchenham B, Linkman S, Law D. DESMET: a methodology for evaluating software engineering methods and tools. Comput Control Eng. 1997;8(3):120–6. Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4.0: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, Institute of Education; 2010. Przybyła P, et al. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res synthesis methods. 2018;9(3):470–88. Howard BE, et al. SWIFT-Review: a text-mining workbench for systematic review. Syst reviews. 2016;5(1):87. Cheng S, et al. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conserv Biol. 2018;32(4):762–4. Ouzzani M, et al. Rayyan—a web and mobile app for systematic reviews. Syst reviews. 2016;5(1):210. Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol. 2022;144:22–42. DistillerSR. DistillerSR: Literature Review Software [Internet]. 2025. Kohl C, et al. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environ Evid. 2018;7(1):8. Aromataris E, et al. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. JBI Evid Implement. 2015;13(3):132–40. Devane D, et al. Study within a review (SWAR). J Evidence-Based Med. 2022;15(4):328. Akaraci S. SWAR 25: A comparison of artificial intelligence (AI) aided and manual reviewing in abstract screening. . 2023; Available from: https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/SWARFileStore/SWAR25%20Selin%20Akaraci%20(2023%20FEB%2028%202026).pdf Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews . in Proceedings of the 19th international conference on evaluation and assessment in software engineering . 2015. Van De Schoot R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125–33. PICOPortal2025. PICO Portal . 10.10.2025]; Available from: https://picoportal.net/ Minion JT, et al. Pico portal. J Can Health Libr Association. 2021;42(3):181. Gates A et al. Decoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions . 2020. Table 1 Table 1 is available in the Supplementary Files section. Supplementary Files SupplementaryMaterial.docx Table1.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Major revision 21 Feb, 2026 Reviewers agreed at journal 01 Feb, 2026 Reviewers invited by journal 13 Dec, 2025 Editor assigned by journal 11 Nov, 2025 First submitted to journal 21 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-7915337\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":false,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":559722075,\"identity\":\"6f5642c0-32cf-484a-b808-a43a4c745223\",\"order_by\":0,\"name\":\"Selin Akaraci\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA4UlEQVRIiWNgGAWjYBAC+QbGZjCDjb2BgRkixoZfi8EB5sMQdTwHiNXCwJYMYUkkEK2Fx9jg5x47eT7Jx9ukCxjs5Bkk0hLwapFv4DFO7HmWbNgmnVYmPYMh2bBBIu0AfmsO8Bgf4DnAnMAmnWMmzcPAnMAgkd5AQAv/54N/DtQnsEmeAWmpJ0YLj3Eyz4HDCWwSPCAth4FaCDjM4DCPsbHMgeOGbTxpxdYzDECMZwl4tci39xhLvjlQLS/ffnjj7YKKanl+9jQD/A5jRrIRjAjECpojSVE8CkbBKBgFIwgAAND+OREl32pXAAAAAElFTkSuQmCC\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast FMHS: Queen's University Belfast Faculty of Medicine Health and Life Sciences\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Selin\",\"middleName\":\"\",\"lastName\":\"Akaraci\",\"suffix\":\"\"},{\"id\":559722076,\"identity\":\"dc903ccf-6de8-4a23-9428-93461de9de85\",\"order_by\":1,\"name\":\"Sophie Marie Jones\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Sophie\",\"middleName\":\"Marie\",\"lastName\":\"Jones\",\"suffix\":\"\"},{\"id\":559722077,\"identity\":\"b04f8e0e-fa34-42fc-b9e3-08960a8c6365\",\"order_by\":2,\"name\":\"Christopher Tate\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Christopher\",\"middleName\":\"\",\"lastName\":\"Tate\",\"suffix\":\"\"},{\"id\":559722078,\"identity\":\"2a85dff0-c681-457b-986f-76513f0c1c84\",\"order_by\":3,\"name\":\"Niamh O'Kane\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Niamh\",\"middleName\":\"\",\"lastName\":\"O'Kane\",\"suffix\":\"\"},{\"id\":559722079,\"identity\":\"09450f2b-04db-4273-a888-c89ba5c5d537\",\"order_by\":4,\"name\":\"Leandro Garcia\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Leandro\",\"middleName\":\"\",\"lastName\":\"Garcia\",\"suffix\":\"\"},{\"id\":559722080,\"identity\":\"aa58dccc-1ad4-474c-9777-49109124a628\",\"order_by\":5,\"name\":\"Mike Clarke\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Mike\",\"middleName\":\"\",\"lastName\":\"Clarke\",\"suffix\":\"\"},{\"id\":559722081,\"identity\":\"0bb32d0a-5388-49b3-afff-8014d985678a\",\"order_by\":6,\"name\":\"Maureen Dobbins\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"McMaster University\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Maureen\",\"middleName\":\"\",\"lastName\":\"Dobbins\",\"suffix\":\"\"},{\"id\":559722082,\"identity\":\"152b7e14-79ee-4425-9b02-e8ff41f8fb0a\",\"order_by\":7,\"name\":\"Ruth Hunter\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Queen's University Belfast\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Ruth\",\"middleName\":\"\",\"lastName\":\"Hunter\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-10-21 16:38:37\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-7915337/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-7915337/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":100746194,\"identity\":\"a1a5241e-ad4b-4176-811b-fd186e5ac223\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:34\",\"extension\":\"xml\",\"order_by\":2,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":11370,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"sysrSYSRD2501445.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/851b97ed0a923dde5b0c188c.xml\"},{\"id\":100746189,\"identity\":\"307b0693-1c9b-463d-bb22-878defb3606c\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:33\",\"extension\":\"xml\",\"order_by\":3,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":974,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"SYSRD250144523751.go.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/a384e709efa7e45686c7ae0e.xml\"},{\"id\":100746195,\"identity\":\"d4e4e9d2-d26c-4587-9a62-6ab7128f9029\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:34\",\"extension\":\"xml\",\"order_by\":4,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":841,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"SYSRD2501445Import.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/73ad09ad703ac6fed2420159.xml\"},{\"id\":100746188,\"identity\":\"b9619566-1214-4b06-9c66-cec61f2f6a04\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:32\",\"extension\":\"xml\",\"order_by\":5,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":182916,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"SYSRD25014450enriched.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/524c5a7a2c1aab9b1decc78a.xml\"},{\"id\":100796478,\"identity\":\"74cafa47-dbed-431d-b91e-054d82b2a036\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 13:43:28\",\"extension\":\"xml\",\"order_by\":8,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":181768,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"SYSRD25014450structuring.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/77bdeb3868a4f094fa6a1f9d.xml\"},{\"id\":100746182,\"identity\":\"78c443e3-c570-4247-a507-ade499c38c87\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:27\",\"extension\":\"html\",\"order_by\":9,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":194287,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"earlyproof.html\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/85ef8c8d4ca8b6ce7c55e36c.html\"},{\"id\":100746181,\"identity\":\"7567dec7-fdb3-4b9a-b67e-fdc710cdf9ca\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:26\",\"extension\":\"jpg\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":115334,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eFeature Matrix of AI-Assisted Screening Tools\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"1.jpg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/51cbfeb24556791dc4ffe58e.jpg\"},{\"id\":100746186,\"identity\":\"541b3715-16a6-41ba-a297-96b262f6c77f\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:31\",\"extension\":\"jpg\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":62193,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eAverage reviewer scores for survey questions on easiness to learn, user-friendliness, and time to learn the tool.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"2.jpg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/2d160abbc509fd51497a3ca0.jpg\"},{\"id\":100798037,\"identity\":\"857e588f-1610-43c6-bb9c-ea583f7eb278\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 13:52:21\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":898149,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/72ff6d95-9945-4f79-8802-21de680932b5.pdf\"},{\"id\":100746180,\"identity\":\"c151d132-382a-4d1e-b5bd-f59eec0121cc\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:26\",\"extension\":\"docx\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":51529,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"SupplementaryMaterial.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/3064133d277f86e7fd8673d4.docx\"},{\"id\":100746193,\"identity\":\"4a5e1ef4-7e6b-4973-be80-5413faedef31\",\"added_by\":\"auto\",\"created_at\":\"2026-01-21 03:31:33\",\"extension\":\"docx\",\"order_by\":2,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":17838,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Table1.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7915337/v1/1667935dca09901362bb094a.docx\"}],\"financialInterests\":\"\",\"formattedTitle\":\"Evaluating the use of Artificial Intelligence (AI) in Systematic Review Abstract Screening: A Comparative Study of AI-aided Tools\",\"fulltext\":[{\"header\":\"1. Introduction\",\"content\":\"\\u003cp\\u003eSystematic reviews (SRs) are pivotal for informing evidence-based practice, consensus building, and the identification of research gaps. Timely evidence synthesis is essential for supporting decision-making, yet the traditional review process can delay the delivery of actionable insights[\\u003cspan additionalcitationids=\\\"CR2\\\" citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e]. The rapid growth of scientific literature, partly driven by advances in artificial intelligence (AI) and digital technologies[\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e] has further increased the volume of published articles, intensifying the workload for systematic reviewers. Therefore, accelerating the screening process without compromising the accuracy of manual review could enhance the efficiency of evidence generation and deliver timely results. To address this, a growing number of artificial intelligence (AI)-aided software tools have been developed to improve the speed and consistency of screening [\\u003cspan additionalcitationids=\\\"CR7\\\" citationid=\\\"CR6\\\" class=\\\"CitationRef\\\"\\u003e6\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e]. These tools vary in accessibility and functionality, ranging from freely available applications that provide core screening features to subscription-based systems offering more advanced capabilities such as automated duplicate removal, predictive prioritization, and workflow management. Some tools incorporate machine learning algorithms that continuously learn from reviewer decisions to prioritize unscreened records according to their likelihood of inclusion, while others rely primarily on manual review with optional AI support.\\u003c/p\\u003e \\u003cp\\u003eSome AI-aided tools offer support across multiple stages of the review process (Rayyan, PICO Portal), while others focus on specific tasks such as abstract screening (Abstrackr), which is among the most well-supported, widely implemented, and technically advanced features [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e]. However, the initial adoption of AI-aided tools in SRs was relatively slow, as evidenced in a 2018 study that surveyed systematic reviewers from the field of biomedical sciences[\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e]. Several factors contributed to this, including licensing costs, steep learning curves, and limited support for some freely available tools. Additionally, a study published in 2018 reported that a lack of awareness contributed to low uptake of AI-aided screening tools [\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e]. However, a more recent survey (published in 2021) conducted among reviewers involved in evidence synthesis, health technology assessment, and guideline development, who all had conducted interventions, scoping, accuracy, and prognostic reviews, found that 79% reported using automation tools during the screening phase [\\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eA growing body of literature, including several reviews, has explored the landscape of AI-aided screening tools, revealing key trends in focus, functionality, and limitations [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e, \\u003cspan additionalcitationids=\\\"CR15 CR16 CR17\\\" citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e], with a common focus on feature identification and comparison. Some reviews [\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e] [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e] have systematically catalogued the technical capabilities of existing screening platforms, particularly in relation to dual screening, reference management, and machine learning integration. These studies often relied on structured frameworks (e.g., DESMET) [\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e]or weighted scoring systems to evaluate tool functionality [\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eOther features of freely available AI-aided screening tools, such as usability and accessibility, have been evaluated, particularly for non-technical users[\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e]. Marshall et al. (2019)[\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e] conducted a descriptive study introducing several tools for screening tasks, including Abstrackr[\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e], EPPI-Reviewer [\\u003cspan citationid=\\\"CR20\\\" class=\\\"CitationRef\\\"\\u003e20\\u003c/span\\u003e], RobotAnalyst [\\u003cspan citationid=\\\"CR21\\\" class=\\\"CitationRef\\\"\\u003e21\\u003c/span\\u003e], SWIFT-Review [\\u003cspan citationid=\\\"CR22\\\" class=\\\"CitationRef\\\"\\u003e22\\u003c/span\\u003e], Colandr [\\u003cspan citationid=\\\"CR23\\\" class=\\\"CitationRef\\\"\\u003e23\\u003c/span\\u003e], and Rayyan [\\u003cspan citationid=\\\"CR24\\\" class=\\\"CitationRef\\\"\\u003e24\\u003c/span\\u003e], with a focus on their applicability for researchers without programming expertise. Harrison et al. (2020) [\\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e] emphasized user friendliness and interface design, identifying Covidence and Rayyan as the most popular tools due to their ease of use. Similarly, two other studies [\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR25\\\" class=\\\"CitationRef\\\"\\u003e25\\u003c/span\\u003e] conducted scoping reviews to map the overall landscape of automation in screening. Blaziot\\u0026rsquo;s study included 12 systematic reviews in the health sciences that employed some form of AI method. Among these, eight reviews used AI during the title and abstract screening stages and included evaluations conducted by the authors of each review. The findings revealed time savings and workload reductions associated with the use of AI. The 47 studies included in Khalil et al (2022) scoping review examined tools for the semi-automation of various stages of systematic reviews, encompassing both freely available and commercially licensed software (e.g., DistillerSR [\\u003cspan citationid=\\\"CR26\\\" class=\\\"CitationRef\\\"\\u003e26\\u003c/span\\u003e]); the authors noted that the screening stage has reached a relatively high level of maturity, although many tools remain insufficiently validated, are often domain-specific, and may present usability or accessibility challenges. Other studies have aimed to evaluate performance and real-world applications of AI-aided screening tools, though such work remains relatively limited. Reis et al. (2023), for example, assessed three screening tools \\u0026ndash; Rayyan, Abstrackr, and Colandr \\u0026ndash; based on speed, accuracy, and perceived usefulness, incorporating both objective outcomes and user perspectives. Their findings suggested that Rayyan software offered the best user experience and achieved the highest true-positive rate, meaning it correctly identified the largest number of relevant studies[\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e]. All three tools demonstrated comparable performance in identifying true negatives, as well as in measures of proportion missed, workload reduction, and time savings[\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e]. Similarly, Kohl et al. (2018)[\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e] compared several AI-aided tools used at different stages of the review process, including screening, highlighting their respective features and usability. Despite evaluating only a small number of tools, they identified CADIMA[\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e] as the most favourable tool.. Furthermore, some studies evaluated AI-aided tool performance only within a single domain such as health care [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eTaken together, these studies demonstrate a growing understanding and application of AI-aided screening tools, particularly in terms of available features and general usability. However, several key gaps remain. First, there is a need for studies that systematically re-evaluate AI-aided screening tools with respect to their updated feature sets and performance improvements. Second, existing evaluations have not assessed these tools in the context of umbrella reviews-studies that synthesize evidence from multiple systematic reviews rather than primary studies-which are becoming increasingly common alongside AI development [\\u003cspan citationid=\\\"CR28\\\" class=\\\"CitationRef\\\"\\u003e28\\u003c/span\\u003e]. Umbrella reviews present additional challenges, as automation tools must not only assess topical relevance and adhere to pre-defined eligibility criteria, but also accurately identify study designs by distinguishing between systematic reviews and primary research. Third, there is a need to evaluate these tools in multidomain or interdisciplinary contexts, as performance may vary when applied to reviews spanning diverse fields. In addition, there is a broader methodological gap regarding the transparent and standardized comparison of available screening tools. This has been underscored in a recent study [\\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e] that highlighted the need for larger evaluation datasets from completed reviews, to enable more robust comparisons of tool performance and generalizability across review types and disciplines.\\u003c/p\\u003e \\u003cp\\u003eTo address these gaps, we aimed to systematically catalogue existing tools that automate abstract screening and conduct a comparative study of web-based, freely available AI-aided abstract screening tools by evaluating their feature sets, screening performance, and user experience. We focused on tools that are freely accessible or offer at least a trial version to ensure that all evaluated methods could be readily accessed, tested, and reproduced by other researchers without subscription barriers. Using a previously completed umbrella review focusing on the impacts of urban environments on cognitive health as a case study, we performed three complementary analyses in this Study Within a Review (SWAR)[\\u003cspan citationid=\\\"CR29\\\" class=\\\"CitationRef\\\"\\u003e29\\u003c/span\\u003e]. These were (1) a feature analysis based on established frameworks; (2) a performance assessment to assess the ability of tools to retrieve relevant studies within an initial screening subset; and (3) a user survey capturing subjective perceptions of learnability, interface intuitiveness, and proficiency time. By situating this evaluation within an umbrella review context, our study provides novel insights into how current freely available AI-aided tools perform under more demanding methodological requirements, and highlights avenues for future development in AI-aided abstract screening.\\u003c/p\\u003e\"},{\"header\":\"2. Methods\",\"content\":\"\\u003cp\\u003eThis SWAR was registered in the SWAR repository as SWAR 25[\\u003cspan citationid=\\\"CR30\\\" class=\\\"CitationRef\\\"\\u003e30\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cdiv id=\\\"Sec3\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.1 Identification of AI-Aided Abstract Screening Tools\\u003c/h2\\u003e \\u003cp\\u003eWe identified freely available AI-aided abstract screening tools through a comprehensive search strategy utilising multiple sources, including systematic electronic database searches to capture relevant published studies. University library webpages specifically detailing systematic review methodologies were also consulted, as these resources often curate and recommend evidence synthesis tools. The Systematic Review Toolbox [\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e], an online platform providing a comprehensive repository of software tools designed to support various stages of systematic reviews and evidence synthesis, was temporarily unavailable during our search period due to ongoing maintenance. However, we contacted the Systematic Review Toolbox team directly and received written guidance regarding currently available and emerging tools suitable for abstract screening. Additionally, we conducted extensive web searches and participated in webinars, including those hosted by academic institutions such as King\\u0026rsquo;s College London (e.g., \\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://libguides.kcl.ac.uk/systematicreview/ai\\u003c/span\\u003e\\u003cspan address=\\\"https://libguides.kcl.ac.uk/systematicreview/ai\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e), to understand the latest developments in AI-aided screening tools. Finally, we employed Research Rabbit (\\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://www.researchrabbit.ai/\\u003c/span\\u003e\\u003cspan address=\\\"https://www.researchrabbit.ai/\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e), an AI-powered research discovery platform, to identify additional relevant literature and tools, and to sense check if any pertinent studies were overlooked in the above process.\\u003c/p\\u003e \\u003cp\\u003eTo map the evidence on the evaluation of AI-aided abstract screening tools, we constructed a matrix with identified tools listed in the rows and individual studies in the columns (see Table S1). This approach allowed us to visualize the extent and distribution of tool evaluations, identify which tools had been widely studied versus those with limited or no evaluation, and to detect potential redundancies or gaps in the literature.\\u003c/p\\u003e \\u003cp\\u003eAn evaluation framework was developed to assess the identified AI-aided abstract screening tools (see Table S2) and was composed of three sections: (1) performance metrics, (2) feature analysis (see Table S3), and (3) user survey (see file S1). This framework was based on previous studies that evaluated AI-aided screening tools at various stages of systematic reviews [\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e]. Four team members (SA, SMJ, CT, NOK) were assigned to test the tools, with each tool evaluated by two randomly selected raters. All raters received training based on our recently completed umbrella review synthesising evidence on the association between urban environment features and cognitive health. Subsequently, each rater independently evaluated the tools.\\u003c/p\\u003e \\u003cdiv id=\\\"Sec4\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.1.1 Database Search\\u003c/h2\\u003e \\u003cp\\u003eSearches were conducted in Embase, MEDLINE, and the Cochrane Library to identify studies using AI-aided abstract screening tools across various evidence synthesis methodologies. The search covered publications from January 2021 to April 2024. This date range was chosen to build upon the comprehensive systematic review previously conducted by Blaziot et al. [\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e], which covered AI-aided abstract screening tools up to 2021, allowing our search to focus on capturing recent developments and publications from 2021 onward. The initial search was conducted in November 2024 to ensure inclusion of the most recent studies.\\u003c/p\\u003e \\u003cp\\u003eThe search strategy was developed collaboratively by two members of the research team (SJ and SA), adapting search terms from Blaziot et al.[\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e] and Khalil et al.[\\u003cspan citationid=\\\"CR25\\\" class=\\\"CitationRef\\\"\\u003e25\\u003c/span\\u003e]. The strategy was piloted in MEDLINE to ensure optimal balance between comprehensiveness and precision. The finalised search strings were subsequently adapted for the Cochrane Library. The full search strategy is provided in Table S4.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec5\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.1.2 AI-aided Screening Tool Identification\\u003c/h2\\u003e \\u003cp\\u003eScreening was conducted in two phases using Covidence. Titles and abstracts were independently screened by two team members (SA and SMJ), followed by full-text screening. Any disagreements were resolved through discussion.\\u003c/p\\u003e \\u003cp\\u003eWe included systematic reviews, rapid reviews, umbrella reviews, evidence gap maps, evidence mapping, scoping reviews, and similar review types that employed AI-aided abstract screening tools with clearly reported AI methods. Methodology papers introducing novel AI models or screening software were also included. Only articles written in English were included. Priority was given to tools designed for interdisciplinary searches, although healthcare-specific tools were included. Review protocols, conference proceedings, and studies lacking full-text availability were excluded to ensure the inclusion of complete and accessible evidence.\\u003c/p\\u003e \\u003cp\\u003eFollowing the identification of eligible studies, additional screening criteria were applied to determine which tools to evaluate in detail. Specifically, the additional criteria included tools that were currently functioning, did not require coding, free to use, and not limited to a specific subject domain.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec6\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.2 Assessment of AI-aided Abstract Screening Tools\\u003c/h2\\u003e \\u003cp\\u003eAll activities related to testing the identified AI-aided tools in this study were guided by a previously conducted umbrella review, with pilot testing of the inclusion/exclusion criteria among the research team included as one of these preparatory tasks. Following the selection of AI-aided tools, we tested their usability and performance using a set of 2,212 abstracts identified in the umbrella review. The review applied the search strategy provided in Table S5 to PubMed, Embase, and PsycINFO, without restrictions on publication date, and examined the effects of urban environments on cognitive health. Before assigning AI tools for evaluation, we assessed the consistency of each reviewer\\u0026rsquo;s screening judgments to confirm that all reviewers could reliably apply the same eligibility criteria. Establishing this baseline consistency ensured that any observed differences in screening outcomes during AI-assisted evaluations could be attributed to tool performance rather than variability in human judgment. To achieve this, we conducted a calibration exercise using a 10% random sample of abstracts from the umbrella review dataset (n\\u0026thinsp;=\\u0026thinsp;250), stratified by researchers not involved in the screening to include a representative distribution of previously included and excluded studies. The original screening decisions, used as the reference standard, were made by a different team of experienced systematic reviewers following dual independent screening and resolution of conflicts through consensus.\\u003c/p\\u003e \\u003cp\\u003eEach of the three current reviewers (SJ, CT, and NOK) independently screened the calibration sample while blinded to the original decisions. Their inclusion and exclusion judgments were compared to the reference standard by one of the authors of the present study (SA). Conflicts among screening decisions were discussed and resolved through consensus. Raw rates of agreement exceeded 95% across all reviewers, and Cohen\\u0026rsquo;s kappa values ranged from 0.900 to 0.986, indicating a high degree of agreement according to the Landis and Koch interpretation scale [REF]. These results provided strong evidence that current reviewers could consistently and accurately apply the previously defined eligibility criteria. Subsequently, AI-aided tools were allocated to reviewers, with each tool being tested by two reviewers. To mitigate against any potential bias, the distribution of tools was designed to ensure that reviewers had either no prior experience or only minimal experience with the tools.\\u003c/p\\u003e \\u003cdiv id=\\\"Sec7\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.2.1 Feature Analysis\\u003c/h2\\u003e \\u003cp\\u003eThe feature analysis aimed to systematically identify and compare the functionalities embedded within each tested tool, providing potential users with insights to select tools that best align with their specific needs and preferences. The feature set used in this study was adapted from previously published frameworks[\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e] and was refined through internal discussion among the research team. These included, for example, the ability to import and allocate references, remove duplicates, export screening results, and support distinct title/abstract and full-text screening phases. Additional features such as conflict resolution tools, keyword highlighting, project progress tracking, and the ability to attach comments or PDFs to references were also evaluated. Active user support availability was also considered as a potential facilitator of tool adoption. The final set of features assessed is provided in Table S3.\\u003c/p\\u003e \\u003cp\\u003eTwo types of features were included in the analysis. Binary features captured whether a given functionality was present or absent, while compound features assessed the degree or quality of support, typically structured around multiple pre-defined levels of implementation. For each tool, it was first determined whether a particular feature was available. If so, the feature's implementation was then compared to conformance criteria as defined in Table S2.\\u003c/p\\u003e \\u003cp\\u003eFeature analysis was carried out by three researchers (CT, SJ, and NOK), with each tool independently rated by two of the three researchers as described in Section \\u003cspan refid=\\\"Sec6\\\" class=\\\"InternalRef\\\"\\u003e2.2\\u003c/span\\u003e. All assessments were subsequently reviewed and verified by a single team member (SA) to ensure consistency. The user survey and performance evaluation were conducted by four researchers (CT, NOK, SA, and SJ), with each tool evaluated by two raters. Where applicable, intra-observer and inter-rater agreement were considered during the evaluation process, following approaches used in previous studies as done in a previously published study [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e].\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec8\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.2.2 Performance Metric\\u003c/h2\\u003e \\u003cp\\u003eOur primary performance metric was recall, defined as the proportion of relevant studies identified within a given portion of the screening queue. We selected recall as the key outcome because, in evidence synthesis, the ability to adequately identify relevant studies early in the screening process is essential to reducing workload while maintaining review quality. This was particularly critical in the context of our umbrella review, where missing relevant systematic reviews could compromise the comprehensiveness and validity of the synthesis.\\u003c/p\\u003e \\u003cp\\u003eInitially, we adopted an exploratory approach in which screening continued until reviewers judged that the remaining unscreened abstracts were likely to be irrelevant. However, this strategy proved inconsistent across tools. In some cases, particularly with Abstrackr, no clear point of convergence was reached, and screening needed to continue through nearly the full dataset to ensure all relevant studies were captured.\\u003c/p\\u003e \\u003cp\\u003eTo overcome this variability and allow for standardised comparison, we implemented a fixed evaluation threshold: we screened the first 25% of records prioritised by each tool and calculated the recall achieved within that subset. This approach reflects a commonly used heuristic in the AI-aided screening literature, where early recall performance is used as a proxy for workload reduction potential[\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e]. While this threshold is not intended as a stopping rule, it provides a pragmatic benchmark to assess whether a tool surfaces a substantial proportion of relevant studies early in the screening process.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec9\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.2.3 User Survey\\u003c/h2\\u003e \\u003cp\\u003eTo evaluate the perceived usability and functionality of the AI-aided abstract screening tools, we conducted a post-use user experience survey. The objective of the survey was to capture subjective impressions of tool usability, ease of learning, and feature adequacy as perceived by the reviewers involved in tool testing.\\u003c/p\\u003e \\u003cp\\u003eThe survey explored several key dimensions. These included the ease with which reviewers were able to learn how to use each tool, their perceptions of user-friendliness and interface intuitiveness, and the amount of time required to gain proficiency in the tool. In addition, reviewers were asked whether they identified any missing or insufficient features and were invited to suggest potential improvements.\\u003c/p\\u003e \\u003cp\\u003eThe full list of survey questions is provided in File S1. The instrument was adapted from the previously published framework [\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e], ensuring methodological rigor and comparability with prior studies of AI-aided evidence synthesis tools. The questionnaire included both closed- and open-ended questions to allow for structured feedback as well as exploratory input.\\u003c/p\\u003e \\u003cp\\u003eAll reviewers who participated in the tool performance evaluation completed the survey immediately following their use of each assigned tool. Survey responses were analysed descriptively to identify common usability patterns and inform future tool development recommendations.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/div\\u003e\"},{\"header\":\"3. Results\",\"content\":\"\\u003cdiv id=\\\"Sec11\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.1 Identification of AI-aided Abstract Screening Tools\\u003c/h2\\u003e \\u003cp\\u003eIn total, we identified 43 tools (see File S3), eight of which were deemed suitable for analysis: AbstrackR, CADIMA, Colandr, Rayyan, PICO Portal [\\u003cspan citationid=\\\"CR29\\\" class=\\\"CitationRef\\\"\\u003e29\\u003c/span\\u003e], RobotAnalyst, and ASReview. An overview of all identified tools is provided in Supplementary Table S3. Of these eight tools, six-AbstrackR, ASReview, Colandr, PICO Portal, Rayyan, and RobotAnalyst-were included in the evaluation, as they offered AI-based prioritization and/or active learning functionalities at the time of assessment. CADIMA and LitReview were excluded because they did not support AI-assisted title and abstract screening during the study period.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec12\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.2 Feature Analysis\\u003c/h2\\u003e \\u003cp\\u003eThe feature analysis revealed that all seven AI-aided abstract screening tools included in this study offered core functionalities necessary for basic screening workflows. Results of the feature analysis are shown in Table\\u0026nbsp;\\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e. Common features across tools included reference importing, inclusion and exclusion marking, and the ability to export results - functions that are foundational to any systematic review process. All tools supported formal export in standard formats and allowed some form of structured screening. Some tools also offered visual progress tracking (n\\u0026thinsp;=\\u0026thinsp;4) and keyword highlighting (n\\u0026thinsp;=\\u0026thinsp;4), aiding usability and navigation during screening.\\u003c/p\\u003e \\u003cp\\u003eDespite this shared baseline, tools varied in the level of support for collaboration, reviewer management, and workflow customisation. However, as our evaluation focused on independent, single-user screening, these features were assessed based on their documentation and availability in the interface, rather than hands-on multi-user testing. In tools where such features were present, they typically appeared in pre-defined forms (e.g., fixed roles or basic multi-user settings), but advanced customisation options were not commonly observed. This interpretation should be treated with caution, as we did not assess collaborative functionality through live, team-based screening scenarios.\\u003c/p\\u003e \\u003cp\\u003eOne notable distinction across tools was in the handling of duplicate references. PICO Portal [\\u003cspan citationid=\\\"CR33\\\" class=\\\"CitationRef\\\"\\u003e33\\u003c/span\\u003e] was the only tool in our evaluation that fully supported automated duplicate removal, offering a streamlined process for detecting and removing duplicates without manual intervention. In contrast, while other tools provided partial support, such as identifying or flagging potential duplicates, they typically did not offer complete or automatic removal and required users to manage duplicates manually. Similarly, project auditing (i.e., tracking reviewer actions and decisions over time) and PRISMA diagram generation were unsupported in all the tools except for PICO Portal, despite their importance for transparency and reporting in systematic reviews.\\u003c/p\\u003e \\u003cp\\u003eSupport for multilingual interfaces and non-Latin characters appeared limited across most tools based on available documentation and interface settings. While some platforms indicated partial support for other languages or character sets, our evaluation was restricted to English-language articles and interfaces, and we did not directly test multilingual functionality. As such, findings in this area should be interpreted with caution.\\u003c/p\\u003e \\u003cp\\u003eSome tools demonstrated unique strengths. For instance, PICO Portal, Colandr and Abstrackr offered highly intuitive, visually user-friendly interfaces that flattened the learning curve for new users. These tools also integrated direct customer support and detailed help documentation, which may facilitate adoption among less experienced reviewers. In contrast, others prioritised streamlined workflows for single reviewers or incorporated customisable tagging systems, which can support internal classification schemes during screening.\\u003c/p\\u003e \\u003cp\\u003eThe total scores of support levels across features for each tool are presented in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e. According to these results, PICO Portal achieved the highest average score, reflecting consistent support across a wide range of features, including collaboration, usability, and data handling. Colandr, Rayyan and AbstrackR also performed well, particularly in user-focused aspects such as tagging, keyword highlighting, and progress tracking. In contrast, tools like RobotAnalyst and ASReview demonstrated more limited functionality overall, with lower average scores, especially in collaboration and discrepancy resolution. These findings suggest that while most tools support essential screening activities, only a few provide the comprehensive feature sets required for complex, team-based systematic review workflows.\\u003c/p\\u003e \\u003cp\\u003e \\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec13\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.3 Performance\\u003c/h2\\u003e \\u003cp\\u003eOur findings indicated that none of the tools retrieved more than 50% of relevant studies within the first 25% of screened records.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec14\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.4 User Survey\\u003c/h2\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab2\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 2\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eReviewer scores for each tool on easiness to learn, user-friendliness, and time to learn the tool.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"5\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eSurvey Questions\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eReviewer no.\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eEasiness to learn\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eUser-friendliness\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eTime to learn\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eAbstracKR\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e2.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e2.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eAsReview\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eColandr\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003ePICO Portal\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR4\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e10\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eRayyan\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e10\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR4\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eRobotAnalyst\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eR4\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e7.5\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e10\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e presents the individual reviewer ratings for easiness to learn, user-friendliness, and time to learn the tool across the evaluated tools, while Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e illustrates the average scores for each question per tool in a stacked bar graph format. Across all tools, higher scores reflected a more favourable user experience. Several tools stood out positively in these domains: Rayyan, RobotAnalyst, and PICO Portal received the highest average scores for user-friendliness and time efficiency, indicating that users found them intuitive and quick to learn. PICO Portal was also noted for its visually accessible interface, which contributed to an efficient onboarding process. ASReview was also rated positively for ease of learning and required minimal time to reach proficiency, suggesting a generally favourable learning experience.\\u003c/p\\u003e \\u003cp\\u003eWhile tools varied in design and user interaction, those with clear visual layouts, guided workflows, and accessible support documentation tended to receive higher usability ratings. These findings underscore the value of intuitive interface design and streamlined access to enhance the user experience during AI-aided abstract screening.\\u003c/p\\u003e \\u003cp\\u003eIn addition to survey feedback, practical observations during the evaluation process provided further context regarding tool access and setup. RobotAnalyst required a trial account to access the platform. A three-month license was granted to the authors for this evaluation, with no restriction on the number of abstracts uploaded. PICO Portal similarly offered a free trial with full feature access, allowing for uninterrupted use during the review period.\\u003c/p\\u003e \\u003cp\\u003eCADIMA was accessed through a time-limited training account, which expired after three months. Unlike other tools (i.e., PICO Portal, Rayyan, AsReview, Colandr and Abstrackr) CADIMA did not support RIS file uploads. Instead, it offered an internal search function, which during testing did not yield usable references. This limited the team\\u0026rsquo;s ability to proceed with screening through the platform.\\u003c/p\\u003e \\u003cp\\u003eColandr presented some access challenges for team members, specifically related to password resets and account recovery. These issues delayed initial use for some reviewers. However, the tool did provide user guides on its website, which helped address questions related to tool functions once access was obtained.\\u003c/p\\u003e \\u003cp\\u003eThese observations reflect tool configurations and access experiences at the time of evaluation. While functionality and support may change over time, such practical considerations remain important for teams planning to adopt AI-assisted tools in time-sensitive review contexts.\\u003c/p\\u003e \\u003cp\\u003eReviewers were asked to identify any features they found to be missing or insufficient in the tools, as well as to suggest improvements that could enhance the user experience. Several common themes emerged across responses, including requests for clearer feedback on AI behaviour, automated deduplication, improved filtering and tagging functions, and enhanced documentation or user support.\\u003c/p\\u003e \\u003cp\\u003eA frequently mentioned concern was the lack of clarity around AI functionality and prioritisation mechanisms. Reviewers using tools with prioritisation features, such as PICO Portal and Rayyan, noted uncertainty regarding how the software reordered records based on relevance during screening. Reviewers suggested that visual indicators such as confidence scores or AI progress feedback could help clarify when it may be reasonable to stop screening. Similarly, in PICO Portal, reviewers expressed a desire to better understand how the \\\"priority screening algorithm\\\" was functioning in real-time.\\u003c/p\\u003e \\u003cp\\u003eAnother recurring theme from the user survey was the need for improved duplicate handling. Several tools either lacked automated deduplication entirely or offered only limited manual support. Reviewers highlighted that integrating automatic duplicate removal, particularly in large-scale reviews, would substantially reduce screening burden. This was noted specifically in relation to Abstrackr and RobotAnalyst.\\u003c/p\\u003e \\u003cp\\u003eDocumentation and onboarding support were also identified as areas for improvement. One reviewer mentioned difficulty locating official documentation for RobotAnalyst, which led to a trial-and-error learning approach despite the tool being otherwise well-received. Similarly, Colandr users experienced issues with logging in and noted challenges in using filtering features, including occasional script errors and difficulty locating tags for studies, even after setup.\\u003c/p\\u003e \\u003cp\\u003eReviewers also recommended enhancing filtering, tagging, and study categorisation options, particularly for Colandr and ASReview. While ASReview allowed note attachment, users suggested the addition of highlighting functionality, as well as clarity on whether multi-user collaboration was supported. In some cases, visual or interface limitations (e.g., image rendering errors in Abstrackr) were also noted as affecting usability.\\u003c/p\\u003e \\u003cp\\u003eFinally, reviewers emphasised the value of progress tracking features. Tools that lacked a visible indication of screening completion (e.g., percentage completed) were perceived as less transparent. Reviewers recommended that tools include more robust project progress displays to support workflow management and planning.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"4. Discussion\",\"content\":\"\\u003cdiv id=\\\"Sec16\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.1 Summary of Findings\\u003c/h2\\u003e \\u003cp\\u003eOur study evaluated the performance, features, and user experience of six freely available AI-aided abstract screening tools using an umbrella review as a case study. While performance differences between tools were not substantial, none were able to retrieve more than 50% of the included studies after screening 25% of abstracts, indicating limitations in early recall. Nevertheless, most tools adequately supported core screening tasks, and several demonstrated additional strengths in usability, tagging, and exporting functions. A likely reason for this performance could be the added complexity of our case study, and that the tools had to do more than judge topical relevance because they also needed to distinguish systematic reviews from primary research. This two-tiered classification task is inherently more challenging than typical screening, which usually focuses solely on content relevance. Furthermore, except for PICO Portal, none of the freely available tools automatically removed duplicates. This limitation can substantially increase screening burden and reduce efficiency in large-scale reviews.\\u003c/p\\u003e \\u003cp\\u003eDespite these challenges, several tools were still able to surface a subset of pertinent studies early in the process, indicating they could help lighten reviewer workload. Although complete manual screening remains necessary in most cases, these platforms can serve as valuable assistants by streamlining the initial screening stages.\\u003c/p\\u003e \\u003cp\\u003ePICO Portal consistently emerged as the highest-performing tool across multiple dimensions, offering strong support for collaboration, usability, and progress tracking. Colandr and Rayyan also performed well, especially in user-facing features like article tagging, keyword highlighting, and ease of navigation, though they lacked advanced collaborative functions such as project auditing and user-role customisation.\\u003c/p\\u003e \\u003cp\\u003eOur survey results showed generally positive user experiences but also highlighted gaps in functionality and accessibility. Importantly, many free tools restrict essential features such as duplicate removal or allocation to paid plans or require time intensive account setup, limiting their appeal for time-constrained or resource-limited researchers.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec17\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.2 Comparison with Existing Evidence\\u003c/h2\\u003e \\u003cp\\u003ePrevious evaluations of AI-aided screening tools have consistently reported substantial workload reductions, which aligns with our findings [\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e] [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e]. However, to our knowledge, no prior study has evaluated these tools specifically in the context of umbrella reviews or systematically assessed recall rates at early screening stages, making direct comparisons with our results challenging. Similarly, while some studies analysed tools that require paid subscriptions [\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e], these are not directly comparable to our study as we focused only on tools that are freely available for at least one case study.\\u003c/p\\u003e \\u003cp\\u003eAmong the available tools, Rayyan has been highlighted in the literature as a favourable option. For example, a recent study [\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e] reported that Rayyan correctly identified nearly 80% of relevant records, recommending it as the most effective software in their evaluation. In our study, Rayyan demonstrated strengths in user experience and feature availability, but our performance analysis did not reproduce the high recall rates reported previously, underscoring the potential influence of review context and dataset characteristics.\\u003c/p\\u003e \\u003cp\\u003ePICO Portal has received limited formal assessment, likely because it was only introduced in 2020. One study [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e] included PICO Portal in a broad feature analysis that focused on functionalities across all stages of the systematic review process rather than on screening performance specifically. The study evaluated the approach feature of screening, which referred to the type of AI methodology used to prioritise records during title/abstract screening, but it did not assess other aspects such as usability, efficiency, or recall performance. An earlier evaluation [\\u003cspan citationid=\\\"CR34\\\" class=\\\"CitationRef\\\"\\u003e34\\u003c/span\\u003e] highlighted PICO Portal\\u0026rsquo;s PICO-based keyword highlighting and AI-driven prioritisation as key strengths, while noting a steep learning curve and usability challenges for new users. In contrast, our team found PICO Portal intuitive and user-friendly, suggesting potential improvements in the interface since its initial evaluation.\\u003c/p\\u003e \\u003cp\\u003eOn the other hand, while Abstrackr has been positively reviewed in prior studies [\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR35\\\" class=\\\"CitationRef\\\"\\u003e35\\u003c/span\\u003e] for its screening efficiency, our team comprising researchers with over five years of experience and involvement in multiple reviews experienced difficulties using it effectively for title/abstract screening. Such discrepancies between previous evaluations and our experience are unsurprising given the rapidly evolving nature of AI-aided review tools and differences in study designs, datasets, and reviewer experience.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec18\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.3 Strengths, Limitations and Implications\\u003c/h2\\u003e \\u003cp\\u003eA key strength of this study is its use of a real-world umbrella review on an interdisciplinary topic that comprised more than 2,000 abstracts. Using this previous study as a case study allowed us to evaluate tool performance across a broad and complex abstract set. Another strength is the multi-dimensional evaluation framework, which included not only performance and feature analysis but also direct user experience through structured survey data.\\u003c/p\\u003e \\u003cp\\u003eHowever, several limitations must be noted. First, our findings are based on a single case study, which limits generalisability. Future research should examine these tools across different domains to determine whether results are consistent. Additionally, while we documented the presence of features such as multi-user support and non-Latin character handling, we did not test their real-world performance and thus cannot comment on their practical utility.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec19\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.4 Implications and Future Directions\\u003c/h2\\u003e \\u003cp\\u003eOur findings suggest that AI-aided screening tools are promising but remain underused in evidence synthesis. A significant barrier is the lack of awareness, training, and user support, particularly for researchers without computational backgrounds. Institutions can help close this gap by offering regular training workshops and integrating AI tools into research methodology curricula to increase both exposure and user confidence.\\u003c/p\\u003e \\u003cp\\u003eDevelopers also play a critical role in facilitating adoption. Usability and onboarding support should be prioritised to lower the entry barrier for new users and flatten the learning curve. Our experience with the PICO Portal team illustrates the positive impact of proactive engagement: the developers provided onboarding materials, live demonstrations, and responsive assistance, all of which helped address the steep learning curve and can encourage broader use. Such direct interaction between developers and end-users can significantly improve adoption and satisfaction, particularly for researchers unfamiliar with AI technologies.\\u003c/p\\u003e \\u003cp\\u003eFrom a development standpoint, there is clear room for improvement. Key areas include enhancing duplicate detection, implementing reliable study design recognition (e.g., distinguishing reviews from primary research), and offering transparent performance tracking. Additionally, improving multi-language support and non-Latin character recognition would expand the utility of these tools in global and multilingual research contexts.\\u003c/p\\u003e \\u003cp\\u003eAuthors of research studies themselves can also facilitate automation by clearly reporting study design and methodology in abstracts. This practice would help both human reviewers and AI tools accurately classify studies during the screening phase, which is particularly important in umbrella reviews where distinguishing between primary studies and reviews is essential but often requires full-text access.\\u003c/p\\u003e \\u003cp\\u003eFinally, while generative AI and no-code tools offer new opportunities for automation, free platforms still fall short of offering fully comprehensive solutions for complex, multidisciplinary reviews. Tools such as PICO Portal are moving in the right direction by incorporating progress tracking and detailed performance reports, but broader functionality without reliance on paywalls is essential to ensure equity, scalability, and wider adoption of AI-aided screening in systematic review practice.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"5. Conclusions\",\"content\":\"\\u003cp\\u003eThis study evaluated six freely available AI-aided abstract screening tools using an umbrella review as a test case. Our findings suggest that these tools are useful and increasingly accessible for supporting systematic reviews, even for researchers without programming experience. While most tools performed adequately in core screening tasks, limitations remain, particularly in early recall, duplicate removal, study design recognition, and multi-user functionality.\\u003c/p\\u003e \\u003cp\\u003eDespite these challenges, AI-aided screening tools hold strong potential to enhance the efficiency and consistency of evidence synthesis. For broader adoption, however, improvements in usability, feature completeness, and institutional training support are needed. As the field evolves, developers, researchers, and institutions all have a role to play in advancing and adapting these tools to better meet the practical demands of diverse review contexts.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003e6.1 Ethics approval and consent to participate\\u003c/p\\u003e\\n\\u003cp\\u003eNot applicable\\u003c/p\\u003e\\n\\u003cp\\u003e6.2 Competing interest\\u003c/p\\u003e\\n\\u003cp\\u003eThe authors declare that they have no competing interests\\u003c/p\\u003e\\n\\u003cp\\u003e6.3 Consent for Publication\\u003c/p\\u003e\\n\\u003cp\\u003eNot applicable\\u003c/p\\u003e\\n\\u003cp\\u003e6.4 Funding\\u003c/p\\u003e\\n\\u003cp\\u003eThis work was supported by the Evidence Synthesis Ireland SWAR Award.\\u003c/p\\u003e\\n\\u003cp\\u003e6.5 Author Contributions\\u003c/p\\u003e\\n\\u003cp\\u003eSA, RFH, LG, and MC contributed to the conceptualization of the study. SA, SMJ, CT, and NOK contributed to the methodology and to data analysis and interpretation. SA drafted the original manuscript. SA, SMJ, CT, NOK, RFH, LG, MC and MD reviewed and edited the manuscript. All authors read and approved the final version of the manuscript.\\u003c/p\\u003e\\n\\u003cp\\u003e6.6 Acknowledgements\\u003c/p\\u003e\\n\\u003cp\\u003eWe thank Dr Morteza Namvar for his valuable contributions to the discussions that informed the development of this study.\\u003c/p\\u003e\\n\\u003cp\\u003e6.7 Availability of data and material\\u003c/p\\u003e\\n\\u003cp\\u003eAll data generated or analysed during this study are included in this published article [and its supplementary information files]\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\u003cli\\u003e\\u003cspan\\u003eBeller EM, et al. Are systematic reviews up-to-date at the time of publication? Syst reviews. 2013;2(1):36.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eSampson M, et al. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61(6):531\\u0026ndash;6.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eAllers K, et al. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol. 2018;95:102\\u0026ndash;10.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHao Q et al. \\u003cem\\u003eAI Expands Scientists' Impact but Contracts Science's Focus.\\u003c/em\\u003e arXiv preprint arXiv:2412.07727, 2024.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePorto JR, et al. Quantifying the Scope of Artificial Intelligence\\u0026ndash;Assisted Writing in Orthopaedic Medical Literature: An Analysis of Prevalence and Validation of AI-Detection Software. JAAOS-Journal of the American Academy of Orthopaedic Surgeons; 2022. p. 105435.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003evan Dijk SH, et al. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ open. 2023;13(7):e072254.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eFabiano N, et al. How to optimize the systematic review process using AI tools. JCPP Adv. 2024;4(2):e12234.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eWallace BC et al. \\u003cem\\u003eDeploying an interactive machine learning system in an evidence-based practice center: abstrackr\\u003c/em\\u003e. in \\u003cem\\u003eProceedings of the 2nd ACM SIGHIT international health informatics symposium\\u003c/em\\u003e. 2012.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eCowie K, et al. Web-based software tools for systematic literature review in medicine: systematic search and feature analysis. JMIR Med Inf. 2022;10(5):e33219.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eMarshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst reviews. 2019;8(1):163.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVan Altena A, Spijker R, Olabarriaga S. Usage of automation tools in systematic reviews. Res synthesis methods. 2019;10(1):72\\u0026ndash;82.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eO\\u0026rsquo;Connor AM, et al. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst reviews. 2018;7(1):3.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eScott AM, et al. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol. 2021;138:80\\u0026ndash;94.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVan der Mierden S, et al. Software tools for literature screening in systematic reviews in biomedical research. Altex. 2019;36(3):508\\u0026ndash;17.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHarrison H, et al. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med Res Methodol. 2020;20(1):7.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eSchmidt L et al. \\u003cem\\u003eA narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses.\\u003c/em\\u003e Zeitschrift f\\u0026uuml;r Evidenz, Fortbildung und Qualit\\u0026auml;t im Gesundheitswesen, 2023. 181: pp. 65\\u0026ndash;75.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eBlaizot A, et al. Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synthesis Methods. 2022;13(3):353\\u0026ndash;62.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eDos Reis AHS, et al. Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study. Syst reviews. 2023;12(1):68.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKitchenham B, Linkman S, Law D. DESMET: a methodology for evaluating software engineering methods and tools. Comput Control Eng. 1997;8(3):120\\u0026ndash;6.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eThomas J, Brunton J, Graziosi S. \\u003cem\\u003eEPPI-Reviewer 4.0: software for research synthesis.\\u003c/em\\u003e EPPI-Centre Software. London: Social Science Research Unit, Institute of Education; 2010.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePrzybyła P, et al. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res synthesis methods. 2018;9(3):470\\u0026ndash;88.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHoward BE, et al. SWIFT-Review: a text-mining workbench for systematic review. Syst reviews. 2016;5(1):87.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eCheng S, et al. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conserv Biol. 2018;32(4):762\\u0026ndash;4.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eOuzzani M, et al. Rayyan\\u0026mdash;a web and mobile app for systematic reviews. Syst reviews. 2016;5(1):210.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKhalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol. 2022;144:22\\u0026ndash;42.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eDistillerSR. \\u003cem\\u003eDistillerSR: Literature Review Software [Internet].\\u003c/em\\u003e 2025.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKohl C, et al. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environ Evid. 2018;7(1):8.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eAromataris E, et al. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. JBI Evid Implement. 2015;13(3):132\\u0026ndash;40.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eDevane D, et al. Study within a review (SWAR). J Evidence-Based Med. 2022;15(4):328.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eAkaraci S. \\u003cem\\u003eSWAR 25: A comparison of artificial intelligence (AI) aided and manual reviewing in abstract screening.\\u003c/em\\u003e. 2023; Available from: \\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/SWARFileStore/SWAR25%20Selin%20Akaraci%20(2023%20FEB%2028%202026).pdf\\u003c/span\\u003e\\u003cspan address=\\\"https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/SWARFileStore/SWAR25%20Selin%20Akaraci%20(2023%20FEB%2028%202026).pdf\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eMarshall C, Brereton P. \\u003cem\\u003eSystematic review toolbox: a catalogue of tools to support systematic reviews\\u003c/em\\u003e. in \\u003cem\\u003eProceedings of the 19th international conference on evaluation and assessment in software engineering\\u003c/em\\u003e. 2015.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVan De Schoot R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125\\u0026ndash;33.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePICOPortal2025. \\u003cem\\u003ePICO Portal\\u003c/em\\u003e. 10.10.2025]; Available from: \\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://picoportal.net/\\u003c/span\\u003e\\u003cspan address=\\\"https://picoportal.net/\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eMinion JT, et al. Pico portal. J Can Health Libr Association. 2021;42(3):181.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eGates A et al. \\u003cem\\u003eDecoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions\\u003c/em\\u003e. 2020.\\u003c/span\\u003e\\u003c/li\\u003e\\u003c/ol\\u003e\"},{\"header\":\"Table 1\",\"content\":\"\\u003cp\\u003eTable 1 is available in the Supplementary Files section.\\u003c/p\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":true,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":false,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":true,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"systematic-reviews\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"sysr\",\"sideBox\":\"Learn more about [Systematic Reviews](http://systematicreviewsjournal.biomedcentral.com/)\",\"snPcode\":\"\",\"submissionUrl\":\"https://www.editorialmanager.com/sysr/default.aspx\",\"title\":\"Systematic Reviews\",\"twitterHandle\":\"@MedicalEvidence\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"BMC/SO AJ\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":true},\"keywords\":\"\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-7915337/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-7915337/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003ch2\\u003eBackground:\\u003c/h2\\u003e \\u003cp\\u003eManual abstract screening in systematic reviews is a time-consuming and labour-intensive task. With the rise of artificial intelligence (AI), the number of published articles has grown substantially, adding to the workload of review studies that rely on robust and timely evidence synthesis. At the same time, AI-aided screening tools have been developed to accelerate this process. While previous studies have demonstrated the efficiency of such tools, ongoing technological advances necessitate updated evaluations, particularly for tools that are freely accessible. In review types such as umbrella reviews, where both the topic area and study design are central to eligibility decisions, the performance of AI-aided tools remains underexplored.\\u003c/p\\u003e\\u003ch2\\u003eMethods:\\u003c/h2\\u003e \\u003cp\\u003eWe conducted a comparative evaluation of six freely available AI-aided abstract screening tools: Rayyan, RobotAnalyst, PICO Portal, Abstrackr, ASReview, and Colandr using a previously completed umbrella review of interdisciplinary urban planning and public health studies. We assessed (1) early recall performance (i.e., identification of included studies within the first 25% of screening), (2) feature availability and depth, and (3) user experience. This Study WIthin a Review (SWAR) was registered in the SWAR repository as SWAR 25.\\u003c/p\\u003e\\u003ch2\\u003eResults:\\u003c/h2\\u003e \\u003cp\\u003eAll evaluated tools supported the review process by facilitating screening and offering features such as prioritization and keyword highlighting. However, none identified more than 50% of the previously included studies within the first 25% of screening. Feature analysis and user feedback suggested that Rayyan and PICO Portal provided the most useful functionality for our interdisciplinary umbrella review context, although limitations were noted in duplicate removal and in recognising the importance of study design in eligibility decisions.\\u003c/p\\u003e\\u003ch2\\u003eConclusions:\\u003c/h2\\u003e \\u003cp\\u003eAlthough a growing number of AI-assisted abstract screening tools are publicly and freely available, their accuracy, usability, and adaptability to different review designs remain limited. Enhanced support for duplicate detection and integration of study design considerations could improve their utility in umbrella reviews and other complex evidence syntheses. Continued evaluation and user training may support broader adoption across diverse research contexts.\\u003c/p\\u003e\",\"manuscriptTitle\":\"Evaluating the use of Artificial Intelligence (AI) in Systematic Review Abstract Screening: A Comparative Study of AI-aided Tools\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2026-01-21 03:30:57\",\"doi\":\"10.21203/rs.3.rs-7915337/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0},{\"type\":\"decision\",\"content\":\"Major revision\",\"date\":\"2026-02-21T20:37:41+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"\",\"date\":\"2026-02-01T12:02:15+00:00\",\"index\":0,\"fulltext\":\"\"},{\"type\":\"reviewersInvited\",\"content\":\"\",\"date\":\"2025-12-13T07:55:55+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorAssigned\",\"content\":\"\",\"date\":\"2025-11-11T21:33:40+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"submitted\",\"content\":\"Systematic Reviews\",\"date\":\"2025-10-21T12:37:48+00:00\",\"index\":\"\",\"fulltext\":\"\"}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"systematic-reviews\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"sysr\",\"sideBox\":\"Learn more about [Systematic Reviews](http://systematicreviewsjournal.biomedcentral.com/)\",\"snPcode\":\"\",\"submissionUrl\":\"https://www.editorialmanager.com/sysr/default.aspx\",\"title\":\"Systematic Reviews\",\"twitterHandle\":\"@MedicalEvidence\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"BMC/SO AJ\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"2539e76b-2027-41a0-91b1-e9a1af04944c\",\"owner\":[],\"postedDate\":\"January 21st, 2026\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"under-review\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2026-04-28T23:49:07+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2026-01-21 03:30:57\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-7915337\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-7915337\",\"identity\":\"rs-7915337\",\"version\":[\"v1\"]},\"buildId\":\"XKTyCvWXoU3ODBz1xrDgd\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}