Designing an AI-Driven SLR Workflow for Academic Research: A Rubric for Comparative Analysis of AI Tools

Designing an AI-Driven SLR Workflow for Academic Research: A Rubric for Comparative Analysis of AI Tools | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Case Report Designing an AI-Driven SLR Workflow for Academic Research: A Rubric for Comparative Analysis of AI Tools Nisha Biju, Alisha Sinha, Sarthak Saini, Ganesh Beniwal, Syaamantak Das This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6328602/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 29 Dec, 2025 Read the published version in Discover Artificial Intelligence → Version 1 posted You are reading this latest preprint version Abstract Systematic Literature Reviews (SLR) are a critical aspect of academic research, as they provide a comprehensive and rigorous synthesis of existing evidence. However, conducting such reviews presents several challenges that researchers must be aware of to ensure the quality and validity of their work. One of the most significant challenges is developing a clear research question through a comprehensive literature search. This is because it can be difficult to identify appropriate and relevant studies. Additionally, the traditional literature review process has the potential for bias. To address these challenges, researchers need a search protocol and strategy that includes a set of available artificial intelligence-based tools as an ad-hoc tech stack (workflow) to conduct SLR. This study evaluated a proposed model of a workflow and stack of such tools using a rubric to identify the best possible set of tools for the task. For one-year, several AI-based tools were evaluated in two phases, and a proposed workflow using the best of those tools was identified. Sixty-three students used the proposed model to conduct their academic research. The results showed that it is possible to perform academic research more efficiently if a workflow is used for performing SLR. Systematic Literature Review Academic Research Information Search Figures Figure 1 1. Introduction A Systematic Literature Review (SLR) is a rigorous and transparent method for identifying, evaluating, and inter- preting existing research in a specific field [Garc´ıa-Pen˜alvo, 2022]. It aims to fill knowledge gaps and guide future research by integrating and summarizing empirical studies on a particular topic [Ferreras-Ferna´ndez et al., 2016 ]. This type of review is particularly useful in academic research, where it can provide a comprehensive understanding of the existing evidence [Pati and Lorusso, 2018]. The process involves several key phases, including planning, conducting, and reporting the review[Garc´ıa-Pen˜alvo, 2022]. To ensure methodological rigour and quality, researchers adhere to standardized methodologies and guidelines, such as the PRISMA statement [Pati and Lorusso, 2018]. Conducting an SLR is a challenging task because it requires meticulous attention to detail, critical thinking, and a deep understanding of the research topic. The work by [Anderson and Jayaratne, 2015] highlights the need for a clear research question, comprehensive literature search, and rigorous quality assessment to complete SLR. Early work by [McManus et al., 1998] emphasizes the difficulty in identifying relevant studies, particularly in new or interdisciplinary fields, and suggests the use of expert contacts. A recent work by [Haddaway et al., 2020] discusses the potential for bias in traditional literature reviews and the importance of rigorous methods in systematic reviews. Despite its importance in providing a comprehensive overview of the existing knowledge in a particular field, current SLR methodologies often fall short in several aspects. Firstly, the sheer volume of published research pa- pers poses a significant challenge, with the number of scholarly articles published annually increasing every year [Chu and Evans, 2021]. This surge in publications makes it increasingly difficult to identify, evaluate, and synthesize all relevant studies. Traditional manual searches often rely on keywords, which can lead to missed references or biased results. Furthermore, publication bias and selective reporting can result in an incomplete picture of the available evidence, with studies with positive or significant results more likely to be published than those with negative or inconclusive findings [Mlinari´c et al., 2017]. To overcome these limitations, systematic literature reviews must employ innovative strategies, such as machine learning algorithms and automated screening tools, to efficiently and accurately identify relevant studies [van Dinter et al., 2021]. 1.1 Application of AI for SLR Artificial Intelligence (AI) can significantly improve the SLR process for academic research by automating various tasks, making the process more efficient, accurate, and comprehensive. A range of studies have explored the potential of AI-based tools to automate SLR. AI provides methods to represent and infer knowledge, efficiently manipulate texts, and learn from vast amounts of data, which apply to the analysis of scientific literature [de la Torre-Lo´pez et al., 2023 ]. By using AI software designed to automate systematic review, researchers can expedite the search process, making it more practical, sustainable, and cost-effective 1 . AI tools can constantly search databases for the latest research, support evidence synthesis, and ensure that relevant papers are incorporated into the review to keep it up-to- date[Atkinson, 2023]. AI algorithms also ensure an improved way of error-free data extraction and statistical analysis compared to humans, leading to increased accuracy[Collins et al., 2021]. Recent work by [Blaizot et al., 2022], en- dorsing a previous work of [Cohen et al., 2010], found that AI methods can enhance efficiency and quality in evidence synthesis, with the latter demonstrating high accuracy in literature classification. However, human validation is still crucial in implementing these tools. The work by [Feng et al., 2022] emphasized the importance of recall in AI models for literature screening, while the work by [Das and Islam, 2021] highlighted the wide range of AI and machine learn- ing techniques being used in libraries, including for collection management and user interaction. When information search principles are applied appropriately, SLR have a clear advantage over traditional literature reviews, and the advent of AI-powered tools further enhances this advantage. 1.1.1 Technological Challenges The use of AI tools for SLR in academic research presents several technological challenges. The work of [Blaizot et al., 2022] also highlights the need for extensive human validation in the implementation of AI methods, indicating that these tools are not yet fully autonomous. The work of [Sharadgah and Sa’di, 2022 ] emphasize the potential of AI in en- hancing efficiency and quality in evidence synthesis, but also underscores the importance of detailed methodological descriptions and the need for further exploration of the challenges and risks associated with AI. Further analysis is essential to comprehend the pedagogical, ethical, and cultural aspects of AI-enhanced education comprehensively. These studies collectively suggest that while AI tools hold promise for improving the SLR process, their full potential can only be realized through continued research and development. 1.1.2 Pedagogical Challenges The pedagogical challenges of using AI for SLR in academic research include addressing teacher capability issues, as teachers need to be qualified to teach AI tools competently and effectively. Misconceptions about AI among teachers are common, and there is a need for external content experts to prepare teachers’ content delivery knowledge [Dai, 2023]. AI in education is not just about technology but involves social and historical factors, including technical developments, scientific practices, institutional applications, and power struggles[Williamson, 2023]. The integration of AI into classrooms is expected to continue serving as a space for the democratic formation of public thought and concepts related to various aspects of life [Mouta et al., 2023]. Moreover, the adoption of AI in education faces challenges such as the selection of suitable learning programs, evaluation of student learning outcomes, and fostering teachers’ professionalism and self-efficacy for AI-integrated lessons. Teachers need to have sufficient knowledge related to AI tools and technologies to understand and effectively use the educational roles of AI [Park et al., 2023]. Additionally, the use of AI in education has been influenced by various national strategies and policies, with countries like the United States, the United Kingdom, Germany, Japan, and China releasing policies related to intelligent education. These policies emphasize the integration of AI into the teaching process and the promotion of AI applications throughout the educational system [Shi et al., 2022]. The rapid development of AI calls for innovation, and various countries have recognized the importance of AI education in cultivating new talents and have begun to incorporate AI into educational policies and teaching models. 2. Literature Review 2.1 Information Search: An overview of Protocol and strategies The information search protocols and strategies adopted by humans can vary depending on the context, such as academic research, decision-making, or online information seeking. In academic research, a systematic approach to searching is essential, involving the development of librarian-mediated searches for systematic reviews and medical literature. This includes steps such as formulating a question, creating thorough search strategies in multiple databases, and having the search terms checked by relevant experts [Bramer et al., 2018]. In decision-making research, two process tracing techniques – explicit information search, and verbal protocols are used to understand the information processing strategies that individuals use in reaching a decision [Payne, 1976]. When it comes to online information seeking, humans use various strategies, such as central or peripheral processing, based on information processing theories. For example, the limited capacity model of message processing suggests that people usually process part of the message provided due to limited capacity. Additionally, the information-foraging theory posits that humans choose behaviours that tend to optimize the utility of information gained [Mohamed et al., 2023]. Research on information search protocols and strategies has revealed several key findings. The work by [Rodi et al., 2017] identified a pattern in Wikipedia users’ navigation, where they start with broad topics and progressively narrow down their search. This suggests a strategy of moving from the general to the specific. The work of [Sacchi and Burigo, 2008] found that individuals tend to use a sequential strategy when the information source is perceived as reliable, regardless of their knowledge level. This indicates a reliance on the credibility of the source. The study by [Joho et al., 2015] highlighted the use of temporal expressions in search queries, with mixed success, and the difficulty in finding future information. A recent work [Schiff and Mo¨ller, 2021] discussed the optimization of user interaction with information retrieval systems, emphasizing the need for human-aware collaborative planning strategies. These studies collectively suggest that humans employ a range of strategies in information search, including a progression from general to specific, reliance on source credibility, and the use of temporal expressions. 2.2 Challenges with traditional SLR process Traditional literature review processes have several challenges that can lead to biased or incorrect conclusions. These challenges include a lack of appropriate critical appraisal of study validity, treating all evidence as equally valid, and limited stakeholder engagement leading to a review that is of limited relevance 2 [Haddaway et al., 2020]. Challenges associated with information search in SLR for academic research include the difficulty in defining precise research questions, selecting relevant databases, and establishing appropriate search terms [Chong et al., 2022]. The process often requires a pilot study to refine search strategies, and the use of Boolean operators to combine search terms can be complex. Additionally, the inclusion and exclusion criteria must be clearly defined to ensure relevant literature is captured. The screening process is usually conducted in two levels: initial screening of titles and abstracts, followed by full-text screening. Moreover, the vast number of records retrieved from databases like Google Scholar can make the search inefficient. The involvement of librarians in the systematic review process can impact the quality of the search strategy, with activities ranging from consulting on selecting resources to designing complete search strategies [Eskrootchi et al., 2020]. SLR has a clear advantage over traditional literature reviews when systematic review princi- ples are applied sensitively[Mallett et al., 2012]. Systematic reviews rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing bias. However, despite the growing interest in systematic reviews, traditional approaches to reviewing the literature continue to persist in contemporary publications. 2.2.1 Limitations of PRISMA The limitations of SLR models like PRISMA include publication bias, as PRISMA’s efforts may not completely eliminate it, and its primary design for quantitative data, making it less suitable for qualitative research or other evidence synthesis approaches [Santos et al., 2023]. PRISMA is also criticized for not being designed for reviews that involve narrative, qualitative, or mixed methods rather than quantitative methods, and for its heavy emphasis on meta- analysis, which excludes other synthesis methods [Haddaway et al., 2018]. Additionally, PRISMA may not adequately handle novel review outputs like systematic maps, and it may not be easily adapted for methods that rely more on the earlier stages of the review process (searching and screening). Furthermore, PRISMA’s suggested requirements for review conduct are minimal, affecting the overall comprehensiveness of the review [Haddaway et al., 2018]. There is also a need for guidance regarding the reporting of systematic reviews of outcome measurement instruments (OMIs), as some components of PRISMA items are of limited relevance to such reviews [Elsman et al., 2022]. 2.3 Research Gaps To overcome the challenges of conducting an SLR, a search protocol and strategy (an ad-hoc tech stack-based workflow) that utilizes various artificial intelligence-based tools are required. The following research questions were explored in this study: What is the proposed model of a search protocol and stack of tools for conducting SLR using AI-based tools? How can artificial intelligence-based tools be used (like a tech stack) to address the challenges of conducting literature reviews? How can a rubric be used to evaluate and identify the best set of tools for conducting SLR using AI-based tools? This study evaluated a proposed model of a search protocol and stack of such tools using a rubric to identify the best set of tools for the task. It involved Sixty-three students who used the proposed model to conduct their academic research. 3. Study Design There are several studies available on the use of AI tools for SLR in academic research. One such study [de la Torre-Lo´pez et al., 2023] provides a survey of AI techniques proposed in the last 15 years to help researchers conduct systematic analyses of scientific literature. The study analyzes the AI techniques currently available, with special emphasis on their purpose, inputs and outputs, and human intervention, if any. Another study [Myllyaho et al., 2021] focuses on the validation methods used for practical AI systems reported in the literature. Additionally, a systematic review [Gonza´lez-Calatayud et al., 2021] analyzes the use of AI for student assessment, while another systematic literature review [Ramesh and Sanampudi, 2022] provides an overview of automated essay scoring systems. However, there is no specific rubric available to evaluate AI tools used for systematic literature review in academic research. The study works in two phases: Phase 1: A pilot study analysing the experiences of 18 participants doing SLR using the four steps mentioned below. Phase 2: A detailed study involving 45 students. It had an additional final tool (Jenni.ai) which was recommended for report writing. An SLR has several process steps which are defined differently in the literature. For this study, the common steps involved in performing an SLR for academic research were identified which are as follows: 1. Step 1: Formulating tentative research questions : Getting started with a basic research question in the beginning. 2. Step 2: Identifying a base(seed) paper: To conduct a literature review, one needs to find an initial base(seed) paper [Sesagiri Raamkumar et al., 2018], build a reading list from it, understand how the paper builds upon previous research, and identify follow-up papers. 3. Step 3: Assessing the quality of the literature: It is common for the method used in a research study to be slightly complex to comprehend in the original publication. Therefore, researchers must understand the study’s goals and areas of investigation by examining how other scholars have cited it in their work. This enables the assessment of the literature’s quality. 4. Step 4: Analyzing and synthesizing additional evidence from extracting data : Systematically collect data from the included studies based on paper content. Note: This research aims to identify the right set of information using AI-based tools for SLR processes. Writing and other additional steps are beyond the scope of this research. However, the scope of this research is limited to identifying the right set of information using AI-based tools. 3.1. Developing the Rubric The evaluation of each tool was done against a set of three different possible scores (Score 3, Score 2, Score 1) for a given evaluation criterion. The criterion was further divided based on certain parameters which are mentioned for each step: 3.1.1. Rubric for step 1: For step 1, A) expediting of the search process, B) features of the tools, C) navigation and structure and D ) specific purpose of the tool were chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows: 1. Criteria (A): Expedite the Search process Sub-criteria (i): Use of keywords – SLR tools were evaluated based on their ability to expedite the search process and use of keywords. Each tool was allocated a score based on performance. • Score 3: The tool identifies and summarizes the literature based on a research question, even if the papers do not match the keywords . • Score 2: The tool identifies and summarizes the literature based on a research question, only if the papers match the exact keywords . • Score 1: The tool does not summarize the literature based on a research question on the basis of keywords. 2. Criteria (B): Features of the tools Sub-criteria (i): Summary of Abstract – The tools were evaluated based on their ability to generate summaries of relevant papers. • Score 3: The tool allows the user to get a summary of the top N papers along with the author’s name, year of publication and citations of papers. • Score 2: The tool allows the user to get a summary of the limited papers along with limited information about the author’s name, year of publication and citations of papers. • Score 1: The tool allows the user to get a summary of the limited information about the author’s name, year of publication and citations of papers . Sub-criteria (ii): Diversity of Choices – The tools cater to the unique needs and preferences of a diverse range of users. • Score 3: The tool is designed to address the diverse needs of users. E.g. searching papers based on study type (e.g. RCT (randomized controlled trial)) AND time frame (within the last N years). • Score 2: The tool is designed to address the limited needs of users. E.g. searching papers based on study type (e.g. RCT) OR time frame (within the last N years). • Score 1: The tool is designed to address the basic needs of users. E.g. searching papers based ONLY ON the time frame (within the last N years). Sub-criteria (iii): Accessibility – Cost incurred to use the tool • Score 3: Freemium model. • Score 2: Free basic model. • Score 1: Subscription required. Sub-criteria (iv): Library options – Option to curate specific group of papers. • Score 3: The tool can categorize and save papers . • Score 2: The tool can’t categorize but save papers . • Score 1: The tool can’t categorize or save papers. Sub-criteria (v): Share/Export options – to save a paper for future usage. • Score 3: Both import and export available . • Score 2: Export available but can’t import or vice-versa. • Score 1: No import/export available. 3. Criteria (C): Navigation and structure Sub-criteria (i): Clear and Organized Menu – Are the tools having clear instructions for navigation? • Score 3: Links for navigation are clearly labelled and placed consistently. • Score 2: Links for navigation are not clearly labelled and placed. • Score 1: Links for navigation are not available for most features. Sub-criteria (ii): Ease of Navigation • Score 3: Enables smooth navigation with minimal chances of getting lost. • Score 2: Links are missing, causing users to get lost. • Score 1: Navigation options are limited, causing users to get lost when leaving the page due to missing links. 4. Criteria (D): Specific purpose of the tool Sub-criteria (i): Finding academic research papers • Score 3: The user can effortlessly locate research papers relevant to their topic, with a high degree of precision . • Score 2: The users can generally find relevant research papers, but the process might require some manual refinement and additional time . • Score 1: Users struggle to find relevant research papers, and the search results often yield outdated or irrelevant content . Sub-criteria (ii): Discovery of concepts • Score 3: Discovering concepts across papers is moderately effective but may have some limitations. The tool generally identifies relevant concepts but may occasionally miss subtler or less common ones. • Score 2: Discovering concepts across papers is largely ineffective . Users struggle to identify relevant concepts, and the tool’s results often miss critical themes or connections. • Score 1: No feature or functionality for discovering concepts across papers within the system. Users are left to rely entirely on manual methods and external resources for identifying and connecting concepts in the literature. Sub-criteria (iii): Meta-Information extraction from paper • Score 3: The utility of the tool for extracting information from papers is highly effective . It consistently and accurately extracts information from a 10 to 25 number of papers with ease. • Score 2: The utility of the tool for extracting information from papers is moderately effective but may have some limitations. • Score 1: The utility of the tool for extracting information from papers is mostly ineffective . 3.1.2. Rubric for step 2: For step 2, A) Identifying base paper (seed paper) and B) Generation of knowledge graph were considered as criteria: The sub-criteria for each of these are as follows: A) Seed paper search, B) Visualization of connected(related) papers, C) Search Sharing and D) Specific purpose of the tool were chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows: 1. Criteria (A): Identifying base paper (Seed paper) Sub-criteria (i): Seed paper search • Score 3: The tool searches the list of papers that are most relevant to the topic from the keywords given. • Score 2: The tool searches the list of moderately relevant papers , prior and derivative works based on the given paper as a query. • Score 1: The tool searches the list of not-so-relevant papers based on the citation of the paper, provided as a query. Sub-criteria (ii): Generation and Visualization of knowledge graph • Score 3: The tool presents the papers that are related in a legible, interactive, and dynamic visualization. • Score 2: The tool provides a visualization that is legible, and dynamic but not interactive. • Score 1: The tool provides a legible visualization of the papers only. Sub-criteria (iii): Search Sharing • Score 3: The tool allows the sharing of the result, as visualizations, list of papers, and to reference manager. • Score 2: The tool allows the sharing of the results as visualizations and a list of papers only. • Score 1: The tool allows the sharing of the result in the form of a list of papers only. Sub-criteria (iv): Data export • Score 3: Allows users to export data and reports in a variety of formats, such as BIB, BibTeX, CSV, RIS and JSON. • Score 2: Allows users to export data in CSV or PDF but exporting search results from a database into a citation management tool using RSV (Rich Structured View) is not available. • Score 1: Partially supports data/report export, such as by allowing users to export data only in one format. 2. Criteria (B): Creating and revising Knowledge graph Sub-criteria (i): No. of connected papers generated • Score 3: Greater than 30. • Score 2: Between 20 and 30, • Score 1: Between 10 and 20. Sub-criteria (ii): Creating knowledge graph • Score 3: The tool offers an intuitive user interface, supports multiple data import formats, and automated entity recognition features. • Score 2: The tool provides a user-friendly interface and supports at least one data import format. • Score 1: The tool allows for rudimentary graph creation with limited data import options. Sub-criteria (iii): Revising knowledge graph • Score 3: The tool offers real-time, visual, and bulk editing features, along with advanced validation checks and role-based access control. • Score 2: The tool provides basic editing features and includes either validation checks or role-based access control. • Score 1: The tool allows for limited editing capabilities but lacks comprehensive features for validation and access control. 3.1.3. Rubric for step 3: For step 3, A) Advanced features, B)Visual interface, and C) specific tasks of the tool were chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows: 1. Criteria (A): Advanced features Sub-criteria (i): Boolean Search • Score 3: The tool allows users to get full accessibility like boolean search using (“AND”, “OR”, “NOT”), Brackets, Wildcard searches, Fuzzy matching and Proximity matching. • Score 2: The tool allows users to get limited accessibility like boolean search using (“AND”, “OR”, “NOT”), Brackets, Wildcard searches, Fuzzy matching, and Proximity matching. • Score 1: The tool fails users to provide accessibility like boolean search using (“AND”, “OR”, “NOT”), Brackets, Wildcard searches, Fuzzy matching, and Proximity matching. Sub-criteria (ii): Reference Check • Score 3: The tool displays a list of reliable references that have been cited and have no significant disputes or retractions. • Score 2: The tool displays a list of references with partial reliability, which may have value but should be used with caution. Some references cited may have disputes or disagreements. • Score 1: The tool does not verify the reliability and citation of referenced lists. 2. Criteria (B): Specific task of the tool Sub-criteria (i): Visual interface • Score 3: Enables smooth navigation with minimal chances of getting lost. • Score 2: Links/Icons are missing, causing users to get lost. • Score 1: Navigation options are limited, causing users to get lost when leaving the page due to missing links. Criteria (B): Specific task of the tool Sub-criteria (i): Critical Analysis • Score 3: The tool allows users to critically engage with publications, understand how a publication and its results have been cited, and find relevant literature on the topic in question. • Score 2: The tool allows users to access limited engagements with publications, understand how a publi- cation and its results have been cited, and find relevant literature on the topic in question. • Score 1: The tool allows users to access limited publications but does not allow them to understand how a publication and its results have been cited, or to find relevant literature on the topic in question. Sub-criteria (ii): Identifying research gaps • Score 3: The tool was highly effective for identifying and explaining the gaps in research by analyzing how research papers have been cited and the context in which these citations occur. • Score 2: The tool was partially effective for identifying and explaining the gaps in research by analyzing how research papers have been cited and the context in which there are few or no citations. • Score 1: The tool fails to identify and explain the gaps in research by analyzing how research papers have been cited and the context in which there are no citations. Sub-criteria (iii): Custom dashboard • Score 3: The tool allows to import of files from Zotero or Mendeley library and the upload of CSV files. • Score 2: The tool does not allow importing files from Zotero or Mendeley library but uploads file formats like PDF or CSV. • Score 1: The tool does not allow to import library or PDF but can upload CSV files. 3.1.4. Rubric for step 4: For step 4, A) Advanced features, and B) Question Answering interface were chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows: 1. Criteria (A): Advanced features Sub-criteria (i): Quality of Reference Sources • Score 3: The tool produces credible references from academic articles that are published in indexed journals. • Score 2: The tool produces credible references from web resources only. • Score 1: The tool does not produce any credible reference for the answer. Sub-criteria (ii): Synthesis of Answers • Score 3: The tool provides options to synthesize results to the questions, through options such as summary, popular opinion and number of positive vs. negative reviews. • Score 2: The tool provides some form of options to synthesize results from the answers. • Score 1: The tool provides no options to synthesize the answers. Sub-criteria (iii): Sharing of Consolidated Output • Score 3: Sharing of full results of the query as a text document. • Score 2: Sharing of partial results of the query. • Score 1: Does not allow sharing of results, the user has to manually copy-paste the results. 2. Criteria (B): Usability feature Sub-criteria (i): Question Answering interface • Score 3: A clear option for asking questions is indicated on the tool and the user is presented with the follow-up information. • Score 2: Options for asking questions are laid out on the application, however, the flow is not streamlined. • Score 1: Do not have such an option. 3.2. Benchmark Identification In 2023, a survey was conducted by paperpile.com [3] on systematic literature search. The survey found that Google Scholar is the most extensively used tool for this purpose. Despite its popularity, a 2013 study [Boeker et al., 2013] suggested that Google Scholar is not the most optimal tool for systematic literature review. Nevertheless, due to its widespread usage and comprehensive coverage, Google Scholar can serve as a reference point for comparing AI-powered tools for literature searches. Its advantages include advanced search capabilities, continuous updates, user familiarity, accessibility, integration, and industry-standard status. However, Google Scholar also has certain limitations, such as limited search filters, varied quality of sources, language restrictions, citation bias, lack of transparency in its algorithms, redundancy in search results, metadata inconsistencies, and limited disciplinary filters. Additionally, it lacks some critical features for systematic literature retrieval, like tools for incremental query optimization, export of a large number of references, a visual search builder, or a history function. 3.3. Dataset Phase 1: 1. No. of AI tools: Initially, 15 AI tools were analyzed for this research. Based on the final ranking top 3 tools for each level are shown in Table 4. 2. Period: June 2023 - December 2023. It should be noted that these tools are constantly being upgraded and new features are added. This ranking is based on the tool’s features during a specific period. 3. Participants: 18 researchers doing active research in various domains of postgraduate qualification took part in this study as a part of a University course. Phase 2: 1. No. of AI tools: 6 AI tools formulating a workflow were recommended to students for this phase. 2. Period: June - July 2024 3. Participants: 45 students interested in learning about AI tools for SLR using a certificate course. 3.4. Identification of Appropriate AI tools for SLR Here’s a step-by-step process used to analyze a set of AI tools for comparison, using the given rubric for the study: 1. Gathering the AI tools - Identifying the specific AI tools from an initial superset of AI tools. 2. Finetuning the rubric - Creating own customized rubric based on specific needs for comparing the tools. Ensuring that the rubric covers relevant aspects like features, performance, usability, and cost. 3. Define scoring criteria - Each rubric item had clear criteria for scoring. 4. Gathering evidences - Collecting relevant data to support the evaluation process. This includes screenshots, official documentation (including whitepapers), publicly available user reviews, etc. 5. Prioritize key findings: Looking for patterns and trends across the data to identify the most important factors for decision-making. 6. Make recommendations: Recommend the most suitable tool(s) based on the specific needs and priorities. Clearly articulate the rationale behind the recommendations. 7. Considering limitations: Acknowledging any limitations of the analysis, such as the subjectivity of the rubric or incomplete data. The Identified tools were arranged in a workflow and recommended to the students in Phase 2. 4. Observation and Analysis 4.1. Observations from Phase 1 From the common steps involved in performing an SLR for academic research, as identified in the study design, it was observed that in step 1, Semantic Scholar had the highest score of 33, followed by Elicit and PaperDigest scoring 29 each. The benchmark Google Scholar’s score is 26. In step 2, Litmaps scored the highest (score 21), followed by Open Knowledge Map (score 20) and Connected papers (score 17). The benchmark Google Scholar’s score was 5. For step 3, only Scite.ai (score 18) was available which satisfied all the given parameters. It must be noted that Google Scholar is not an ideal benchmark to evaluate step 3 of the SLR process. Scite.ai and Google Scholar are both useful for academic research, but they differ in their approaches and functionalities. Scite.ai provides richer context with “Smart Citations”, summaries, and advanced sorting, while Google Scholar offers basic sorting and no summarization feature. Scite.ai aims to show diverse results by highlighting under-cited articles, while Google Scholar can sometimes return repetitive results or prioritize highly cited papers. Scite.ai offers a free basic plan and premium plans with more functionalities, while Google Scholar is free to use. Both offer saving options, though Scite.ai integrates with external software, and Scite.ai offers more varied import/export options. And finally, for step 4, Consensus.app (score 12) was the highest in the category followed by perplexity (score 9). Again, Google Scholar is not an ideal benchmark to compare these AI tools. 4.1.1. The proposed workflow 1. Step 1: Semantic Scholar - For initial paper search. 2. Step 2: Litmaps - Identify seed paper and create a knowledge graph of related papers. 3. Step 3: Scite.ai - Find relevant validation about the paper. 4. Step 4: Consensus.ai - Get a summary with proper references. 4.1.2. Feedback from users Eighteen postgraduate students from various academic fields, such as virtual reality, eye-tracking, physical education, and information retrieval, were asked to conduct a Systematic Literature Review (SLR) using a proposed workflow. The proposed workflow was evaluated using rubrics, and 77% of the students rated it 4 or higher on a scale of 5. Of these students, 72.7% reported that the search functionality of the workflow significantly improved their overall SLR process in terms of accuracy, completeness, and ease of use. Additionally, 72.8% of the students acknowledged that the proposed workflow improved their efficiency in conducting SLR. Regarding the accuracy of the results, 72.7% rated it 4 or higher on a scale of 5. Moreover, 91% of the students agreed that the proposed model helped them complete the SLR process faster and with fewer errors. They also expressed their willingness to recommend the proposed workflow to others. A combination of user responses is shown in Table 1. 4.2. Observations from Phase 2 The study’s recommendation of the Systematic Literature Review (SLR) workflow, involving AI tools, reveals positive feedback from participants. Out of 45 respondents, 29 rated the workflow as highly effective (either 4 or 5 on a 5-point scale), with several citing its ease of use and the ability to efficiently identify relevant literature as key benefits. The integration of AI tools such as Litmaps, SciSpace, and Jenni.ai improved the speed and accuracy of their research, allowing for better data-driven decisions. Many participants, with over 60% indicating satisfaction, highlighted the workflow’s ability to reduce errors and improve overall research productivity. Additionally, 32 respondents appreciated the replacement of general search engines like Google with more specialized AI-driven tools, which significantly increased their satisfaction. 4.2.1. The proposed workflow 1. Step 1: Elicit - For initial paper search. 2. Step 2: Litmaps - Identify seed paper and create a knowledge graph of related papers. 3. Step 3: scispace - Find relevant validation about the paper. 4. Step 4: Consensus.ai - Get a summary with proper references. 5. Step 5: Jenni.ai - writing. 4.2.2. Feedback from users The feedback from users on the AI-driven SLR workflow was largely positive. Out of 45 respondents, 32 found the workflow useful for identifying relevant literature more efficiently than traditional search methods like Google. Ease of use and increased efficiency in conducting SLRs were frequently mentioned, with many participants also appreciating the use of data-driven parameters to improve accuracy and reduce errors. The most favoured formats for further learning were recorded lectures (MOOC format), preferred by 20 participants, and online workshops, preferred by 15. Overall, the workflow was well-received, and many expressed interest in attending future courses. 4.2.3. Frequency of Top 5 Feature Combinations A Combination analysis was conducted on the data obtained from user feedback to identify the feature preferences. It was also done to identify which alternative stacks or combinations students would prefer. Only the top 5 feature combinations were considered for this analysis. We coded the features of the workflow as follows: A: Searching for and identifying related and relevant literature, instead of using Google search. B: Increased efficiency in conducting SLR. C: Ease of use. D: Use data-driven parameters to increase the accuracy of results and reduce the chances of errors. The most common combination is all four features together (A+B+C+D), chosen by nearly 29% of respondents (shown in Fig. 1). This suggests that a significant portion of users find value in the entire feature set. The second most common combinations are A+B+C and A+B+D, each selected by 13.33% of respondents. This indicates that when not selecting all features, users often choose three out of the four, with either ease of use or data-driven parameters being omitted. Feature A (Searching for and identifying related literature) appears in most of the top combinations, reinforcing its importance as identified in earlier analyses. Very few respondents selected only one feature, suggesting that most users find value in multiple aspects of the workflow. There’s a strong correlation between features A and B, as they appear together in many of the top combinations. While “Ease of Use” (C) is often selected, it’s not always chosen in combination with other features, which could indicate that some users might prioritize functionality over ease of use. Table 1: Results of the combination analysis Feature Combination Number of Responses A+B+C+D 13 (28.89%) A+B+C 6 (13.33%) A+B+D 6 (13.33%) A+C 4 (8.89 %) A+B 4 (8.89 %) A only 3 (6.67 %) A+C+D 2 (4.44 %) B+C 2 (4.44 %) B+D 1 (2.22 %) C+D 1 (2.22 %) B only 1 (2.22 %) C only 1 (2.22 %) D only 1 (2.22 %) Based on the combination analysis of the workflow features, we can draw several important insights about user preferences and the perceived value of different feature combinations. The most striking observation is that nearly 29% of respondents found all four features useful, suggesting that a comprehensive approach to literature review assistance is highly valued. This preference for a full-featured solution indicates that users appreciate a tool that addresses multiple aspects of the systematic literature review process simultaneously. 5. Discussion 5.1. Comparing Elicit, Semantic Scholar and Paper Digest Elicit is praised for its unique ability to predict research outcomes, while Semantic Scholar is noted for its extensive literature search capabilities and comprehensive coverage of academic databases. PaperDigest is highlighted for its personalized summaries and recommendations, as well as its ability to track research progress over time. However, each tool also has its limitations. For example, Elicit is limited to specific research areas and data types, Semantic Scholar can be overwhelming for beginners, and PaperDigest’s summaries may oversimplify complex research. In conclusion, Elicit, Semantic Scholar, and PaperDigest are each suited to different research needs, with Elicit being the ideal choice for experimental prediction, Semantic Scholar for comprehensive literature search, and PaperDigest for personalized research summaries. Based on these observations, Elicit was selected as the preferred tool for the workflow used in Phase 2, despite Semantic Scholar having the higher score. Table 2 shows the comparison between Elicit Semantic Scholar and PaperDigest. 5.2. Comparison between Litmaps, Open Knowledge maps and Connected Paper Litmaps offers customizable maps and interactive features to visualize the research landscape and emerging topics. It also provides a user-friendly interface for exploring research trends and related papers. The Open knowledge map offers a unique perspective for understanding broader thematic relationships and exploring concepts beyond literature. It is a versatile and open-source tool for exploring connections between diverse topics using knowledge maps. Connected Papers, on the other hand, provides a simple and fast solution for basic citation network analysis and quick discovery of related papers. It is efficient with its citation-based approach to quickly discover related papers based on citations and visualize research landscapes. For comprehensive knowledge mapping, Open Knowledge Maps is an ideal tool, but it requires technical knowledge. Litmaps offers the most intuitive interface for visually exploring research landscapes, while Connected Papers is the simplest and fastest option available for quickly finding related research and citations. Table 3 shows a comparison between Litmaps, Open Knowledge maps and Connected Paper. 5.3. Comparison between Consensus.ai and perplexity Consensus.ai is a good option for collaborative writing and research due to its valuable collaborative tools and citation management features. It is also ideal for analyzing large amounts of text and identifying key points and areas of agreement and disagreement. However, it is limited to analyzing existing text and cannot generate original content. Perplexity, on the other hand, is great for extensive content creation and analysis due to its advanced AI techniques, diverse writing styles, and extensive analysis features. It is also versatile in text analysis and can extract insights from various sources. However, it may lack source validity in generated content (may refer to grey literature) and requires careful fact-checking. Table 5 shows a comparison between Consensus.ai and perplexity. 5.4. User Response on the Proposed Workflow Feedback from both courses provides compelling evidence that AI tools significantly enhance the systematic literature review process. The results show that AI tools like Semantic Scholar, Litmaps, and Scite.ai outperform traditional tools like Google Scholar by offering better organization, validation, and accuracy. These tools also reduce errors and speed up the SLR process, with users reporting greater satisfaction and efficiency. The proposed workflow, which integrates AI tools at every stage of the SLR process, was highly rated by partic- ipants in both studies, showing that AI tools are becoming indispensable in academic research. The combination of improved search functionality, increased accuracy, and ease of use makes AI an essential part of modern SLR processes, allowing researchers to conduct more thorough and reliable reviews in less time. It was also noted that the prominence of Searching for and identifying related and relevant literature (A) across various combinations underscores its fundamental importance to users, likely due to its role in streamlining the often time-consuming process of literature discovery. The frequent pairing of this feature with increased efficiency (B) fur- ther emphasizes users’ focus on productivity enhancements. Interestingly, while ease of use (C) was often selected, it wasn’t universally chosen in combination with other features. This could suggest that some users are willing to trade simplicity for more advanced functionality, especially when it comes to improving search capabilities and efficiency. 5.5. Limitations of the Study 5.5.1. Rubrics-based evaluation model Rubrics are a valuable tool for comparing AI tools, but they do have limitations. One study found that rubrics can be limited by the subjectivity of the evaluator, the complexity of the evaluation process, and the potential for evaluator bias [Tikayat Ray et al., 2023]. Additionally, the use of rubrics may not fully capture the dynamic and evolving nature of AI tools, which can make it challenging to create a comprehensive and accurate evaluation [Pan et al., 2018]. These limitations highlight the need for a balanced approach that combines rubric-based evaluations with other methods to ensure a comprehensive and accurate assessment of AI tools. 5.5.2. The constant evolution and growth of AI-based tools Creating a standard rubric for the comparison of AI tools is challenging due to the diverse nature of AI technologies and the complex evaluation criteria involved. The metrics and benchmarks used to evaluate AI tools are multifaceted and can vary significantly based on the specific application and context [Leung et al., 2023]. The effectiveness, reliability, and robustness of AI tools are typically assessed using metrics such as coverage, similarity, failures, mutation score, error rate, time, and domain expert opinion [Riccio et al., 2020]. However, converting these metrics to evaluate AI tools for SLR is a challenging task. 6. Conclusion The study investigated the feasibility of using a rubric-based evaluation approach to compare AI tools for constructing systematic literature review (SLR) workflows in academic research. A comprehensive rubric was established, encom- passing critical functionalities, performance metrics, and usability factors, to assess the strengths and weaknesses of various AI tools and identify potential candidates for streamlining the SLR process. While the rubric provided valuable insights, limitations such as potential oversimplification, subjectivity, and the dynamic nature of AI technol- ogy were acknowledged. As such, future research should consider complementing rubric-based evaluations with user testing, in-depth case studies, and ongoing monitoring of AI tool development. The variety of feature combinations selected also highlights the diverse needs of users engaged in systematic lit- erature reviews. While some prioritize a full suite of tools, others find specific combinations more suitable for their workflow. This diversity in preferences underscores the importance of offering flexible, modular solutions that can cater to different user needs and expertise levels. Overall, the analysis suggests that tools supporting systematic liter- ature reviews should prioritize robust search capabilities and efficiency improvements, while also offering data-driven features and maintaining user-friendliness to appeal to the broadest range of users. Ultimately, the most effective approach to selecting AI tools for SLR workflow creation may necessitate a combi- nation of quantitative and qualitative methods, tailored to individual researchers’ specific needs and priorities. This research lays a valuable foundation for further exploration of AI tools in the realm of SLR, paving the way for more efficient, accurate, and accessible research methodologies in the future. Declarations All authors agree that the adult participants (no minors involved) consented to participate in the study. This participation was conducted under the previously sought institutional approval, referenced as #IITB-IRB/2023/037. Funding - No funding was received for conducting this study. Conflict of interest/Competing interests - The authors have no conflicts of interest to declare that are relevant to the content of this article. Ethics approval and consent to participate - All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and its later amendments or comparable ethical standards. Clinical trial number: not applicable Data availability - Data can be shared with reviewers, if required. Code availability - Not Applicable Author contribution -Conceptualization: Syaamantak Das, Sarthak Saini; Methodology: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha; Formal analysis and investigation: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha; Writing - original draft preparation: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha, Ganesh Beniwal; Writing - review and editing: Syaamantak Das, Nisha Biju, Alisha Sinha, Sarthak Saini, Ganesh Beniwal; Supervision: Syaamantak Das References Anderson and Jayaratne, 2015] Anderson, N. and Jayaratne, Y. S. N. (2015). Methodological challenges when per- forming a systematic review. European journal of orthodontics , 37 3:248–50. Atkinson, 2023] Atkinson, C. F. (2023). Cheap, quick, and rigorous: artificial intelligence and the systematic litera- ture review. Social Science Computer Review , page 08944393231196281. Blaizot et al., 2022] Blaizot, A., Veettil, S. K., Saidoung, P., Moreno-Garc´ıa, C. F., Wiratunga, N., Aceves-Martins, M., Lai, N. M., and Chaiyakunapruk, N. (2022). Using artificial intelligence methods for systematic review in health sciences: A systematic review. Research Synthesis Methods , 13:353 – 362. Boeker et al., 2013] Boeker, M., Vach, W., and Motschall, E. (2013). Google scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC medical research methodology , 13(1):1– 12. Bramer et al., 2018] Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., and Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association: JMLA , 106(4):531. Chong et al., 2022] Chong, S. W., Lin, T. J., and Chen, Y. (2022). A methodological review of systematic literature reviews in higher education: Heterogeneity and homogeneity. Educational Research Review , 35:100426. Chu and Evans, 2021] Chu, J. S. and Evans, J. A. (2021). Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences , 118(41):e2021636118. Cohen et al., 2010] Cohen, A. M., Ambert, K. H., and McDonagh, M. S. (2010). A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. AMIA ... Annual Symposium proceedings. AMIA Symposium , 2010:121–5. Collins et al., 2021] Collins, C., Dennehy, D., Conboy, K., and Mikalef, P. (2021). Artificial intelligence in informa- tion systems research: A systematic literature review and research agenda. International Journal of Information Management , 60:102383. Dai, 2023] Dai, Y. (2023). Negotiation of epistemological understandings and teaching practices between primary teachers and scientists about artificial intelligence in professional development. Research in Science Education , 53(3):577–591. Das and Islam, 2021] Das, R. K. and Islam, M. S. U. (2021). Application of artificial intelligence and machine learning in libraries: A systematic review. ArXiv , abs/2112.04573. de la Torre-Lo´pez et al., 2023] de la Torre-Lo´pez, J., Ram´ırez, A., and Romero, J. R. (2023). Artificial intelligence to automate the systematic review of scientific literature. Computing , pages 1–24. Elsman et al., 2022] Elsman, E. B., Butcher, N. J., Mokkink, L. B., Terwee, C. B., Tricco, A., Gagnier, J. J., Aiyegbusi, O. L., Barnett, C., Smith, M., Moher, D., et al. (2022). Study protocol for developing, piloting and dis- seminating the prisma-cosmin guideline: a new reporting guideline for systematic reviews of outcome measurement instruments. Systematic reviews , 11(1):121. Eskrootchi et al., 2020] Eskrootchi, R., Shahraki Mohammadi, A., Panahi, S., and Zahedi, R. (2020). Librarians’ participation in the systematic reviews published by iranian researchers and its impact on the quality of reporting search strategy. Evidence Based Library and Information Practice , 15(2):69–84. Feng et al., 2022] Feng, Y., Liang, S., Zhang, Y., Chen, S., Wang, Q., Huang, T., Sun, F., Liu, X., Zhu, H., and Pan, H. (2022). Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis. Journal of the American Medical Informatics Association : JAMIA . Ferreras-Ferna´ndez et al., 2016] Ferreras-Fern´andez, T., Rodero, H. M., Garc´ıa-Pen˜alvo, F. J., and Merlo-Vega, J. A. (2016). The systematic review of literature in lis: an approach. Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality . Garc´ıa-Pen˜alvo, 2022] Garc´ıa-Pen˜alvo, F. J. (2022). Developing robust state-of-the-art reports: Systematic literature reviews. Gonza´lez-Calatayud et al., 2021] Gonz´alez-Calatayud, V., Prendes-Espinosa, P., and Roig-Vila, R. (2021). Artificial intelligence for student assessment: A systematic review. Applied Sciences , 11(12):5467. Haddaway et al., 2020] Haddaway, N. R., Bethel, A., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., Pullin, A. S., Savilaakso, S., and Stewart, G. B. (2020). Eight problems with literature reviews and how to fix them. Nature Ecology & Evolution , 4:1582 – 1589. Haddaway et al., 2018] Haddaway, N. R., Macura, B., Whaley, P., and Pullin, A. S. (2018). Roses reporting standards for systematic evidence syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environmental Evidence , 7:1–8. Joho et al., 2015] Joho, H., Jatowt, A., and Blanco, R. (2015). Temporal information searching behaviour and strategies. Inf. Process. Manag. , 51:834–850. Leung et al., 2023] Leung, T. I., de Azevedo Cardoso, T., Mavragani, A., and Eysenbach, G. (2023). Best practices for using ai tools as an author, peer reviewer, or editor. Journal of Medical Internet Research , 25(1):e51584. Mallett et al., 2012] Mallett, R., Hagen-Zanker, J., Slater, R., and Duvendack, M. (2012). The benefits and challenges of using systematic reviews in international development research. Journal of development effectiveness , 4(3):445– 455. McManus et al., 1998] McManus, R. J., Wilson, S. R., Delaney, B. C., Fitzmaurice, D. A., Hyde, C., Tobias, R. S., Jowett, S., and Hobbs, F. D. R. (1998). Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ , 317:1562 – 1563. Mlinari´c et al., 2017] Mlinari´c, A., Horvat, M., and Sˇupak Smolˇci´c, V. (2017). Dealing with the positive publication bias: Why you should really publish your negative results. Biochemia medica , 27(3):447–452. Mohamed et al., 2023] Mohamed, H., Salsberg, J., and Kelly, D. (2023). An integrative review protocol on interven- tions to improve users’ ability to identify trustworthy online health information. Plos one , 18(4):e0284028. Mouta et al., 2023] Mouta, A., Torrecilla-S´anchez, E. M., and Pinto-Llorente, A. M. (2023). Design of a future scenarios toolkit for an ethical implementation of artificial intelligence in education. Education and Information Technologies , pages 1–26. Myllyaho et al., 2021] Myllyaho, L., Raatikainen, M., Ma¨nnisto¨, T., Mikkonen, T., and Nurminen, J. K. (2021). Systematic literature review of validation methods for ai systems. Journal of Systems and Software , 181:111050. Pan et al., 2018] Pan, M., Linner, T., Pan, W., Cheng, H., and Bock, T. (2018). A framework of indicators for assessing construction automation and robotics in the sustainability context. Journal of Cleaner Production , 182:82– 95. Park et al., 2023] Park, J., Teo, T. W., Teo, A., Chang, J., Huang, J. S., and Koo, S. (2023). Integrating artificial intelligence into science lessons: Teachers’ experiences and views. International Journal of STEM Education , 10(1):61. Pati and Lorusso, 2018] Pati, D. and Lorusso, L. N. (2018). How to write a systematic review of the literature. HERD: Health Environments Research & Design Journal , 11:15 – 30. Payne, 1976] Payne, J. W. (1976). Task complexity and contingent processing in decision making: An information search and protocol analysis. Organizational behavior and human performance , 16(2):366–387. Ramesh and Sanampudi, 2022] Ramesh, D. and Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review , 55(3):2495–2527. Riccio et al., 2020] Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., and Tonella, P. (2020). Testing machine learning based systems: a systematic mapping. Empirical Software Engineering , 25:5193–5254. Rodi et al., 2017] Rodi, G. C., Loreto, V., and Tria, F. (2017). Search strategies of wikipedia readers. PLoS ONE , 12. Sacchi and Burigo, 2008] Sacchi, S. and Burigo, M. (2008). Strategies in the information search process: Interaction among task structure, knowledge, and source. The Journal of General Psychology , 135:252 – 270. Santos et al., 2023] Santos, M. V., Morgado-Dias, F., and Silva, T. C. (2023). Oil sector and sentiment analysis—a review. Energies , 16(12):4824. Schiff and Mo¨ller, 2021] Schiff, S. and Mo¨ller, R. (2021). On human-aware information seeking. In CHAI@KI . [Sesagiri Raamkumar et al., 2018] Sesagiri Raamkumar, A., Foo, S., and Pang, N. (2018). Can i have more of these please? assisting researchers in finding similar research papers from a seed basket of papers. The Electronic Library , 36(3):568–587. Sharadgah and Sa’di, 2022] Sharadgah, T. A. and Sa’di, R. A. (2022). A systematic review of research on the use of artificial intelligence in english language teaching and learning (2015-2021): What are the current effects? Journal of Information Technology Education: Research . Shi et al., 2022] Shi, D., Zhou, J., Wang, D., and Wu, X. (2022). Research status, hotspots, and evolutionary trends of intelligent education from the perspective of knowledge graph. Sustainability , 14(17):10934. Tikayat Ray et al., 2023] Tikayat Ray, A., Cole, B. F., Pinon Fischer, O. J., Bhat, A. P., White, R. T., and Mavris, D. N. (2023). Agile methodology for the standardization of engineering requirements using large language models. Systems , 11(7):352. van Dinter et al., 2021] van Dinter, R., Tekinerdogan, B., and Catal, C. (2021). Automation of systematic literature reviews: A systematic literature review. Information and Software Technology , 136:106589. Williamson, 2023] Williamson, B. (2023). The social life of ai in education. International Journal of Artificial Intelligence in Education , pages 1–8. Tables Tables 2 to 5 are available in the Supplementary Files section. Footnotes 1 https://typeset.io/resources/ai-tools-for-systematic-literature-review/ 2 https://blogs.lse.ac.uk/impactofsocialsciences/2020/10/19/8-common-problems-with-literature-reviews-and-how-to-fix-them/ 3 https://paperpile.com/g/academic-search-engines/ Additional Declarations No competing interests reported. Supplementary Files Table2to5.docx Cite Share Download PDF Status: Published Journal Publication published 29 Dec, 2025 Read the published version in Discover Artificial Intelligence → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6328602","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Case Report","associatedPublications":[],"authors":[{"id":458684123,"identity":"581cccec-f4a1-4092-a6d3-f43ce79d1d89","order_by":0,"name":"Nisha Biju","email":"","orcid":"","institution":"Centre for Educational Technology, Indian Institute of Technology Bombay","correspondingAuthor":false,"prefix":"","firstName":"Nisha","middleName":"","lastName":"Biju","suffix":""},{"id":458684124,"identity":"19640d9a-3fe2-4e95-8a93-e1c4a9fa94f9","order_by":1,"name":"Alisha Sinha","email":"","orcid":"","institution":"Centre for Educational Technology, Indian Institute of Technology Bombay","correspondingAuthor":false,"prefix":"","firstName":"Alisha","middleName":"","lastName":"Sinha","suffix":""},{"id":458684125,"identity":"26f06921-5680-4bd7-a13d-426462b00390","order_by":2,"name":"Sarthak Saini","email":"","orcid":"","institution":"Indian Institute of Science Education and Research Pune","correspondingAuthor":false,"prefix":"","firstName":"Sarthak","middleName":"","lastName":"Saini","suffix":""},{"id":458684126,"identity":"da0f4399-5f53-4ada-81d6-507411990660","order_by":3,"name":"Ganesh Beniwal","email":"","orcid":"","institution":"Centre for Educational Technology, Indian Institute of Technology Bombay","correspondingAuthor":false,"prefix":"","firstName":"Ganesh","middleName":"","lastName":"Beniwal","suffix":""},{"id":458684127,"identity":"53db1474-9f7e-4394-87f3-b6cc87cc3873","order_by":4,"name":"Syaamantak Das","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYFADHgYGCQYGGx42EAvCxw8koFrSeNjYSNRymIGBjYBi+RnJDz/dqLlTx99zgPHGjz/nZfjkew8w/KhhkDHHocXgRpqxdM6xZxISZxuYLXvbbgMdxpfA2HOMgceyAYcWiQQD6Ry2wxIM5xnYJHgbQFp4DBh4Gxh4DA7gclj65985/w5LyAO1SP75cw6shfEvHi0MN3LMpHPbDksYnG1gk+ZhOwDWwozPFoMzb8qsc/sOS248c7DZWrYtGaglL+GwzDEJ3A5rT998O+fbYX65M8kHb775Y2cv33z24MM3NTb2OB0mkABjMTbABQ+A4wgX4Mdl1igYBaNgFIwCGAAADYpTNZKCRLQAAAAASUVORK5CYII=","orcid":"","institution":"Centre for Educational Technology, Indian Institute of Technology Bombay","correspondingAuthor":true,"prefix":"","firstName":"Syaamantak","middleName":"","lastName":"Das","suffix":""}],"badges":[],"createdAt":"2025-03-28 13:38:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6328602/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6328602/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s44163-025-00628-8","type":"published","date":"2025-12-29T15:58:12+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":83542714,"identity":"33340ce6-46a8-4657-96e6-07d859b70395","added_by":"auto","created_at":"2025-05-28 08:27:32","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":31227,"visible":true,"origin":"","legend":"\u003cp\u003ePercentage vs Combination\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6328602/v1/c09f1a42113db9f078e8dbfa.png"},{"id":99545454,"identity":"0fad46f3-9ede-4586-92f6-34db43106403","added_by":"auto","created_at":"2026-01-05 16:07:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1233436,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6328602/v1/5d05af74-164a-4afa-ab5f-129cbe8dc062.pdf"},{"id":83542715,"identity":"f1f816f1-7a25-4636-9a8e-81c63dacbbe3","added_by":"auto","created_at":"2025-05-28 08:27:32","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":37347,"visible":true,"origin":"","legend":"","description":"","filename":"Table2to5.docx","url":"https://assets-eu.researchsquare.com/files/rs-6328602/v1/0d23c887a22102bf46efa161.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Designing an AI-Driven SLR Workflow for Academic Research: A Rubric for Comparative Analysis of AI Tools","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eA Systematic Literature Review (SLR) is a rigorous and transparent method for identifying, evaluating, and inter- preting existing research in a specific field [Garc\u0026acute;ıa-Pen˜alvo, 2022]. It aims to fill knowledge gaps and guide future research by integrating and summarizing empirical studies on a particular topic [Ferreras-Ferna\u0026acute;ndez et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2016\u003c/span\u003e]. This type of review is particularly useful in academic research, where it can provide a comprehensive understanding of the existing evidence [Pati and Lorusso, 2018]. The process involves several key phases, including planning, conducting, and reporting the review[Garc\u0026acute;ıa-Pen˜alvo, 2022]. To ensure methodological rigour and quality, researchers adhere to standardized methodologies and guidelines, such as the PRISMA statement [Pati and Lorusso, 2018].\u003c/p\u003e \u003cp\u003eConducting an SLR is a challenging task because it requires meticulous attention to detail, critical thinking, and a deep understanding of the research topic. The work by [Anderson and Jayaratne, 2015] highlights the need for a clear research question, comprehensive literature search, and rigorous quality assessment to complete SLR. Early work by [McManus et al., 1998] emphasizes the difficulty in identifying relevant studies, particularly in new or interdisciplinary fields, and suggests the use of expert contacts. A recent work by [Haddaway et al., 2020] discusses the potential for bias in traditional literature reviews and the importance of rigorous methods in systematic reviews.\u003c/p\u003e \u003cp\u003eDespite its importance in providing a comprehensive overview of the existing knowledge in a particular field, current SLR methodologies often fall short in several aspects. Firstly, the sheer volume of published research pa- pers poses a significant challenge, with the number of scholarly articles published annually increasing every year [Chu and Evans, 2021]. This surge in publications makes it increasingly difficult to identify, evaluate, and synthesize all relevant studies. Traditional manual searches often rely on keywords, which can lead to missed references or biased results. Furthermore, publication bias and selective reporting can result in an incomplete picture of the available evidence, with studies with positive or significant results more likely to be published than those with negative or inconclusive findings [Mlinari\u0026acute;c et al., 2017]. To overcome these limitations, systematic literature reviews must employ innovative strategies, such as machine learning algorithms and automated screening tools, to efficiently and accurately identify relevant studies [van Dinter et al., 2021].\u003c/p\u003e\n\u003ch3\u003e1.1 Application of AI for SLR\u003c/h3\u003e\n\u003cp\u003eArtificial Intelligence (AI) can significantly improve the SLR process for academic research by automating various tasks, making the process more efficient, accurate, and comprehensive. A range of studies have explored the potential of AI-based tools to automate SLR. AI provides methods to represent and infer knowledge, efficiently manipulate texts, and learn from vast amounts of data, which apply to the analysis of scientific literature [de la Torre-Lo\u0026acute;pez et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2023\u003c/span\u003e]. By using AI software designed to automate systematic review, researchers can expedite the search process, making it more practical, sustainable, and cost-effective \u003csup\u003e1\u003c/sup\u003e. AI tools can constantly search databases for the latest research, support evidence synthesis, and ensure that relevant papers are incorporated into the review to keep it up-to- date[Atkinson, 2023]. AI algorithms also ensure an improved way of error-free data extraction and statistical analysis compared to humans, leading to increased accuracy[Collins et al., 2021]. Recent work by [Blaizot et al., 2022], en- dorsing a previous work of [Cohen et al., 2010], found that AI methods can enhance efficiency and quality in evidence synthesis, with the latter demonstrating high accuracy in literature classification. However, human validation is still crucial in implementing these tools. The work by [Feng et al., 2022] emphasized the importance of recall in AI models for literature screening, while the work by [Das and Islam, 2021] highlighted the wide range of AI and machine learn- ing techniques being used in libraries, including for collection management and user interaction. When information search principles are applied appropriately, SLR have a clear advantage over traditional literature reviews, and the advent of AI-powered tools further enhances this advantage.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e1.1.1 Technological Challenges\u003c/h2\u003e \u003cp\u003eThe use of AI tools for SLR in academic research presents several technological challenges. The work of [Blaizot et al., 2022] also highlights the need for extensive human validation in the implementation of AI methods, indicating that these tools are not yet fully autonomous. The work of [Sharadgah and Sa\u0026rsquo;di, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2022\u003c/span\u003e] emphasize the potential of AI in en- hancing efficiency and quality in evidence synthesis, but also underscores the importance of detailed methodological descriptions and the need for further exploration of the challenges and risks associated with AI. Further analysis is essential to comprehend the pedagogical, ethical, and cultural aspects of AI-enhanced education comprehensively.\u003c/p\u003e \u003cp\u003eThese studies collectively suggest that while AI tools hold promise for improving the SLR process, their full potential can only be realized through continued research and development.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e1.1.2 Pedagogical Challenges\u003c/h3\u003e\n\u003cp\u003eThe pedagogical challenges of using AI for SLR in academic research include addressing teacher capability issues, as teachers need to be qualified to teach AI tools competently and effectively. Misconceptions about AI among teachers are common, and there is a need for external content experts to prepare teachers\u0026rsquo; content delivery knowledge [Dai, 2023]. AI in education is not just about technology but involves social and historical factors, including technical developments, scientific practices, institutional applications, and power struggles[Williamson, 2023]. The integration of AI into classrooms is expected to continue serving as a space for the democratic formation of public thought and concepts related to various aspects of life [Mouta et al., 2023]. Moreover, the adoption of AI in education faces challenges such as the selection of suitable learning programs, evaluation of student learning outcomes, and fostering teachers\u0026rsquo; professionalism and self-efficacy for AI-integrated lessons. Teachers need to have sufficient knowledge related to AI tools and technologies to understand and effectively use the educational roles of AI [Park et al., 2023].\u003c/p\u003e \u003cp\u003eAdditionally, the use of AI in education has been influenced by various national strategies and policies, with countries like the United States, the United Kingdom, Germany, Japan, and China releasing policies related to intelligent education. These policies emphasize the integration of AI into the teaching process and the promotion of AI applications throughout the educational system [Shi et al., 2022]. The rapid development of AI calls for innovation, and various countries have recognized the importance of AI education in cultivating new talents and have begun to incorporate AI into educational policies and teaching models.\u003c/p\u003e"},{"header":"2. Literature Review","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Information Search: An overview of Protocol and strategies\u003c/h2\u003e \u003cp\u003eThe information search protocols and strategies adopted by humans can vary depending on the context, such as academic research, decision-making, or online information seeking. In academic research, a systematic approach to searching is essential, involving the development of librarian-mediated searches for systematic reviews and medical literature. This includes steps such as formulating a question, creating thorough search strategies in multiple databases, and having the search terms checked by relevant experts [Bramer et al., 2018]. In decision-making research, two process tracing techniques \u0026ndash; explicit information search, and verbal protocols are used to understand the information processing strategies that individuals use in reaching a decision [Payne, 1976]. When it comes to online information seeking, humans use various strategies, such as central or peripheral processing, based on information processing theories. For example, the limited capacity model of message processing suggests that people usually process part of the message provided due to limited capacity. Additionally, the information-foraging theory posits that humans choose behaviours that tend to optimize the utility of information gained [Mohamed et al., 2023].\u003c/p\u003e \u003cp\u003eResearch on information search protocols and strategies has revealed several key findings. The work by [Rodi et al., 2017] identified a pattern in Wikipedia users\u0026rsquo; navigation, where they start with broad topics and progressively narrow down their search. This suggests a strategy of moving from the general to the specific. The work of [Sacchi and Burigo, 2008] found that individuals tend to use a sequential strategy when the information source is perceived as reliable, regardless of their knowledge level. This indicates a reliance on the credibility of the source. The study by [Joho et al., 2015] highlighted the use of temporal expressions in search queries, with mixed success, and the difficulty in finding future information. A recent work [Schiff and Mo\u0026uml;ller, 2021] discussed the optimization of user interaction with information retrieval systems, emphasizing the need for human-aware collaborative planning strategies. These studies collectively suggest that humans employ a range of strategies in information search, including a progression from general to specific, reliance on source credibility, and the use of temporal expressions.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e2.2 Challenges with traditional SLR process\u003c/h3\u003e\n\u003cp\u003eTraditional literature review processes have several challenges that can lead to biased or incorrect conclusions. These challenges include a lack of appropriate critical appraisal of study validity, treating all evidence as equally valid, and limited stakeholder engagement leading to a review that is of limited relevance \u003csup\u003e2\u003c/sup\u003e[Haddaway et al., 2020]. Challenges associated with information search in SLR for academic research include the difficulty in defining precise research questions, selecting relevant databases, and establishing appropriate search terms [Chong et al., 2022]. The process often requires a pilot study to refine search strategies, and the use of Boolean operators to combine search terms can be complex. Additionally, the inclusion and exclusion criteria must be clearly defined to ensure relevant literature is captured. The screening process is usually conducted in two levels: initial screening of titles and abstracts, followed by full-text screening. Moreover, the vast number of records retrieved from databases like Google Scholar can make the search inefficient. The involvement of librarians in the systematic review process can impact the quality of the search strategy, with activities ranging from consulting on selecting resources to designing complete search strategies [Eskrootchi et al., 2020]. SLR has a clear advantage over traditional literature reviews when systematic review princi- ples are applied sensitively[Mallett et al., 2012]. Systematic reviews rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing bias. However, despite the growing interest in systematic reviews, traditional approaches to reviewing the literature continue to persist in contemporary publications.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.2.1 Limitations of PRISMA\u003c/h2\u003e \u003cp\u003eThe limitations of SLR models like PRISMA include publication bias, as PRISMA\u0026rsquo;s efforts may not completely eliminate it, and its primary design for quantitative data, making it less suitable for qualitative research or other evidence synthesis approaches [Santos et al., 2023]. PRISMA is also criticized for not being designed for reviews that involve narrative, qualitative, or mixed methods rather than quantitative methods, and for its heavy emphasis on meta- analysis, which excludes other synthesis methods [Haddaway et al., 2018]. Additionally, PRISMA may not adequately handle novel review outputs like systematic maps, and it may not be easily adapted for methods that rely more on the earlier stages of the review process (searching and screening). Furthermore, PRISMA\u0026rsquo;s suggested requirements for review conduct are minimal, affecting the overall comprehensiveness of the review [Haddaway et al., 2018]. There is also a need for guidance regarding the reporting of systematic reviews of outcome measurement instruments (OMIs), as some components of PRISMA items are of limited relevance to such reviews [Elsman et al., 2022].\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e2.3 Research Gaps\u003c/h3\u003e\n\u003cp\u003eTo overcome the challenges of conducting an SLR, a search protocol and strategy (an ad-hoc tech stack-based workflow) that utilizes various artificial intelligence-based tools are required. The following research questions were explored in this study:\u003c/p\u003e \u003col\u003e\n \u003cli\u003eWhat is the proposed model of a search protocol and stack of tools for conducting SLR using AI-based tools?\u003c/li\u003e\n \u003cli\u003eHow can artificial intelligence-based tools be used (like a tech stack) to address the challenges of conducting literature reviews?\u003c/li\u003e\n \u003cli\u003eHow can a rubric be used to evaluate and identify the best set of tools for conducting SLR using AI-based tools?\u003c/li\u003e\n\u003c/ol\u003e\u003cp\u003eThis study evaluated a proposed model of a search protocol and stack of such tools using a rubric to identify the best set of tools for the task. It involved Sixty-three students who used the proposed model to conduct their academic research.\u003c/p\u003e"},{"header":"3. Study Design","content":"\u003cp\u003eThere are several studies available on the use of AI tools for SLR in academic research. One such study [de la Torre-Lo\u0026acute;pez et al., 2023] provides a survey of AI techniques proposed in the last 15 years to help researchers conduct systematic analyses of scientific literature. The study analyzes the AI techniques currently available, with special emphasis on their purpose, inputs and outputs, and human intervention, if any. Another study [Myllyaho et al., 2021] focuses on the validation methods used for practical AI systems reported in the literature. Additionally, a systematic review [Gonza\u0026acute;lez-Calatayud et al., 2021] analyzes the use of AI for student assessment, while another systematic literature review [Ramesh and Sanampudi, 2022] provides an overview of automated essay scoring systems.\u0026nbsp;However, there is\u0026nbsp;no specific rubric available to evaluate AI tools used for systematic literature review in academic research.\u003c/p\u003e\n\u003cp\u003eThe study works in two phases:\u003c/p\u003e\n\u003cp\u003ePhase 1: A pilot study analysing the experiences of 18 participants doing SLR using the four steps mentioned below. Phase 2: A detailed study involving 45 students. It had an additional final tool \u003cem\u003e(Jenni.ai)\u0026nbsp;\u003c/em\u003ewhich was recommended for report writing.\u003c/p\u003e\n\u003cp\u003eAn SLR has several process steps which are defined differently in the literature. For this study, the common steps involved in performing an SLR for academic research were identified which are as follows:\u003c/p\u003e\n\u003cp\u003e1. \u003cstrong\u003eStep 1: Formulating tentative research questions\u003c/strong\u003e: Getting started with a basic research question in the beginning.\u003c/p\u003e\n\u003cp\u003e2. Step 2: Identifying a base(seed) paper:\u003c/p\u003e\n\u003cp\u003eTo conduct a literature review, one needs to find an initial base(seed) paper [Sesagiri Raamkumar et al., 2018], build a reading list from it, understand how the paper builds upon previous research, and identify follow-up papers.\u003c/p\u003e\n\u003cp\u003e3. Step 3: Assessing the quality of the literature:\u003c/p\u003e\n\u003cp\u003eIt is common for the method used in a research study to be slightly complex to comprehend in the original publication. Therefore, researchers must understand the study\u0026rsquo;s goals and areas of investigation by examining how other scholars have cited it in their work. This enables the assessment of the literature\u0026rsquo;s quality.\u003c/p\u003e\n\u003cp\u003e4. \u003cstrong\u003eStep 4: Analyzing and synthesizing additional evidence from extracting data\u003c/strong\u003e: Systematically collect data from the included studies based on paper content.\u003c/p\u003e\n\u003cp\u003eNote: This research aims to identify the right set of information using AI-based tools for SLR processes. Writing and other additional steps are beyond the scope of this research. However, the scope of this research is limited to identifying the right set of information using AI-based tools.\u003c/p\u003e\n\u003cp\u003e3.1. \u0026nbsp; Developing the Rubric\u003c/p\u003e\n\u003cp\u003eThe evaluation of each tool was done against a set of three different possible scores (Score 3, Score 2, Score 1) for a given evaluation criterion. The criterion was further divided based on certain parameters which are mentioned for each step:\u003c/p\u003e\n\u003cp\u003e3.1.1. \u0026nbsp; \u0026nbsp; Rubric for step 1:\u003c/p\u003e\n\u003cp\u003eFor step 1,\u0026nbsp;\u003cem\u003eA)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eexpediting\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003esearch\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eprocess,\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eB)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003efeatures\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003etools,\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eC)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003enavigation\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003estructure\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eD\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003e)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003especific\u0026nbsp;\u003c/em\u003e\u003cem\u003epurpose\u003c/em\u003e\u003cem\u003e\u0026nbsp;of the tool\u0026nbsp;\u003c/em\u003ewere chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows:\u003c/p\u003e\n\u003cp\u003e1. Criteria (A): Expedite the Search process\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria (i): Use of keywords\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; SLR tools were evaluated based on their ability to expedite the search process and use of keywords. Each tool was allocated a score based on performance.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool identifies and summarizes the literature based on a research question, \u003cem\u003eeven if the papers do not match the keywords\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool identifies and summarizes the literature based on a research question, \u003cem\u003eonly if the papers match the exact keywords\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool \u003cem\u003edoes not summarize\u0026nbsp;\u003c/em\u003ethe literature based on a research question on the basis of keywords.\u003c/p\u003e\n\u003cp\u003e2. Criteria (B): Features of the tools\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(i):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eSummary\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eof\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eAbstract\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; The tools were evaluated based on their ability to generate summaries of relevant papers.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool allows the user to get a summary of the \u003cem\u003etop N papers\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003ealong with the author\u0026rsquo;s name, year of publication and citations of papers.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool allows the user to get a summary of the \u003cem\u003elimited\u003c/em\u003e\u003cem\u003e\u0026nbsp;papers\u0026nbsp;\u003c/em\u003ealong with limited information about the author\u0026rsquo;s name, year of publication and citations of papers.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool allows the user to get a summary of the \u003cem\u003elimited\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003einformation\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eabout\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eauthor\u0026rsquo;s\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ename,\u0026nbsp;\u003c/em\u003e\u003cem\u003eyear\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003epublication\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ecitations\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003epapers\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(ii):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eDiversity\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eof\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eChoices\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; The tools cater to the unique needs and preferences of a diverse range of users.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool is designed to address the diverse needs of users. E.g. searching papers based on study type (e.g. RCT (randomized controlled trial)) AND time frame (within the last N years).\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool is designed to address the limited needs of users. E.g. searching papers based on study type (e.g. RCT) OR time frame (within the last N years).\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool is designed to address the basic needs of users. E.g. searching papers based ONLY ON the time frame (within the last N years).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(iii):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eAccessibility\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; Cost incurred to use the tool\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Freemium model.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Free basic model.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Subscription required.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(iv):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eLibrary\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eoptions\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; Option to curate specific group of papers.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool \u003cem\u003ecan\u0026nbsp;categorize\u0026nbsp;and\u0026nbsp;save papers\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool can\u0026rsquo;t categorize but \u003cem\u003esave\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003epapers\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool can\u0026rsquo;t categorize or save papers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(v):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eShare/Export\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eoptions\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; to save a paper for future usage.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: \u003cem\u003eBoth\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eimport\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eexport\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eavailable\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Export available but can\u0026rsquo;t import or vice-versa.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: No import/export available.\u003c/p\u003e\n\u003cp\u003e3. Criteria (C): Navigation and structure\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eSub-criteria\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e(i):\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eClear\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eand\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eOrganized\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003eMenu\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u0026ndash; Are the tools having clear instructions for navigation?\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Links for navigation are clearly labelled and placed consistently.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Links for navigation are not clearly labelled and placed.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Links for navigation are not available for most features.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Ease of Navigation\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Enables smooth navigation with minimal chances of getting lost.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Links are missing, causing users to get lost.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Navigation options are limited, causing users to get lost when leaving the page due to missing links.\u003c/p\u003e\n\u003cp\u003e4. Criteria (D): Specific purpose of the tool\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Finding academic research papers\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The user can effortlessly locate research papers relevant to their topic, with \u003cem\u003ea\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ehigh\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003edegree\u003c/em\u003e\u003cem\u003e\u0026nbsp;of precision\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The users can generally find relevant research papers, but the process might require \u003cem\u003esome manual refinement\u0026nbsp;and\u0026nbsp;additional\u0026nbsp;time\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Users struggle to find relevant research papers, and the search results often yield \u003cem\u003eoutdated\u003c/em\u003e\u003cem\u003e\u0026nbsp;or irrelevant content\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Discovery of concepts\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Discovering concepts across papers is \u003cem\u003emoderately\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eeffective\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003ebut may have some limitations. The tool generally identifies relevant concepts but may occasionally miss subtler or less common ones.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Discovering concepts across papers is \u003cem\u003elargely\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eineffective\u003c/em\u003e. Users struggle to identify relevant concepts, and the tool\u0026rsquo;s results often miss critical themes or connections.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: \u003cem\u003eNo\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003efeature or functionality\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003efor discovering concepts across papers within the system. Users are left to rely entirely on manual methods and external resources for identifying and connecting concepts in the literature.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iii): Meta-Information extraction from paper\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The utility of the tool for extracting information from papers is \u003cem\u003ehighly\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eeffective\u003c/em\u003e. It consistently and accurately extracts information from a 10 to 25 number of papers with ease.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The utility of the tool for extracting information from papers is \u003cem\u003emoderately\u003c/em\u003e\u003cem\u003e\u0026nbsp;effective\u0026nbsp;\u003c/em\u003ebut may have some limitations.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The utility of the tool for extracting information from papers is \u003cem\u003emostly\u003c/em\u003e\u003cem\u003e\u0026nbsp;ineffective\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e3.1.2. \u0026nbsp; \u0026nbsp;Rubric for step 2:\u003c/p\u003e\n\u003cp\u003eFor step 2, \u003cem\u003eA)\u003c/em\u003e\u003cem\u003e\u0026nbsp;Identifying base paper (seed paper) and B)\u0026nbsp;\u003c/em\u003e\u003cem\u003eGeneration\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;knowledge\u0026nbsp;\u003c/em\u003e\u003cem\u003egraph\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003ewere considered as criteria: The sub-criteria for each of these are as follows: A) Seed paper search, B) Visualization of connected(related) papers,\u003c/p\u003e\n\u003cp\u003eC) Search Sharing and D) Specific purpose of the tool were chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows:\u003c/p\u003e\n\u003cp\u003e1. Criteria (A): Identifying base paper (Seed paper)\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Seed paper search\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool searches the list of papers that are \u003cem\u003emost\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003erelevant to\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003etopic\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003efrom the keywords given.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool searches the list of \u003cem\u003emoderately relevant papers\u003c/em\u003e, prior and derivative works based on the given paper as a query.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool searches the list of \u003cem\u003enot-so-relevant\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003epapers\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003ebased on the citation of the paper, provided as a query.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Generation and Visualization of knowledge graph\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool presents the papers that are related in a legible, interactive, and dynamic visualization.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool provides a visualization that is legible, and dynamic but not interactive.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool provides a legible visualization of the papers only.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iii): Search Sharing\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool allows the sharing of the result, as visualizations, list of papers, and to reference manager.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool allows the sharing of the results as visualizations and a list of papers only.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool allows the sharing of the result in the form of a list of papers only.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iv): Data export\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Allows users to export data and reports in a variety of formats, such as BIB, BibTeX, CSV, RIS and JSON.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Allows users to export data in CSV or PDF but exporting search results from a database into a citation management tool using RSV (Rich Structured View) is not available.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Partially supports data/report export, such as by allowing users to export data only in one format.\u003c/p\u003e\n\u003cp\u003e2. Criteria (B): Creating and revising Knowledge graph\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): No. of connected papers generated\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Greater than 30.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Between 20 and 30,\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Between 10 and 20.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Creating knowledge graph\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool offers an intuitive user interface, supports multiple data import formats, and automated entity recognition features.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool provides a user-friendly interface and supports at least one data import format.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool allows for rudimentary graph creation with limited data import options.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iii): Revising knowledge graph\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool offers real-time, visual, and bulk editing features, along with advanced validation checks and role-based access control.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool provides basic editing features and includes either validation checks or role-based access control.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool allows for limited editing capabilities but lacks comprehensive features for validation and access control.\u003c/p\u003e\n\u003cp\u003e3.1.3. \u0026nbsp; \u0026nbsp;Rubric for step 3:\u003c/p\u003e\n\u003cp\u003eFor step 3, \u003cem\u003eA)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eAdvanced\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003efeatures,\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eB)Visual\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003einterface,\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eC)\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003especific\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003etasks\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003etool\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003ewere chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows:\u003c/p\u003e\n\u003cp\u003e1. Criteria (A): Advanced features\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Boolean Search\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool allows users to get \u003cem\u003efull accessibility\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003elike boolean search using (\u0026ldquo;AND\u0026rdquo;, \u0026ldquo;OR\u0026rdquo;, \u0026ldquo;NOT\u0026rdquo;), Brackets, Wildcard searches, Fuzzy matching and Proximity matching.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool allows users to get \u003cem\u003elimited\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eaccessibility\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003elike boolean search using (\u0026ldquo;AND\u0026rdquo;, \u0026ldquo;OR\u0026rdquo;, \u0026ldquo;NOT\u0026rdquo;), Brackets, Wildcard searches, Fuzzy matching, and Proximity matching.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool fails users to provide accessibility like boolean search using (\u0026ldquo;AND\u0026rdquo;, \u0026ldquo;OR\u0026rdquo;, \u0026ldquo;NOT\u0026rdquo;), Brackets, Wildcard searches, Fuzzy matching, and Proximity matching.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Reference Check\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool displays a list of reliable references that have been cited and have no significant disputes or retractions.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool displays a list of references with partial reliability, which may have value but should be used with caution. Some references cited may have disputes or disagreements.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool does not verify the reliability and citation of referenced lists.\u003c/p\u003e\n\u003cp\u003e2. Criteria (B): Specific task of the tool\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Visual interface\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Enables smooth navigation with minimal chances of getting lost.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Links/Icons are missing, causing users to get lost.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Navigation options are limited, causing users to get lost when leaving the page due to missing links.\u003c/p\u003e\n\u003cp\u003eCriteria (B): Specific task of the tool\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Critical Analysis\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool allows users to critically engage with publications, understand how a publication and its results have been cited, and find relevant literature on the topic in question.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool allows users to access limited engagements with publications, understand how a publi- cation and its results have been cited, and find relevant literature on the topic in question.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool allows users to access limited publications but does not allow them to understand how a publication and its results have been cited, or to find relevant literature on the topic in question.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Identifying research gaps\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool was highly effective for identifying and explaining the gaps in research by analyzing how research papers have been cited and the context in which these citations occur.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool was partially effective for identifying and explaining the gaps in research by analyzing how research papers have been cited and the context in which there are few or no citations.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool fails to identify and explain the gaps in research by analyzing how research papers have been cited and the context in which there are no citations.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iii): Custom dashboard\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool allows to import of files from Zotero or Mendeley library and the upload of CSV files.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool does not allow importing files from Zotero or Mendeley library but uploads file formats like PDF or CSV.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool does not allow to import library or PDF but can upload CSV files.\u003c/p\u003e\n\u003cp\u003e3.1.4. \u0026nbsp; \u0026nbsp;Rubric for step 4:\u003c/p\u003e\n\u003cp\u003eFor step 4, \u003cem\u003eA) Advanced features, and B) Question Answering interface\u0026nbsp;\u003c/em\u003ewere chosen as primary criteria for evaluation. Each of these criteria was further subdivided into multiple sub-criteria based on some parameters which are as follows:\u003c/p\u003e\n\u003cp\u003e1. Criteria (A): Advanced features\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Quality of Reference Sources\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool produces credible references from academic articles that are published in indexed journals.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool produces credible references from web resources only.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool does not produce any credible reference for the answer.\u003c/p\u003e\n\u003cp\u003eSub-criteria (ii): Synthesis of Answers\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: The tool provides options to synthesize results to the questions, through options such as summary, popular opinion and number of positive vs. negative reviews.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: The tool provides some form of options to synthesize results from the answers.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: The tool provides no options to synthesize the answers.\u003c/p\u003e\n\u003cp\u003eSub-criteria (iii): Sharing of Consolidated Output\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: Sharing of full results of the query as a text document.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Sharing of partial results of the query.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Does not allow sharing of results, the user has to manually copy-paste the results.\u003c/p\u003e\n\u003cp\u003e2. Criteria (B): Usability feature\u003c/p\u003e\n\u003cp\u003eSub-criteria (i): Question Answering interface\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 3: A clear option for asking questions is indicated on the tool and the user is presented with the follow-up information.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 2: Options for asking questions are laid out on the application, however, the flow is not streamlined.\u003c/p\u003e\n\u003cp\u003e\u0026bull; Score 1: Do not have such an option.\u003c/p\u003e\n\u003cp\u003e3.2. \u0026nbsp; Benchmark Identification\u003c/p\u003e\n\u003cp\u003eIn 2023, a survey was conducted by paperpile.com\u003csup\u003e[3]\u003c/sup\u003e on systematic literature search. The survey found that Google Scholar is the most extensively used tool for this purpose. Despite its popularity, a 2013 study [Boeker et al., 2013] suggested that Google Scholar is not the most optimal tool for systematic literature review. Nevertheless, due to its widespread usage and comprehensive coverage, Google Scholar can serve as a reference point for comparing AI-powered tools for literature searches. Its advantages include advanced search capabilities, continuous updates, user familiarity, accessibility, integration, and industry-standard status. However, Google Scholar also has certain limitations, such as limited search filters, varied quality of sources, language restrictions, citation bias, lack of transparency in its algorithms, redundancy in search results, metadata inconsistencies, and limited disciplinary filters. Additionally, it lacks some critical features for systematic literature retrieval, like tools for incremental query optimization, export of a large number of references, a visual search builder, or a history function.\u003c/p\u003e\n\u003cp\u003e3.3. \u0026nbsp; Dataset\u003c/p\u003e\n\u003cp\u003ePhase 1:\u003c/p\u003e\n\u003cp\u003e1. \u003cstrong\u003eNo.\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eof\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eAI\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003etools:\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eInitially, 15 AI tools were analyzed for this research. Based on the final ranking top 3 tools for each level are shown in Table 4.\u003c/p\u003e\n\u003cp\u003e2. \u003cstrong\u003ePeriod:\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eJune 2023 - December 2023. It should be noted that these tools are constantly being upgraded and new features are added. This ranking is based on the tool\u0026rsquo;s features during a specific period.\u003c/p\u003e\n\u003cp\u003e3. \u003cstrong\u003eParticipants:\u0026nbsp;\u003c/strong\u003e18 researchers doing active research in various domains of postgraduate qualification took part in this study as a part of a University course.\u003c/p\u003e\n\u003cp\u003ePhase 2:\u003c/p\u003e\n\u003cp\u003e1. \u003cstrong\u003eNo.\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eof\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eAI\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003etools:\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e6 AI tools formulating a workflow were recommended to students for this phase.\u003c/p\u003e\n\u003cp\u003e2. \u003cstrong\u003ePeriod:\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eJune - July 2024\u003c/p\u003e\n\u003cp\u003e3. \u003cstrong\u003eParticipants:\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e45 students\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003einterested in learning about AI tools\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003efor SLR using a certificate\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003ecourse.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e3.4. \u0026nbsp; Identification of Appropriate AI tools for SLR\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s a step-by-step process used to analyze a set of AI tools for comparison, using the given rubric for the study:\u003c/p\u003e\n\u003cp\u003e1. Gathering the AI tools - Identifying the specific AI tools from an initial superset of AI tools.\u003c/p\u003e\n\u003cp\u003e2. Finetuning the rubric - Creating own customized rubric based on specific needs for comparing the tools. Ensuring that the rubric covers relevant aspects like features, performance, usability, and cost.\u003c/p\u003e\n\u003cp\u003e3. Define scoring criteria - Each rubric item had clear criteria for scoring.\u003c/p\u003e\n\u003cp\u003e4. Gathering evidences - Collecting relevant data to support the evaluation process. This includes screenshots, official documentation (including whitepapers), publicly available user reviews, etc.\u003c/p\u003e\n\u003cp\u003e5. Prioritize key findings: Looking for patterns and trends across the data to identify the most important factors for decision-making.\u003c/p\u003e\n\u003cp\u003e6. Make recommendations: Recommend the most suitable tool(s) based on the specific needs and priorities. Clearly articulate the rationale behind the recommendations.\u003c/p\u003e\n\u003cp\u003e7. Considering limitations: Acknowledging any limitations of the analysis, such as the subjectivity of the rubric or incomplete data.\u003c/p\u003e\n\u003cp\u003eThe Identified tools were arranged in a workflow and recommended to the students in Phase 2.\u003c/p\u003e"},{"header":"4. Observation and Analysis","content":"\u003ch2\u003e4.1. \u0026nbsp; Observations from Phase 1\u003c/h2\u003e\n\u003cp\u003eFrom the common steps involved in performing an SLR for academic research, as identified in the study design, it was observed that in step 1, Semantic Scholar had the highest score of 33, followed by Elicit and PaperDigest scoring 29 each. The benchmark Google Scholar\u0026rsquo;s score is 26. In step 2, Litmaps scored the highest (score 21), followed by Open Knowledge Map (score 20) and Connected papers (score 17). The benchmark Google Scholar\u0026rsquo;s score was 5. For step 3, only Scite.ai (score 18) was available which satisfied all the given parameters. It must be noted that Google Scholar is not an ideal benchmark to evaluate step 3 of the SLR process. Scite.ai and Google Scholar are both useful for academic research, but they differ in their approaches and functionalities. Scite.ai provides richer context with \u0026ldquo;Smart Citations\u0026rdquo;, summaries, and advanced sorting, while Google Scholar offers basic sorting and no summarization feature. Scite.ai aims to show diverse results by highlighting under-cited articles, while Google Scholar can sometimes return repetitive results or prioritize highly cited papers. Scite.ai offers a free basic plan and premium plans with more functionalities, while Google Scholar is free to use. Both offer saving options, though Scite.ai integrates with external software, and Scite.ai offers more varied import/export options. And finally, for step 4, Consensus.app (score 12) was the highest in the category followed by perplexity (score 9). Again, Google Scholar is not an ideal benchmark to compare these AI tools.\u003c/p\u003e\n\u003ch3\u003e4.1.1. \u0026nbsp; \u0026nbsp; The proposed workflow\u003c/h3\u003e\n\u003cp\u003e1. Step 1: Semantic Scholar - For initial paper search.\u003c/p\u003e\n\u003cp\u003e2. Step 2: Litmaps - Identify seed paper and create a knowledge graph of related papers.\u003c/p\u003e\n\u003cp\u003e3. Step 3: Scite.ai - Find relevant validation about the paper.\u003c/p\u003e\n\u003cp\u003e4. Step 4: Consensus.ai - Get a summary with proper references.\u003c/p\u003e\n\u003ch3\u003e4.1.2. \u0026nbsp; \u0026nbsp;Feedback from users\u003c/h3\u003e\n\u003cp\u003eEighteen postgraduate students from various academic fields, such as virtual reality, eye-tracking, physical education, and information retrieval, were asked to conduct a Systematic Literature Review (SLR) using a proposed workflow. The proposed workflow was evaluated using rubrics, and 77% of the students rated it 4 or higher on a scale of 5. Of these students, 72.7% reported that the search functionality of the workflow significantly improved their overall SLR process in terms of accuracy, completeness, and ease of use. Additionally, 72.8% of the students acknowledged that the proposed workflow improved their efficiency in conducting SLR. Regarding the accuracy of the results, 72.7% rated it 4 or higher on a scale of 5. Moreover, 91% of the students agreed that the proposed model helped them complete the SLR process faster and with fewer errors. They also expressed their willingness to recommend the proposed workflow to others. A combination of user responses is shown in Table 1.\u003c/p\u003e\n\u003ch2\u003e4.2. \u0026nbsp; Observations from Phase 2\u003c/h2\u003e\n\u003cp\u003eThe study\u0026rsquo;s recommendation of the Systematic Literature Review (SLR) workflow, involving AI tools, reveals positive feedback from participants. Out of 45 respondents, 29 rated the workflow as highly effective (either 4 or 5 on a 5-point scale), with several citing its ease of use and the ability to efficiently identify relevant literature as key benefits. The integration of AI tools such as Litmaps, SciSpace, and Jenni.ai improved the speed and accuracy of their research, allowing for better data-driven decisions. Many participants, with over 60% indicating satisfaction, highlighted the workflow\u0026rsquo;s ability to reduce errors and improve overall research productivity. Additionally, 32 respondents appreciated the replacement of general search engines like Google with more specialized AI-driven tools, which significantly increased their satisfaction.\u003c/p\u003e\n\u003ch3\u003e4.2.1. \u0026nbsp; \u0026nbsp;The proposed workflow\u003c/h3\u003e\n\u003cp\u003e1. Step 1: Elicit - For initial paper search.\u003c/p\u003e\n\u003cp\u003e2. Step 2: Litmaps - Identify seed paper and create a knowledge graph of related papers.\u003c/p\u003e\n\u003cp\u003e3. Step 3: scispace - Find relevant validation about the paper.\u003c/p\u003e\n\u003cp\u003e4. Step 4: Consensus.ai - Get a summary with proper references.\u003c/p\u003e\n\u003cp\u003e5. Step 5: Jenni.ai - writing.\u003c/p\u003e\n\u003ch3\u003e4.2.2. \u0026nbsp; \u0026nbsp;Feedback from users\u003c/h3\u003e\n\u003cp\u003eThe feedback from users on the AI-driven SLR workflow was largely positive. Out of 45 respondents, 32 found the workflow useful for identifying relevant literature more efficiently than traditional search methods like Google. Ease of use and increased efficiency in conducting SLRs were frequently mentioned, with many participants also appreciating the use of data-driven parameters to improve accuracy and reduce errors. The most favoured formats for further learning were recorded lectures (MOOC format), preferred by 20 participants, and online workshops, preferred by 15. Overall, the workflow was well-received, and many expressed interest in attending future courses.\u003c/p\u003e\n\u003ch3\u003e4.2.3. \u0026nbsp; \u0026nbsp;Frequency of Top 5 Feature Combinations\u003c/h3\u003e\n\u003cp\u003eA Combination analysis was conducted on the data obtained from user feedback to identify the feature preferences. It was also done to identify which alternative stacks or combinations students would prefer.\u003c/p\u003e\n\u003cp\u003eOnly the top 5 feature combinations were considered for this analysis. We coded the features of the workflow as follows: A: Searching for and identifying related and relevant literature, instead of using Google search. B: Increased efficiency in conducting SLR. C: Ease of use. D: Use data-driven parameters to increase the accuracy of results and reduce the chances of errors.\u003c/p\u003e\n\u003cp\u003eThe most common combination is all four features together (A+B+C+D), chosen by nearly 29% of respondents (shown in Fig. 1). This suggests that a significant portion of users find value in the entire feature set. The second most common combinations are A+B+C and A+B+D, each selected by 13.33% of respondents. This indicates that when not selecting all features, users often choose three out of the four, with either ease of use or data-driven parameters being omitted. Feature A (Searching for and identifying related literature) appears in most of the top combinations, reinforcing its importance as identified in earlier analyses. Very few respondents selected only one feature, suggesting that most users find value in multiple aspects of the workflow. There\u0026rsquo;s a strong correlation between features A and B, as they appear together in many of the top combinations. While \u0026ldquo;Ease of Use\u0026rdquo; (C) is often selected, it\u0026rsquo;s not always chosen in combination with other features, which could indicate that some users might prioritize functionality over ease of use.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 1: \u003c/strong\u003eResults of the combination analysis\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"258\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eFeature\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eCombination\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eNumber\u0026nbsp;of Responses\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+B+C+D\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e13 (28.89%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+B+C\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e6 (13.33%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+B+D\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e6 (13.33%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+C\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e4\u0026nbsp;(8.89 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+B\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e4\u0026nbsp;(8.89 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA only\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e3\u0026nbsp;(6.67 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eA+C+D\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e2\u0026nbsp;(4.44 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eB+C\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e2\u0026nbsp;(4.44 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eB+D\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e1\u0026nbsp;(2.22 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eC+D\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e1\u0026nbsp;(2.22 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eB only\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e1\u0026nbsp;(2.22 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eC only\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e1\u0026nbsp;(2.22 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 127px;\"\u003e\n \u003cp\u003eD only\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 131px;\"\u003e\n \u003cp\u003e1\u0026nbsp;(2.22 %)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\u003c/br\u003e\n\u003cp\u003eBased on the combination analysis of the workflow features, we can draw several important insights about user preferences and the perceived value of different feature combinations. The most striking observation is that nearly 29% of respondents found all four features useful, suggesting that a comprehensive approach to literature review assistance is highly valued. This preference for a full-featured solution indicates that users appreciate a tool that addresses multiple aspects of the systematic literature review process simultaneously.\u003c/p\u003e"},{"header":"5. Discussion","content":"\u003ch2\u003e5.1. \u0026nbsp; Comparing Elicit, Semantic Scholar and Paper Digest\u003c/h2\u003e\n\u003cp\u003eElicit is praised for its unique ability to predict research outcomes, while Semantic Scholar is noted for its extensive literature search capabilities and comprehensive coverage of academic databases. PaperDigest is highlighted for its personalized summaries and recommendations, as well as its ability to track research progress over time. However, each tool also has its limitations. For example, Elicit is limited to specific research areas and data types, Semantic Scholar can be overwhelming for beginners, and PaperDigest\u0026rsquo;s summaries may oversimplify complex research. In conclusion, Elicit, Semantic Scholar, and PaperDigest are each suited to different research needs, with Elicit being the ideal choice for experimental prediction, Semantic Scholar for comprehensive literature search, and PaperDigest for personalized research summaries. Based on these observations, Elicit was selected as the preferred tool for the workflow used in Phase 2, despite Semantic Scholar having the higher score. Table 2 shows the comparison between Elicit Semantic Scholar and PaperDigest.\u003c/p\u003e\n\u003ch2\u003e5.2. \u0026nbsp; Comparison between Litmaps, Open Knowledge maps and Connected Paper\u003c/h2\u003e\n\u003cp\u003eLitmaps offers customizable maps and interactive features to visualize the research landscape and emerging topics. It also provides a user-friendly interface for exploring research trends and related papers. The Open knowledge map offers a unique perspective for understanding broader thematic relationships and exploring concepts beyond literature. It is a versatile and open-source tool for exploring connections between diverse topics using knowledge maps. Connected Papers, on the other hand, provides a simple and fast solution for basic citation network analysis and quick discovery of related papers. It is efficient with its citation-based approach to quickly discover related papers based on citations and visualize research landscapes. For comprehensive knowledge mapping, Open Knowledge Maps is an ideal tool, but it requires technical knowledge. Litmaps offers the most intuitive interface for visually exploring research landscapes, while Connected Papers is the simplest and fastest option available for quickly finding related research and citations. Table 3 shows a comparison between Litmaps, Open Knowledge maps and Connected Paper.\u003c/p\u003e\n\u003ch2\u003e5.3. \u0026nbsp; Comparison between Consensus.ai and perplexity\u003c/h2\u003e\n\u003cp\u003eConsensus.ai is a good option for collaborative writing and research due to its valuable collaborative tools and citation management features. It is also ideal for analyzing large amounts of text and identifying key points and areas of agreement and disagreement. However, it is limited to analyzing existing text and cannot generate original content. Perplexity, on the other hand, is great for extensive content creation and analysis due to its advanced AI techniques, diverse writing styles, and extensive analysis features. It is also versatile in text analysis and can extract insights from various sources. However, it may lack source validity in generated content (may refer to grey literature) and requires careful fact-checking. Table 5 shows a comparison between Consensus.ai and perplexity.\u003c/p\u003e\n\u003ch2\u003e5.4. \u0026nbsp; User Response on the Proposed Workflow\u003c/h2\u003e\n\u003cp\u003eFeedback from both courses provides compelling evidence that AI tools significantly enhance the systematic literature review process. The results show that AI tools like Semantic Scholar, Litmaps, and Scite.ai outperform traditional tools like Google Scholar by offering better organization, validation, and accuracy. These tools also reduce errors and speed up the SLR process, with users reporting greater satisfaction and efficiency.\u003c/p\u003e\n\u003cp\u003eThe proposed workflow, which integrates AI tools at every stage of the SLR process, was highly rated by partic- ipants in both studies, showing that AI tools are becoming indispensable in academic research. The combination of improved search functionality, increased accuracy, and ease of use makes AI an essential part of modern SLR processes, allowing researchers to conduct more thorough and reliable reviews in less time.\u003c/p\u003e\n\u003cp\u003eIt was also noted that the prominence of Searching for and identifying related and relevant literature (A) across various combinations underscores its fundamental importance to users, likely due to its role in streamlining the often time-consuming process of literature discovery. The frequent pairing of this feature with increased efficiency (B) fur- ther emphasizes users\u0026rsquo; focus on productivity enhancements. Interestingly, while ease of use (C) was often selected, it wasn\u0026rsquo;t universally chosen in combination with other features. This could suggest that some users are willing to trade simplicity for more advanced functionality, especially when it comes to improving search capabilities and efficiency.\u003c/p\u003e\n\u003ch2\u003e5.5. \u0026nbsp; Limitations of the Study\u003c/h2\u003e\n\u003ch3\u003e5.5.1. \u0026nbsp; \u0026nbsp;Rubrics-based evaluation model\u003c/h3\u003e\n\u003cp\u003eRubrics are a valuable tool for comparing AI tools, but they do have limitations. One study found that rubrics can be limited by the subjectivity of the evaluator, the complexity of the evaluation process, and the potential for evaluator bias [Tikayat Ray et al., 2023]. Additionally, the use of rubrics may not fully capture the dynamic and evolving nature of AI tools, which can make it challenging to create a comprehensive and accurate evaluation [Pan et al., 2018]. These limitations highlight the need for a balanced approach that combines rubric-based evaluations with other methods to ensure a comprehensive and accurate assessment of AI tools.\u003c/p\u003e\n\u003ch3\u003e5.5.2. \u0026nbsp; \u0026nbsp;The constant evolution and growth of AI-based tools\u003c/h3\u003e\n\u003cp\u003eCreating a standard rubric for the comparison of AI tools is challenging due to the diverse nature of AI technologies and the complex evaluation criteria involved. The metrics and benchmarks used to evaluate AI tools are multifaceted and can vary significantly based on the specific application and context [Leung et al., 2023]. The effectiveness, reliability, and robustness of AI tools are typically assessed using metrics such as coverage, similarity, failures, mutation score, error rate, time, and domain expert opinion [Riccio et al., 2020]. However, converting these metrics to evaluate AI tools for SLR is a challenging task.\u003c/p\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eThe study investigated the feasibility of using a rubric-based evaluation approach to compare AI tools for constructing systematic literature review (SLR) workflows in academic research. A comprehensive rubric was established, encom- passing critical functionalities, performance metrics, and usability factors, to assess the strengths and weaknesses of various AI tools and identify potential candidates for streamlining the SLR process. While the rubric provided valuable insights, limitations such as potential oversimplification, subjectivity, and the dynamic nature of AI technol- ogy were acknowledged. As such, future research should consider complementing rubric-based evaluations with user testing, in-depth case studies, and ongoing monitoring of AI tool development.\u003c/p\u003e\n\u003cp\u003eThe variety of feature combinations selected also highlights the diverse needs of users engaged in systematic lit- erature reviews. While some prioritize a full suite of tools, others find specific combinations more suitable for their workflow. This diversity in preferences underscores the importance of offering flexible, modular solutions that can cater to different user needs and expertise levels. Overall, the analysis suggests that tools supporting systematic liter- ature reviews should prioritize robust search capabilities and efficiency improvements, while also offering data-driven features and maintaining user-friendliness to appeal to the broadest range of users.\u003c/p\u003e\n\u003cp\u003eUltimately, the most effective approach to selecting AI tools for SLR workflow creation may necessitate a combi- nation of quantitative and qualitative methods, tailored to individual researchers\u0026rsquo; specific needs and priorities. This research lays a valuable foundation for further exploration of AI tools in the realm of SLR, paving the way for more efficient, accurate, and accessible research methodologies in the future.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eAll authors agree that the adult participants (no minors involved) consented to participate in the study. This participation was conducted under the previously sought institutional approval, referenced as #IITB-IRB/2023/037.\u003c/p\u003e\n\u003cp\u003eFunding - No funding was received for conducting this study.\u003c/p\u003e\n\u003cp\u003eConflict of interest/Competing interests - The authors have no conflicts of interest to declare that are relevant to the content of this article.\u003c/p\u003e\n\u003cp\u003eEthics approval and consent to participate - All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and its later amendments or comparable ethical standards. Clinical trial number: not applicable\u003c/p\u003e\n\u003cp\u003eData availability - Data can be shared with reviewers, if required.\u003c/p\u003e\n\u003cp\u003eCode availability - Not Applicable\u003c/p\u003e\n\u003cp\u003eAuthor contribution -Conceptualization: Syaamantak Das, Sarthak Saini; Methodology: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha; Formal analysis and investigation: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha; Writing - original draft preparation: Syaamantak Das, Sarthak Saini, Nisha Biju, Alisha Sinha, Ganesh Beniwal; Writing - review and editing: Syaamantak Das, Nisha Biju, Alisha Sinha, Sarthak Saini, Ganesh Beniwal; Supervision: Syaamantak Das\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eAnderson and Jayaratne, 2015] Anderson, N. and Jayaratne, Y. S. N. (2015). Methodological challenges when per- forming a systematic review. \u003cem\u003eEuropean journal of orthodontics\u003c/em\u003e, 37 3:248\u0026ndash;50.\u003c/li\u003e\n \u003cli\u003eAtkinson, 2023] Atkinson, C. F. (2023). Cheap, quick, and rigorous: artificial intelligence and the systematic litera- ture review. \u003cem\u003eSocial Science Computer Review\u003c/em\u003e, page 08944393231196281.\u003c/li\u003e\n \u003cli\u003eBlaizot et al., 2022] Blaizot, A., Veettil, S. K., Saidoung, P., Moreno-Garc\u0026acute;ıa, C. F., Wiratunga, N., Aceves-Martins, M., Lai, N. M., and Chaiyakunapruk, N. (2022). Using artificial intelligence methods for systematic review in health sciences: A systematic review. \u003cem\u003eResearch Synthesis Methods\u003c/em\u003e, 13:353 \u0026ndash; 362.\u003c/li\u003e\n \u003cli\u003eBoeker et al., 2013] Boeker, M., Vach, W., and Motschall, E. (2013). Google scholar as replacement for systematic literature searches: good relative recall and precision are not enough. \u003cem\u003eBMC medical research methodology\u003c/em\u003e, 13(1):1\u0026ndash; 12.\u003c/li\u003e\n \u003cli\u003eBramer et al., 2018] Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., and Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. \u003cem\u003eJournal of the Medical Library Association: JMLA\u003c/em\u003e, 106(4):531.\u003c/li\u003e\n \u003cli\u003eChong et al., 2022] Chong, S. W., Lin, T. J., and Chen, Y. (2022). A methodological review of systematic literature reviews in higher education: Heterogeneity and homogeneity. \u003cem\u003eEducational Research Review\u003c/em\u003e, 35:100426.\u003c/li\u003e\n \u003cli\u003eChu and Evans, 2021] Chu, J. S. and Evans, J. A. (2021). Slowed canonical progress in large fields of science.\u0026nbsp;\u003cem\u003eProceedings\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eNational\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eAcademy\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eSciences\u003c/em\u003e, 118(41):e2021636118.\u003c/li\u003e\n \u003cli\u003eCohen et al., 2010] Cohen, A. M., Ambert, K. H., and McDonagh, M. S. (2010). A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. \u003cem\u003eAMIA ... Annual Symposium proceedings. AMIA Symposium\u003c/em\u003e, 2010:121\u0026ndash;5.\u003c/li\u003e\n \u003cli\u003eCollins et al., 2021] Collins, C., Dennehy, D., Conboy, K., and Mikalef, P. (2021). Artificial intelligence in informa- tion systems research: A systematic literature review and research agenda. \u003cem\u003eInternational Journal of Information Management\u003c/em\u003e, 60:102383.\u003c/li\u003e\n \u003cli\u003eDai, 2023] Dai, Y. (2023). Negotiation of epistemological understandings and teaching practices between primary teachers and scientists about artificial intelligence in professional development. \u003cem\u003eResearch in Science Education\u003c/em\u003e, 53(3):577\u0026ndash;591.\u003c/li\u003e\n \u003cli\u003eDas and Islam, 2021] Das, R. K. and Islam, M. S. U. (2021). Application of artificial intelligence and machine learning in libraries: A systematic review. \u003cem\u003eArXiv\u003c/em\u003e, abs/2112.04573.\u003c/li\u003e\n \u003cli\u003ede la Torre-Lo\u0026acute;pez et al., 2023] de la Torre-Lo\u0026acute;pez, J., Ram\u0026acute;ırez, A., and Romero, J. R. (2023). Artificial intelligence to automate the systematic review of scientific literature. \u003cem\u003eComputing\u003c/em\u003e, pages 1\u0026ndash;24.\u003c/li\u003e\n \u003cli\u003eElsman et al., 2022] Elsman, E. B., Butcher, N. J., Mokkink, L. B., Terwee, C. B., Tricco, A., Gagnier, J. J., Aiyegbusi, O. L., Barnett, C., Smith, M., Moher, D., et al. (2022). Study protocol for developing, piloting and dis- seminating the prisma-cosmin guideline: a new reporting guideline for systematic reviews of outcome measurement instruments. \u003cem\u003eSystematic reviews\u003c/em\u003e, 11(1):121.\u003c/li\u003e\n \u003cli\u003eEskrootchi et al., 2020] Eskrootchi, R., Shahraki Mohammadi, A., Panahi, S., and Zahedi, R. (2020). Librarians\u0026rsquo; participation in the systematic reviews published by iranian researchers and its impact on the quality of reporting search strategy. \u003cem\u003eEvidence Based Library and Information Practice\u003c/em\u003e, 15(2):69\u0026ndash;84.\u003c/li\u003e\n \u003cli\u003eFeng et al., 2022] Feng, Y., Liang, S., Zhang, Y., Chen, S., Wang, Q., Huang, T., Sun, F., Liu, X., Zhu, H., and Pan, H. (2022). Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis. \u003cem\u003eJournal\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u0026nbsp;the\u0026nbsp;American\u0026nbsp;Medical\u0026nbsp;Informatics\u0026nbsp;\u003c/em\u003e\u003cem\u003eAssociation\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003e: JAMIA\u003c/em\u003e.\u003c/li\u003e\n \u003cli\u003eFerreras-Ferna\u0026acute;ndez et al., 2016] Ferreras-Fern\u0026acute;andez, T., Rodero, H. M., Garc\u0026acute;ıa-Pen\u0026tilde;alvo, F. J., and Merlo-Vega, J. A. (2016). The systematic review of literature in lis: an approach. \u003cem\u003eProceedings of\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ethe Fourth International\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eConference\u0026nbsp;\u003c/em\u003e\u003cem\u003eon\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eTechnological\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eEcosystems\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003efor\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eEnhancing\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eMulticulturality\u003c/em\u003e.\u003c/li\u003e\n \u003cli\u003eGarc\u0026acute;ıa-Pen\u0026tilde;alvo, 2022] Garc\u0026acute;ıa-Pen\u0026tilde;alvo, F. J. (2022). Developing robust state-of-the-art reports: Systematic literature reviews.\u003c/li\u003e\n \u003cli\u003eGonza\u0026acute;lez-Calatayud et al., 2021] Gonz\u0026acute;alez-Calatayud, V., Prendes-Espinosa, P., and Roig-Vila, R. (2021). Artificial intelligence for student assessment: A systematic review. \u003cem\u003eApplied Sciences\u003c/em\u003e, 11(12):5467.\u003c/li\u003e\n \u003cli\u003eHaddaway et al., 2020] Haddaway, N. R., Bethel, A., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., Pullin, A. S., Savilaakso, S., and Stewart, G. B. (2020). Eight problems with literature reviews and how to fix them. \u003cem\u003eNature Ecology \u0026amp; Evolution\u003c/em\u003e, 4:1582 \u0026ndash; 1589.\u003c/li\u003e\n \u003cli\u003eHaddaway et al., 2018] Haddaway, N. R., Macura, B., Whaley, P., and Pullin, A. S. (2018). Roses reporting standards for systematic evidence syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. \u003cem\u003eEnvironmental Evidence\u003c/em\u003e, 7:1\u0026ndash;8.\u003c/li\u003e\n \u003cli\u003eJoho et al., 2015] Joho, H., Jatowt, A., and Blanco, R. (2015). Temporal information searching behaviour and strategies. \u003cem\u003eInf. Process. Manag.\u003c/em\u003e, 51:834\u0026ndash;850.\u003c/li\u003e\n \u003cli\u003eLeung et al., 2023] Leung, T. I., de Azevedo Cardoso, T., Mavragani, A., and Eysenbach, G. (2023). Best practices for using ai tools as an author, peer reviewer, or editor. \u003cem\u003eJournal of Medical Internet Research\u003c/em\u003e, 25(1):e51584.\u003c/li\u003e\n \u003cli\u003eMallett et al., 2012] Mallett, R., Hagen-Zanker, J., Slater, R., and Duvendack, M. (2012). The benefits and challenges of using systematic reviews in international development research. \u003cem\u003eJournal of development effectiveness\u003c/em\u003e, 4(3):445\u0026ndash; 455.\u003c/li\u003e\n \u003cli\u003eMcManus et al., 1998] McManus, R. J., Wilson, S. R., Delaney, B. C., Fitzmaurice, D. A., Hyde, C., Tobias, R. S., Jowett, S., and Hobbs, F. D. R. (1998). Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. \u003cem\u003eBMJ\u003c/em\u003e, 317:1562 \u0026ndash; 1563.\u003c/li\u003e\n \u003cli\u003eMlinari\u0026acute;c et al., 2017] Mlinari\u0026acute;c, A., Horvat, M., and Sˇupak Smolˇci\u0026acute;c, V. (2017). Dealing with the positive publication bias: Why you should really publish your negative results. \u003cem\u003eBiochemia medica\u003c/em\u003e, 27(3):447\u0026ndash;452.\u003c/li\u003e\n \u003cli\u003eMohamed et al., 2023] Mohamed, H., Salsberg, J., and Kelly, D. (2023). An integrative review protocol on interven- tions to improve users\u0026rsquo; ability to identify trustworthy online health information. \u003cem\u003ePlos one\u003c/em\u003e, 18(4):e0284028.\u003c/li\u003e\n \u003cli\u003eMouta et al., 2023] Mouta, A., Torrecilla-S\u0026acute;anchez, E. M., and Pinto-Llorente, A. M. (2023). Design of a future scenarios toolkit for an ethical implementation of artificial intelligence in education. \u003cem\u003eEducation and Information Technologies\u003c/em\u003e, pages 1\u0026ndash;26.\u003c/li\u003e\n \u003cli\u003eMyllyaho et al., 2021] Myllyaho, L., Raatikainen, M., Ma\u0026uml;nnisto\u0026uml;, T., Mikkonen, T., and Nurminen, J. K. (2021). Systematic literature review of validation methods for ai systems. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, 181:111050.\u003c/li\u003e\n \u003cli\u003ePan et al., 2018] Pan, M., Linner, T., Pan, W., Cheng, H., and Bock, T. (2018). A framework of indicators for assessing construction automation and robotics in the sustainability context. \u003cem\u003eJournal of Cleaner Production\u003c/em\u003e, 182:82\u0026ndash; 95.\u003c/li\u003e\n \u003cli\u003ePark et al., 2023] Park, J., Teo, T. W., Teo, A., Chang, J., Huang, J. S., and Koo, S. (2023). Integrating artificial intelligence into science lessons: Teachers\u0026rsquo; experiences and views. \u003cem\u003eInternational Journal of STEM Education\u003c/em\u003e, 10(1):61.\u003c/li\u003e\n \u003cli\u003ePati and Lorusso, 2018] Pati, D. and Lorusso, L. N. (2018). How to write a systematic review of the literature.\u0026nbsp;\u003cem\u003eHERD:\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eHealth\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eEnvironments\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eResearch\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003e\u0026amp;\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eDesign\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eJournal\u003c/em\u003e, 11:15 \u0026ndash; 30.\u003c/li\u003e\n \u003cli\u003ePayne, 1976] Payne, J. W. (1976). Task complexity and contingent processing in decision making: An information search and protocol analysis. \u003cem\u003eOrganizational\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ebehavior\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eand\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ehuman\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eperformance\u003c/em\u003e, 16(2):366\u0026ndash;387.\u003c/li\u003e\n \u003cli\u003eRamesh and Sanampudi, 2022] Ramesh, D. and Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. \u003cem\u003eArtificial Intelligence Review\u003c/em\u003e, 55(3):2495\u0026ndash;2527.\u003c/li\u003e\n \u003cli\u003eRiccio et al., 2020] Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., and Tonella, P. (2020). Testing machine learning based systems: a systematic mapping. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, 25:5193\u0026ndash;5254.\u003c/li\u003e\n \u003cli\u003eRodi et al., 2017] Rodi, G. C., Loreto, V., and Tria, F. (2017). Search strategies of wikipedia readers. \u003cem\u003ePLoS ONE\u003c/em\u003e, 12.\u003c/li\u003e\n \u003cli\u003eSacchi and Burigo, 2008] Sacchi, S. and Burigo, M. (2008). Strategies in the information search process: Interaction among task structure, knowledge, and source. \u003cem\u003eThe Journal of General Psychology\u003c/em\u003e, 135:252 \u0026ndash; 270.\u003c/li\u003e\n \u003cli\u003eSantos et al., 2023] Santos, M. V., Morgado-Dias, F., and Silva, T. C. (2023). Oil sector and sentiment analysis\u0026mdash;a review. \u003cem\u003eEnergies\u003c/em\u003e, 16(12):4824.\u003c/li\u003e\n \u003cli\u003eSchiff and Mo\u0026uml;ller, 2021] Schiff, S. and Mo\u0026uml;ller, R. (2021). On human-aware information seeking. In \u003cem\u003eCHAI@KI\u003c/em\u003e. [Sesagiri Raamkumar et al., 2018] Sesagiri Raamkumar, A., Foo, S., and Pang, N. (2018). Can i have more of these please? assisting researchers in finding similar research papers from a seed basket of papers. \u003cem\u003eThe Electronic Library\u003c/em\u003e, 36(3):568\u0026ndash;587.\u003c/li\u003e\n \u003cli\u003eSharadgah and Sa\u0026rsquo;di, 2022] Sharadgah, T. A. and Sa\u0026rsquo;di, R. A. (2022). A systematic review of research on the use of artificial intelligence in english language teaching and learning (2015-2021): What are the current effects? \u003cem\u003eJournal of Information Technology Education: Research\u003c/em\u003e.\u003c/li\u003e\n \u003cli\u003eShi et al., 2022] Shi, D., Zhou, J., Wang, D., and Wu, X. (2022). Research status, hotspots, and evolutionary trends of intelligent education from the perspective of knowledge graph. \u003cem\u003eSustainability\u003c/em\u003e, 14(17):10934.\u003c/li\u003e\n \u003cli\u003eTikayat Ray et al., 2023] Tikayat Ray, A., Cole, B. F., Pinon Fischer, O. J., Bhat, A. P., White, R. T., and Mavris, D. N. (2023). Agile methodology for the standardization of engineering requirements using large language models. \u003cem\u003eSystems\u003c/em\u003e, 11(7):352.\u003c/li\u003e\n \u003cli\u003evan Dinter et al., 2021] van Dinter, R., Tekinerdogan, B., and Catal, C. (2021). Automation of systematic literature reviews: A systematic literature review. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, 136:106589.\u003c/li\u003e\n \u003cli\u003eWilliamson, 2023] Williamson, B. (2023). The social life of ai in education. \u003cem\u003eInternational\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eJournal\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003eof\u0026nbsp;\u003c/em\u003e\u003cem\u003eArtificial Intelligence\u003c/em\u003e\u003cem\u003e\u0026nbsp;\u003c/em\u003e\u003cem\u003ein\u0026nbsp;Education\u003c/em\u003e, pages 1\u0026ndash;8.\u003c/li\u003e\n\u003c/ol\u003e\n"},{"header":"Tables","content":"\u003cp\u003eTables 2 to 5 are available in the Supplementary Files section.\u003c/p\u003e"},{"header":"Footnotes","content":"\u003cp\u003e\u003csup\u003e1\u003c/sup\u003ehttps://typeset.io/resources/ai-tools-for-systematic-literature-review/\u003c/p\u003e\n\u003cp\u003e\u003csup\u003e2\u003c/sup\u003ehttps://blogs.lse.ac.uk/impactofsocialsciences/2020/10/19/8-common-problems-with-literature-reviews-and-how-to-fix-them/\u003c/p\u003e\n\u003cp\u003e\u003csup\u003e3\u003c/sup\u003ehttps://paperpile.com/g/academic-search-engines/\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Systematic Literature Review, Academic Research, Information Search","lastPublishedDoi":"10.21203/rs.3.rs-6328602/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6328602/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSystematic Literature Reviews (SLR) are a critical aspect of academic research, as they provide a comprehensive and rigorous synthesis of existing evidence. However, conducting such reviews presents several challenges that researchers must be aware of to ensure the quality and validity of their work. One of the most significant challenges is developing a clear research question through a comprehensive literature search. This is because it can be difficult to identify appropriate and relevant studies. Additionally, the traditional literature review process has the potential for bias. To address these challenges, researchers need a search protocol and strategy that includes a set of available artificial intelligence-based tools as an ad-hoc tech stack (workflow) to conduct SLR. This study evaluated a proposed model of a workflow and stack of such tools using a rubric to identify the best possible set of tools for the task. For one-year, several AI-based tools were evaluated in two phases, and a proposed workflow using the best of those tools was identified. Sixty-three students used the proposed model to conduct their academic research. The results showed that it is possible to perform academic research more efficiently if a workflow is used for performing SLR.\u003c/p\u003e","manuscriptTitle":"Designing an AI-Driven SLR Workflow for Academic Research: A Rubric for Comparative Analysis of AI Tools","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-28 08:27:27","doi":"10.21203/rs.3.rs-6328602/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d1c6b802-e63e-421d-b0cc-599b66947d92","owner":[],"postedDate":"May 28th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-01-05T16:04:19+00:00","versionOfRecord":{"articleIdentity":"rs-6328602","link":"https://doi.org/10.1007/s44163-025-00628-8","journal":{"identity":"discover-artificial-intelligence","isVorOnly":false,"title":"Discover Artificial Intelligence"},"publishedOn":"2025-12-29 15:58:12","publishedOnDateReadable":"December 29th, 2025"},"versionCreatedAt":"2025-05-28 08:27:27","video":"","vorDoi":"10.1007/s44163-025-00628-8","vorDoiUrl":"https://doi.org/10.1007/s44163-025-00628-8","workflowStages":[]},"version":"v1","identity":"rs-6328602","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6328602","identity":"rs-6328602","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}