Speed and Sensitivity in Deduplication: Evaluating DeDupli Against Established Tools for Systematic Reviews

Speed and Sensitivity in Deduplication: Evaluating DeDupli Against Established Tools for Systematic Reviews | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Speed and Sensitivity in Deduplication: Evaluating DeDupli Against Established Tools for Systematic Reviews Satchit Sagar, Dikhra Khan, Rajashekhar CH T, Sambit Sagar, Pooja Tiwari, and 10 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7548423/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Duplicate records are a common challenge in systematic reviews, arising from database overlap and multiple citation formats. Failure to address duplicates increases reviewer workload and risks bias. Existing tools such as EndNote and Rayyan offer deduplication features, but evaluations highlight trade-offs between sensitivity and specificity. DeDupli , a novel Streamlit-based, privacy-friendly tool, was developed to balance speed, accuracy, and user oversight through automatic and manual deduplication modes. Methods We generated three synthetic RIS datasets (262, 200, and 320 records) with known duplicate clusters to simulate real-world bibliographic search outputs. Three independent reviewers evaluated deduplication performance across EndNote, Rayyan, and DeDupli (automatic and manual modes) in a crossover design. Reviewers recorded duplicates removed, uniques retained, and processing time. Ground truth was used to calculate true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Performance metrics included sensitivity, specificity, precision, and F1-score. Results Across 1,564 records with 428 true duplicates, DeDupli achieved perfect sensitivity (1.000) , eliminating all duplicates. EndNote and Rayyan achieved perfect specificity and precision (1.000), but each missed 13 duplicates (sensitivity 0.939). DeDupli-Auto processed datasets in under 5 seconds but introduced more false positives (precision 0.899, F1 0.947). DeDupli-Manual required ~5 minutes, reduced false positives, and improved precision (0.934) and F1 (0.966). By contrast, EndNote (2636 s) and Rayyan (2580 s) required ~40 minutes per dataset. Reviewer-level analysis showed minimal variability, confirming consistency across raters. Conclusion DeDupli demonstrated superior sensitivity and substantial time savings compared with EndNote and Rayyan. Its dual-mode design offers flexibility, allowing users to prioritize either speed (automatic) or accuracy (manual). These results highlight DeDupli as a promising, privacy-friendly solution to streamline deduplication in systematic reviews and meta-analyses. Validation on real-world datasets and integration into review workflows are recommended for future work. Systematic Review Meta Analysis Deduplication Automatic deduplication Figures Figure 1 Figure 2 Figure 3 Introduction Systematic reviews (SRs) and meta-analyses are cornerstones of evidence-based medicine, providing high-quality summaries of existing literature to guide clinical practice and policy decisions. A critical early step in conducting SRs involves retrieving citations from multiple bibliographic databases (e.g., MEDLINE, Embase, Cochrane CENTRAL), which often index overlapping records. This inevitably leads to the inclusion of duplicate references, increasing reviewer workload and potentially introducing bias if duplicates are not properly managed [1]. The process of identifying and removing duplicate records— deduplication —is therefore essential for both methodological rigor and efficiency [2]. Traditional reference management software (e.g., EndNote, Mendeley, Zotero) offers deduplication features, but comparative evaluations have shown that these tools are prone to both false negatives (missed duplicates) and false positives (unique records incorrectly removed) [3]. More recently, systematic review–specific platforms such as Rayyan and Covidence have incorporated more advanced deduplication capabilities and generally outperform generic reference managers in both accuracy and usability [4]. Empirical assessments demonstrate that tool performance varies, with important trade-offs between sensitivity and specificity. McKeown and Mir (2021) reported that Rayyan achieved higher sensitivity (i.e., fewer missed duplicates), whereas Covidence and Ovid tools had higher specificity [3]. A recent 2024 evaluation reaffirmed that SR-focused tools are superior overall to traditional reference managers, while also highlighting the importance of transparency and reproducibility in deduplication workflows [4]. Newer approaches leverage automation and machine learning to enhance performance. The Deduklick algorithm, for example, integrates rule-based heuristics and natural language processing to achieve near-perfect precision and recall [5]. Similarly, the Systematic Review Accelerator (SRA) Deduplicator module demonstrated superior accuracy and speed compared to EndNote, significantly reducing manual burden [2]. Despite these advances, few tools strike an optimal balance between speed , accuracy , and user oversight , while also being accessible and privacy-friendly. In this study, we benchmark DeDupli , a novel Streamlit-based deduplication tool, against Rayyan and EndNote. We evaluate its automatic and manual modes across multiple synthetic datasets, assessing sensitivity, specificity, precision, F1-score, and processing time. Methods Dataset preparation We generated three synthetic bibliographic datasets in RIS format to simulate systematic review search outputs. Dataset A contained 262 records , Dataset B contained 200 records , and Dataset C contained 320 records . Each dataset was seeded with duplicate clusters to mimic real-world scenarios of imperfect bibliographic imports. Duplicate creation methods included punctuation changes, abbreviated author names, removal of DOIs, database source alterations, minor title variations, and year shifts. The true duplicate structure was retained in a ground truth file , where records sharing a cluster_id were considered duplicates and singleton clusters were considered unique references. Reviewer assignments Three independent reviewers were assigned datasets in a rotating design to balance tool use: Reviewer 1 evaluated Dataset A in Rayyan, Dataset B in EndNote, Dataset C in DeDupli (Auto + Manual modes). Reviewer 2 evaluated Dataset B in Rayyan, Dataset C in EndNote, Dataset A in DeDupli (Auto + Manual). Reviewer 3 evaluated Dataset C in Rayyan, Dataset A in EndNote, Dataset B in DeDupli (Auto + Manual). This ensured that each dataset was reviewed across all tools and that each reviewer interacted with all tools. Tools compared Rayyan (Qatar Computing Research Institute) – online screening tool with duplicate detection. EndNote (Clarivate Analytics) – reference management software with semi-automated deduplication. DeDupli v1.1 (Dr Sambit Sagar Dr Dikhra Khan) – a privacy-friendly Streamlit application, tested in both Automatic and Manual override modes. https://dedupli-vd42ahhyjevixmtsaaqvut.streamlit.app/ Reviewer workflow Reviewers imported their assigned datasets into the allocated tool and ran the duplicate detection function. For each dataset-tool combination, reviewers reported aggregate counts only: Number of duplicates removed Number of unique references retained Time taken (seconds) Notes/comments Data entry was performed in a standardized Excel template. No per-record labeling was required, reducing reviewer burden. Ground truth reference standard True duplicate and unique counts were derived from the seeded cluster file. True Positives (TP): duplicates correctly removed True Negatives (TN): uniques correctly retained False Positives (FP): uniques wrongly removed False Negatives (FN): duplicates missed Metrics From TP, TN, FP, FN we calculated: Sensitivity (recall) = TP / (TP + FN) Specificity = TN / (TN + FP) Precision = TP / (TP + FP) F1 score = 2·TP / (2·TP + FP + FN) Statistical analysis All analyses were performed in R (version 4.4) using tidyverse, readxl, and janitor. Results Across three datasets (782 records total), each dataset was evaluated once in EndNote, once in Rayyan, and twice in DeDupli (automatic and manual modes), yielding a total of 3,128 record instances for analysis.Across three synthetic datasets (Dataset A = 262, Dataset B = 200, Dataset C = 320; total 782 unique records), ground-truth clustering identified 354 true unique references and 428 duplicate records belonging to duplicate groups. Overall performance across tools All tools demonstrated high overall performance . DeDupli (pooled across Auto and Manual modes) achieved perfect sensitivity (1.000) , identifying and removing all true duplicates across datasets. However, this came at the cost of a small number of false positives (39 uniques incorrectly removed), reducing its specificity to 0.964 and precision to 0.916 , with a resulting F1 score of 0.956 . By contrast, EndNote and Rayyan demonstrated perfect specificity and precision (1.000) , never removing a true unique, but each failed to identify 13 duplicates, yielding lower sensitivity ( 0.939 ) despite similar F1 scores ( 0.969 ). Thus, EndNote and Rayyan maximized specificity, while DeDupli maximized sensitivity. Tool variants: Auto vs Manual When evaluated separately (Table 1), DeDupli-Auto provided near-instantaneous deduplication (mean 5 s per dataset) and achieved perfect sensitivity, but generated a higher number of false positives (n = 24), resulting in lower precision ( 0.899 ) and F1 ( 0.947 ). The DeDupli-Manual mode reduced false positives to 15, improving both precision ( 0.934 ) and F1 ( 0.966 ), with only modest additional processing time (mean 320 s). Importantly, both variants maintained perfect sensitivity, ensuring that no duplicate was missed — a key advantage over the comparator tools. Comparison of EndNote and Rayyan While the pooled totals for EndNote and Rayyan were identical , per-dataset analyses revealed subtle differences. In Dataset A, Rayyan removed 71 duplicates (FN = 3) compared to EndNote’s 70 (FN = 4). In Dataset B, Rayyan again outperformed EndNote slightly (41 vs 40 duplicates removed). Conversely, in Dataset C, EndNote removed 91 duplicates (FN = 5), whereas Rayyan removed only 89 (FN = 7). These small differences balanced out in aggregate, yielding identical pooled sensitivity (0.939) and F1 (0.969). Reviewer-level consistency The reviewer-level analyses (Supplementary Appendix Table) showed consistent patterns across datasets and tools. Inter-reviewer variability was minimal: each reviewer reproduced the same strengths and weaknesses of the tools, suggesting that performance differences arose from the algorithms rather than user factors. Heatmaps of F1 scores confirmed uniformly high performance across reviewers, datasets, and tools. Processing time Marked differences were observed in processing time. DeDupli-Auto averaged less than 5 seconds per dataset , while DeDupli-Manual averaged 320 seconds . In contrast, both EndNote (mean 2636 seconds; ~44 min) and Rayyan (mean 2580 seconds; ~43 min) required substantially longer times, reflecting their more cumbersome workflows and manual steps. This highlights DeDupli’s strength as a privacy-friendly and rapid deduplication solution , with Auto mode optimized for speed and Manual mode offering a balance between speed and accuracy. Discussion In this study, we compared the performance of DeDupli , a novel privacy-friendly deduplication tool, against EndNote and Rayyan using synthetic datasets with known ground truth. Across all datasets, DeDupli achieved perfect sensitivity (1.000), successfully identifying and removing all duplicates without omission. By contrast, EndNote and Rayyan maintained perfect specificity and precision but consistently missed duplicates, reducing sensitivity to 0.939 and leading to lower overall recall. Comparison with prior work Our findings are consistent with prior evaluations of deduplication tools. McKeown and Mir reported that Rayyan exhibited higher sensitivity than EndNote, whereas EndNote was more conservative and therefore less likely to introduce false positives [3]. In our dataset-level analysis, small differences between Rayyan and EndNote were also observed—Rayyan identified slightly more duplicates in some datasets but missed more in others—yet their pooled totals were identical. This highlights that sensitivity and specificity trade-offs remain a recurrent theme in deduplication research. Automation has been repeatedly shown to improve efficiency. The Systematic Review Accelerator (SRA) Deduplicator significantly reduced error rates and improved time efficiency compared to EndNote [2]. Similarly, the Deduklick algorithm demonstrated near-perfect recall and precision using a reproducible and explainable workflow [5]. Our evaluation confirms these trends: DeDupli-Auto achieved perfect recall with processing times under 5 seconds per dataset , markedly faster than EndNote (~44 minutes) and Rayyan (~43 minutes). Strengths of DeDupli Unlike many existing tools, DeDupli offers both automatic and manual modes . The automatic mode provides unparalleled speed, making it well-suited for large-scale reviews where efficiency is critical. The manual mode, while slower (~5 minutes per dataset), enables reviewer oversight and reduced false positives, improving precision and F1-score compared to the automatic mode. This dual-mode design directly addresses concerns in systematic review methodology about balancing speed with transparency and control [7]. The tool’s local, privacy-friendly implementation further differentiates it from cloud-based platforms such as Rayyan, which may raise data security concerns in sensitive contexts. Combined, these features suggest that DeDupli can fill an important niche between traditional reference managers and more complex automated platforms. Limitations This evaluation was performed on synthetic datasets , carefully designed to emulate real-world duplicate structures. While this ensured a definitive ground truth, external validation on real bibliographic search outputs is still required. Our comparisons were limited to EndNote and Rayyan; additional benchmarking against other tools such as Covidence, Ovid, or EPPI-Reviewer would provide a more comprehensive landscape [8]. Finally, although DeDupli markedly reduced time burden, we did not formally assess user experience or integration with existing SR workflows, which remain important considerations. Future directions Future development of DeDupli could explore integration of more advanced similarity measures, including fuzzy string matching and NLP-based algorithms, to further reduce false positives in the automatic mode. Adding functionality for multi-user collaboration and direct integration with SR management platforms would further enhance usability. In addition, future studies should examine how deduplication accuracy and speed translate into downstream improvements in screening efficiency, reviewer fatigue, and overall systematic review timelines. Conclusion In this evaluation, DeDupli demonstrated excellent performance compared with two widely used deduplication tools, EndNote and Rayyan . By achieving perfect sensitivity across all datasets, DeDupli ensured that no duplicate was missed, while offering both an automatic mode for rapid processing and a manual mode that improved precision by reducing false positives. Processing times were markedly shorter than those observed with EndNote and Rayyan, underscoring DeDupli’s efficiency advantage. These findings suggest that DeDupli offers a pragmatic balance between speed, accuracy, and transparency , addressing a long-standing methodological challenge in systematic review workflows. Its local, privacy-friendly design further enhances its suitability for sensitive or large-scale reviews. While external validation on real-world bibliographic datasets and broader comparisons with additional platforms are warranted, DeDupli represents a promising step toward streamlining evidence synthesis and reducing reviewer burden in systematic reviews and meta-analyses. Declarations Transparency and Ethical Requirements Declaration of funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors . The development and evaluation of the DeDupli tool were undertaken as part of the authors’ academic and professional activities without external sponsorship. Declaration of financial/other relationships All authors declare that they have no financial, employment, or other relationships that could be construed as a conflict of interest with respect to this study. Acknowledgements No assistance in the preparation of this article is to be declared. This work has not been previously published or presented in any conference proceedings. Author Contributions Conception and design of the study: Sambit Sagar, Satchit Sagar, Dikhra Khan Development of DeDupli tool: Sambit Sagar, Satchit Sagar, Dikhra Khan Data curation and ground truth preparation: Pooja Tiwari, Nivedita Kundu, Arup Roy, Lalitha Goriparthi, Jagatti Krishna, Aparna Mahalik Performance evaluation and analysis: Dikhra Khan, Amlan Rout, Hemant Khairwa, Rajashekhar CH T Drafting of the manuscript: Dikhra Khan, Sambit Sagar, Satchit Sagar Critical revision of the manuscript for important intellectual content: Arup Roy, Lalitha Goriparthi, Nivedita Kundu, Aparna Mahalik, Jagatti Krishna, Avnish Singh, Amlan Rout Final approval of the version to be published: All authors Accountability: All authors agree to be accountable for all aspects of the work, ensuring accuracy and integrity. Data availability statement The data that support the findings of this study are available from the corresponding author, [Dr Sambit Sagar], upon reasonable request. References Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module. Systematic Reviews . 2015;4:6. doi:10.1186/2046-4053-4-6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-6 Forbes C, Greenwood H, Carter M, Clark J. Automation of duplicate record detection for systematic reviews: Deduplicator. Systematic Reviews . 2024;13(1):206. doi:10.1186/s13643-024-02619-9. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-024-02619-9 McKeown S, Mir ZM. Comparison of methods for identifying duplicate records in systematic reviews. Systematic Reviews . 2021;10(1):150. doi:10.1186/s13643-021-01583-y. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01583-y Janka H, Metzendorf M-I. High precision but variable recall – comparing the performance of five deduplication tools. Journal of the European Association for Health Information and Libraries (JEAHIL) . 2024;20(1):12–17. doi:10.32384/jeahil20607. Available from: https://ojs.eahil.eu/JEAHIL/article/view/607 Borissov N, Haas Q, Minder B, Kopp-Heim D, von Gernler M, Janka H, et al. Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. Systematic Reviews . 2022;11(1):172. doi:10.1186/s13643-022-02045-9. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-022-02045-9 Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev . 2015;4:80. doi:10.1186/s13643-015-0067-6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-015-0067-6 Janka H, Metzendorf M-I. High precision but variable recall – comparing the performance of five deduplication tools. J Eur Assoc Health Inf Libr . 2024;20(1):12–17. doi:10.32384/jeahil20607. Available from: https://ojs.eahil.eu/JEAHIL/article/view/607 van der Mierden S, Tseng YJ, de Bruin J, Hooft L. Computer-assisted screening of records in systematic reviews: a systematic review. Syst Rev . 2019;8:236. doi:10.1186/s13643-019-1175-7. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1175-7 Table Table 1: Performance by tool variant Tool Variant N TP TN FP FN Sensitivity Specificity Precision F1 Mean Time (s) DeDupli-Auto 782 214 520 24 0 1.000 0.956 0.899 0.947 5 DeDupli-Manual 782 214 538 15 0 1.000 0.973 0.934 0.966 320 EndNote 782 201 568 0 13 0.939 1.000 1.000 0.969 2636 Rayyan 782 201 568 0 13 0.939 1.000 1.000 0.969 2580 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7548423","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":511144628,"identity":"131ccee9-3af6-48e2-a94f-fb5679b8da98","order_by":0,"name":"Satchit Sagar","email":"","orcid":"","institution":"EFLU","correspondingAuthor":false,"prefix":"","firstName":"Satchit","middleName":"","lastName":"Sagar","suffix":""},{"id":511144629,"identity":"eb125a2d-d577-476a-aa94-d42484f04221","order_by":1,"name":"Dikhra Khan","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Dikhra","middleName":"","lastName":"Khan","suffix":""},{"id":511144630,"identity":"511521d9-78c3-4498-80e9-e3a8eb89b78f","order_by":2,"name":"Rajashekhar CH T","email":"","orcid":"","institution":"AIIMS-CAPFIMS","correspondingAuthor":false,"prefix":"","firstName":"Rajashekhar","middleName":"CH","lastName":"T","suffix":""},{"id":511144631,"identity":"4584e790-46e3-41d7-88d3-065cc11ae28c","order_by":3,"name":"Sambit Sagar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEUlEQVRIiWNgGAWjYDCCA0DM2CDBwMB88MEBBgMGGX6QaEIBYS0SDGzJBiAtPJINIC0GBLUwgLWA+DwGIBEGPFr4bh9/+OnmDos6fjZmxgM/Cux4jM+vTvzwwIBBnl/sAFYtkudyjKVzz0hISLYxMxzsMUjmMbvxdrME0GGGM2cnYNVicIaHQTq3TULC4H7/gcMMBsxALWc3gLQkGNzGpYX98W+wlmPMDEAt9TzGM85u/oFfC4OZNJKWwzwG/L3b8NoieYbHzBroF8mZEL8c55G4wbvNIsFAAqdf+IAOu527o44fGGLMH378qZbj7z+7+eaPCht5fmnsWrAACbBKCWKVgwD/AVJUj4JRMApGwQgAAGnqXVefREENAAAAAElFTkSuQmCC","orcid":"","institution":"AIIMS","correspondingAuthor":true,"prefix":"","firstName":"Sambit","middleName":"","lastName":"Sagar","suffix":""},{"id":511144632,"identity":"026edab2-39d9-40d2-bb12-86077bd155e2","order_by":4,"name":"Pooja Tiwari","email":"","orcid":"","institution":"AIIMS-CAPFIMS","correspondingAuthor":false,"prefix":"","firstName":"Pooja","middleName":"","lastName":"Tiwari","suffix":""},{"id":511144633,"identity":"4c3f91ff-7713-4352-bac1-f69b61d0e51c","order_by":5,"name":"Nivedita Kundu","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Nivedita","middleName":"","lastName":"Kundu","suffix":""},{"id":511144634,"identity":"01885eea-8cb8-4269-ad1f-a71cc72fd8c5","order_by":6,"name":"Arup Roy","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Arup","middleName":"","lastName":"Roy","suffix":""},{"id":511144635,"identity":"7dfeebb2-66d0-405e-af02-d903392a2a27","order_by":7,"name":"Lalitha Goriparthi","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Lalitha","middleName":"","lastName":"Goriparthi","suffix":""},{"id":511144636,"identity":"3aaca3d8-b472-480c-a5c9-8837141133f4","order_by":8,"name":"Jagatti Krishna","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Jagatti","middleName":"","lastName":"Krishna","suffix":""},{"id":511144637,"identity":"e23d88da-17c3-47b5-b4c6-9f632a60fd81","order_by":9,"name":"Aparna Mahalik","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Aparna","middleName":"","lastName":"Mahalik","suffix":""},{"id":511144638,"identity":"3e81a101-cf1e-4ba8-8b80-5eee0df0ace4","order_by":10,"name":"Yuvanesh Kabilan","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Yuvanesh","middleName":"","lastName":"Kabilan","suffix":""},{"id":511144639,"identity":"114babf9-b666-4a19-861d-545aa3ad3a5b","order_by":11,"name":"Ayan Dhiman","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Ayan","middleName":"","lastName":"Dhiman","suffix":""},{"id":511144640,"identity":"c7fe5efd-9975-4d87-932f-e622140748b1","order_by":12,"name":"Avnish Singh","email":"","orcid":"","institution":"All India Institute of Medical Sciences Bhubaneswar","correspondingAuthor":false,"prefix":"","firstName":"Avnish","middleName":"","lastName":"Singh","suffix":""},{"id":511144641,"identity":"f122d67b-04ec-4fcf-b8a3-8a0524d243d5","order_by":13,"name":"Amlan Rout","email":"","orcid":"","institution":"All India Institute of Medical Sciences Bhubaneswar","correspondingAuthor":false,"prefix":"","firstName":"Amlan","middleName":"","lastName":"Rout","suffix":""},{"id":511144642,"identity":"377fd7c8-4f30-4484-8f4e-183abed5c10b","order_by":14,"name":"Hemant Khairwa","email":"","orcid":"","institution":"AIIMS","correspondingAuthor":false,"prefix":"","firstName":"Hemant","middleName":"","lastName":"Khairwa","suffix":""}],"badges":[],"createdAt":"2025-09-06 05:23:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7548423/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7548423/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":90914154,"identity":"957ec197-a0ef-4ce8-9c9e-c9a29d941899","added_by":"auto","created_at":"2025-09-09 14:00:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":65839,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003ePerformance metrics by tool variant.\u003c/em\u003e\u003cbr\u003e\nComparison of DeDupli automatic vs manual modes against EndNote and Rayyan. DeDupli-Auto achieved faster results but with lower precision; DeDupli-Manual improved precision and F1 while maintaining perfect sensitivity.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7548423/v1/dae8b386241be56320586a86.png"},{"id":90912662,"identity":"657772b9-5285-4ded-bea7-61fcfd568149","added_by":"auto","created_at":"2025-09-09 13:52:07","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":32869,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eConfusion matrix totals by tool variant.\u003c/em\u003e\u003cbr\u003e\nStacked bar plot of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each tool variant (DeDupli-Auto, DeDupli-Manual, EndNote, and Rayyan). DeDupli variants eliminated all duplicates (FN = 0) but introduced some false positives, with the manual mode reducing FP compared to automatic. EndNote and Rayyan introduced no false positives but missed duplicates (FN \u0026gt; 0), highlighting their lower sensitivity compared to DeDupli.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-7548423/v1/679f813df37a55b96e2d8f5c.png"},{"id":90912654,"identity":"7b19ed5b-11fd-46e7-9265-69b7cc7bee00","added_by":"auto","created_at":"2025-09-09 13:52:07","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":59948,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eHeatmap of F1 scores by reviewer, dataset, and tool.\u003c/em\u003e\u003cbr\u003e\nEach cell represents the F1-score achieved for a reviewer–dataset–tool combination. Uniformly high values across all cells indicate consistent tool performance across reviewers and datasets.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-7548423/v1/c4a9f38f9e4f4b597a00c664.png"},{"id":91349945,"identity":"f57dd159-822f-40d9-ba5d-ce2f77ad09ba","added_by":"auto","created_at":"2025-09-15 14:24:10","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1566234,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7548423/v1/c1f7163d-122c-4342-acdf-af111307967a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eSpeed and Sensitivity in Deduplication: Evaluating DeDupli Against Established Tools for Systematic Reviews\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eSystematic reviews (SRs) and meta-analyses are cornerstones of evidence-based medicine, providing high-quality summaries of existing literature to guide clinical practice and policy decisions. A critical early step in conducting SRs involves retrieving citations from multiple bibliographic databases (e.g., MEDLINE, Embase, Cochrane CENTRAL), which often index overlapping records. This inevitably leads to the inclusion of duplicate references, increasing reviewer workload and potentially introducing bias if duplicates are not properly managed [1]. The process of identifying and removing duplicate records—\u003cstrong\u003ededuplication\u003c/strong\u003e—is therefore essential for both methodological rigor and efficiency [2].\u003c/p\u003e\n\u003cp\u003eTraditional reference management software (e.g., EndNote, Mendeley, Zotero) offers deduplication features, but comparative evaluations have shown that these tools are prone to both false negatives (missed duplicates) and false positives (unique records incorrectly removed) [3]. More recently, systematic review–specific platforms such as \u003cstrong\u003eRayyan\u003c/strong\u003e and \u003cstrong\u003eCovidence\u003c/strong\u003e have incorporated more advanced deduplication capabilities and generally outperform generic reference managers in both accuracy and usability [4].\u003c/p\u003e\n\u003cp\u003eEmpirical assessments demonstrate that tool performance varies, with important trade-offs between sensitivity and specificity. McKeown and Mir (2021) reported that Rayyan achieved higher sensitivity (i.e., fewer missed duplicates), whereas Covidence and Ovid tools had higher specificity [3]. A recent 2024 evaluation reaffirmed that SR-focused tools are superior overall to traditional reference managers, while also highlighting the importance of transparency and reproducibility in deduplication workflows [4].\u003c/p\u003e\n\u003cp\u003eNewer approaches leverage automation and machine learning to enhance performance. The \u003cem\u003eDeduklick\u003c/em\u003e algorithm, for example, integrates rule-based heuristics and natural language processing to achieve near-perfect precision and recall [5]. Similarly, the \u003cem\u003eSystematic Review Accelerator (SRA) Deduplicator\u003c/em\u003e module demonstrated superior accuracy and speed compared to EndNote, significantly reducing manual burden [2].\u003c/p\u003e\n\u003cp\u003eDespite these advances, few tools strike an optimal balance between \u003cstrong\u003espeed\u003c/strong\u003e, \u003cstrong\u003eaccuracy\u003c/strong\u003e, and \u003cstrong\u003euser oversight\u003c/strong\u003e, while also being accessible and privacy-friendly. In this study, we benchmark \u003cstrong\u003eDeDupli\u003c/strong\u003e, a novel Streamlit-based deduplication tool, against Rayyan and EndNote. We evaluate its \u003cstrong\u003eautomatic\u003c/strong\u003e and \u003cstrong\u003emanual\u003c/strong\u003e modes across multiple synthetic datasets, assessing sensitivity, specificity, precision, F1-score, and processing time.\u003c/p\u003e"},{"header":"Methods","content":"\u003ch3\u003e\u003cstrong\u003eDataset preparation\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eWe generated three synthetic bibliographic datasets in RIS format to simulate systematic review search outputs. Dataset A contained \u003cstrong\u003e262 records\u003c/strong\u003e, Dataset B contained \u003cstrong\u003e200 records\u003c/strong\u003e, and Dataset C contained\u0026nbsp;\u003cstrong\u003e320 records\u003c/strong\u003e. Each dataset was seeded with duplicate clusters to mimic real-world scenarios of imperfect bibliographic imports. Duplicate creation methods included punctuation changes, abbreviated author names, removal of DOIs, database source alterations, minor title variations, and year shifts.\u003cbr\u003eThe true duplicate structure was retained in a \u003cstrong\u003eground truth file\u003c/strong\u003e, where records sharing a cluster_id were considered duplicates and singleton clusters were considered unique references.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eReviewer assignments\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eThree independent reviewers were assigned datasets in a rotating design to balance tool use:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReviewer 1\u003c/strong\u003e evaluated Dataset A in Rayyan, Dataset B in EndNote, Dataset C in DeDupli (Auto + Manual modes).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReviewer 2\u003c/strong\u003e evaluated Dataset B in Rayyan, Dataset C in EndNote, Dataset A in DeDupli (Auto + Manual).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReviewer 3\u003c/strong\u003e evaluated Dataset C in Rayyan, Dataset A in EndNote, Dataset B in DeDupli (Auto + Manual).\u003c/p\u003e\n\u003cp\u003eThis ensured that each dataset was reviewed across all tools and that each reviewer interacted with all tools.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eTools compared\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRayyan\u003c/strong\u003e (Qatar Computing Research Institute) \u0026ndash; online screening tool with duplicate detection.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEndNote\u003c/strong\u003e (Clarivate Analytics) \u0026ndash; reference management software with semi-automated deduplication.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeDupli v1.1\u003c/strong\u003e (Dr Sambit Sagar \u0026nbsp;Dr Dikhra Khan) \u0026ndash; a privacy-friendly Streamlit application, tested in both \u003cstrong\u003eAutomatic\u003c/strong\u003e and \u003cstrong\u003eManual override\u003c/strong\u003e modes.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ehttps://dedupli-vd42ahhyjevixmtsaaqvut.streamlit.app/\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReviewer workflow\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eReviewers imported their assigned datasets into the allocated tool and ran the duplicate detection function. For each dataset-tool combination, reviewers reported \u003cstrong\u003eaggregate counts\u003c/strong\u003e only:\u003c/p\u003e\n\u003cp\u003eNumber of duplicates removed\u003c/p\u003e\n\u003cp\u003eNumber of unique references retained\u003c/p\u003e\n\u003cp\u003eTime taken (seconds)\u003c/p\u003e\n\u003cp\u003eNotes/comments\u003c/p\u003e\n\u003cp\u003eData entry was performed in a standardized Excel template. No per-record labeling was required, reducing reviewer burden.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eGround truth reference standard\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eTrue duplicate and unique counts were derived from the seeded cluster file.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTrue Positives (TP):\u003c/strong\u003e duplicates correctly removed\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTrue Negatives (TN):\u003c/strong\u003e uniques correctly retained\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFalse Positives (FP):\u003c/strong\u003e uniques wrongly removed\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFalse Negatives (FN):\u003c/strong\u003e duplicates missed\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eMetrics\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eFrom TP, TN, FP, FN we calculated:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSensitivity (recall)\u003c/strong\u003e = TP / (TP + FN)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSpecificity\u003c/strong\u003e = TN / (TN + FP)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrecision\u003c/strong\u003e = TP / (TP + FP)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eF1 score\u003c/strong\u003e = 2\u0026middot;TP / (2\u0026middot;TP + FP + FN)\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eStatistical analysis\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eAll analyses were performed in \u003cstrong\u003eR (version 4.4)\u003c/strong\u003e using tidyverse, readxl, and janitor.\u0026nbsp;\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eAcross three datasets (782 records total), each dataset was evaluated once in EndNote, once in Rayyan, and twice in DeDupli (automatic and manual modes), yielding a total of 3,128 record instances for analysis.Across three synthetic datasets (Dataset A = 262, Dataset B = 200, Dataset C = 320; total 782 unique records), ground-truth clustering identified 354 true unique references and 428 duplicate records belonging to duplicate groups.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOverall performance across tools\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll tools demonstrated high overall performance . \u003cstrong\u003eDeDupli (pooled across Auto and Manual modes)\u003c/strong\u003e achieved \u003cstrong\u003eperfect sensitivity (1.000)\u003c/strong\u003e, identifying and removing all true duplicates across datasets. However, this came at the cost of a small number of false positives (39 uniques incorrectly removed), reducing its specificity to \u003cstrong\u003e0.964\u003c/strong\u003e and precision to \u003cstrong\u003e0.916\u003c/strong\u003e, with a resulting F1 score of \u003cstrong\u003e0.956\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eBy contrast, \u003cstrong\u003eEndNote and Rayyan\u003c/strong\u003e demonstrated \u003cstrong\u003eperfect specificity and precision (1.000)\u003c/strong\u003e, never removing a true unique, but each failed to identify 13 duplicates, yielding lower sensitivity (\u003cstrong\u003e0.939\u003c/strong\u003e) despite similar F1 scores (\u003cstrong\u003e0.969\u003c/strong\u003e). Thus, EndNote and Rayyan maximized specificity, while DeDupli maximized sensitivity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTool variants: Auto vs Manual\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWhen evaluated separately (Table 1), \u003cstrong\u003eDeDupli-Auto\u003c/strong\u003e provided near-instantaneous deduplication (mean 5 s per dataset) and achieved perfect sensitivity, but generated a higher number of false positives (n = 24), resulting in lower precision (\u003cstrong\u003e0.899\u003c/strong\u003e) and F1 (\u003cstrong\u003e0.947\u003c/strong\u003e).\u003cbr\u003eThe \u003cstrong\u003eDeDupli-Manual\u003c/strong\u003e mode reduced false positives to 15, improving both precision (\u003cstrong\u003e0.934\u003c/strong\u003e) and F1 (\u003cstrong\u003e0.966\u003c/strong\u003e), with only modest additional processing time (mean 320 s). Importantly, both variants maintained perfect sensitivity, ensuring that no duplicate was missed \u0026mdash; a key advantage over the comparator tools.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eComparison of EndNote and Rayyan\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWhile the \u003cstrong\u003epooled totals for EndNote and Rayyan were identical\u003c/strong\u003e, per-dataset analyses revealed subtle differences. In Dataset A, Rayyan removed 71 duplicates (FN = 3) compared to EndNote\u0026rsquo;s 70 (FN = 4). In Dataset B, Rayyan again outperformed EndNote slightly (41 vs 40 duplicates removed). Conversely, in Dataset C, EndNote removed 91 duplicates (FN = 5), whereas Rayyan removed only 89 (FN = 7). These small differences balanced out in aggregate, yielding identical pooled sensitivity (0.939) and F1 (0.969).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReviewer-level consistency\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe reviewer-level analyses (Supplementary Appendix Table) showed consistent patterns across datasets and tools. Inter-reviewer variability was minimal: each reviewer reproduced the same strengths and weaknesses of the tools, suggesting that performance differences arose from the algorithms rather than user factors. Heatmaps of F1 scores confirmed uniformly high performance across reviewers, datasets, and tools.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProcessing time\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMarked differences were observed in processing time. \u003cstrong\u003eDeDupli-Auto\u003c/strong\u003e averaged less than \u003cstrong\u003e5 seconds per dataset\u003c/strong\u003e, while \u003cstrong\u003eDeDupli-Manual\u003c/strong\u003e averaged \u003cstrong\u003e320 seconds\u003c/strong\u003e. In contrast, both \u003cstrong\u003eEndNote (mean 2636 seconds; ~44 min)\u003c/strong\u003e and \u003cstrong\u003eRayyan (mean 2580 seconds; ~43 min)\u003c/strong\u003e required substantially longer times, reflecting their more cumbersome workflows and manual steps. This highlights DeDupli\u0026rsquo;s strength as a \u003cstrong\u003eprivacy-friendly and rapid deduplication solution\u003c/strong\u003e, with Auto mode optimized for speed and Manual mode offering a balance between speed and accuracy.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, we compared the performance of \u003cstrong\u003eDeDupli\u003c/strong\u003e, a novel privacy-friendly deduplication tool, against \u003cstrong\u003eEndNote\u003c/strong\u003e and \u003cstrong\u003eRayyan\u003c/strong\u003e using synthetic datasets with known ground truth. Across all datasets, \u003cstrong\u003eDeDupli achieved perfect sensitivity (1.000), successfully identifying and removing all duplicates without omission.\u003c/strong\u003e By contrast, EndNote and Rayyan maintained perfect specificity and precision but consistently missed duplicates, reducing sensitivity to 0.939 and leading to lower overall recall.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eComparison with prior work\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eOur findings are consistent with prior evaluations of deduplication tools. McKeown and Mir reported that Rayyan exhibited higher sensitivity than EndNote, whereas EndNote was more conservative and therefore less likely to introduce false positives [3]. In our dataset-level analysis, small differences between Rayyan and EndNote were also observed\u0026mdash;Rayyan identified slightly more duplicates in some datasets but missed more in others\u0026mdash;yet their pooled totals were identical. This highlights that sensitivity and specificity trade-offs remain a recurrent theme in deduplication research.\u003c/p\u003e\n\u003cp\u003eAutomation has been repeatedly shown to improve efficiency. The \u003cstrong\u003eSystematic Review Accelerator (SRA) Deduplicator\u003c/strong\u003e significantly reduced error rates and improved time efficiency compared to EndNote [2]. Similarly, the \u003cstrong\u003eDeduklick\u003c/strong\u003e algorithm demonstrated near-perfect recall and precision using a reproducible and explainable workflow [5]. Our evaluation confirms these trends: \u003cstrong\u003eDeDupli-Auto achieved perfect recall with processing times under 5 seconds per dataset\u003c/strong\u003e, markedly faster than EndNote (~44 minutes) and Rayyan (~43 minutes).\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eStrengths of DeDupli\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eUnlike many existing tools, DeDupli offers both \u003cstrong\u003eautomatic\u003c/strong\u003e and \u003cstrong\u003emanual modes\u003c/strong\u003e. The automatic mode provides unparalleled speed, making it well-suited for large-scale reviews where efficiency is critical. The manual mode, while slower (~5 minutes per dataset), enables reviewer oversight and reduced false positives, improving precision and F1-score compared to the automatic mode. This dual-mode design directly addresses concerns in systematic review methodology about balancing speed with transparency and control [7].\u003c/p\u003e\n\u003cp\u003eThe tool\u0026rsquo;s \u003cstrong\u003elocal, privacy-friendly implementation\u003c/strong\u003e further differentiates it from cloud-based platforms such as Rayyan, which may raise data security concerns in sensitive contexts. Combined, these features suggest that DeDupli can fill an important niche between traditional reference managers and more complex automated platforms.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eLimitations\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eThis evaluation was performed on \u003cstrong\u003esynthetic datasets\u003c/strong\u003e, carefully designed to emulate real-world duplicate structures. While this ensured a definitive ground truth, external validation on real bibliographic search outputs is still required. Our comparisons were limited to EndNote and Rayyan; additional benchmarking against other tools such as Covidence, Ovid, or EPPI-Reviewer would provide a more comprehensive landscape [8]. Finally, although DeDupli markedly reduced time burden, we did not formally assess user experience or integration with existing SR workflows, which remain important considerations.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eFuture directions\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eFuture development of DeDupli could explore integration of more advanced similarity measures, including fuzzy string matching and NLP-based algorithms, to further reduce false positives in the automatic mode. Adding functionality for \u003cstrong\u003emulti-user collaboration\u003c/strong\u003e and \u003cstrong\u003edirect integration with SR management platforms\u003c/strong\u003e would further enhance usability. In addition, future studies should examine how deduplication accuracy and speed translate into downstream improvements in screening efficiency, reviewer fatigue, and overall systematic review timelines.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn this evaluation, \u003cstrong\u003eDeDupli\u003c/strong\u003e demonstrated excellent performance compared with two widely used deduplication tools, \u003cstrong\u003eEndNote\u003c/strong\u003e and \u003cstrong\u003eRayyan\u003c/strong\u003e. By achieving \u003cstrong\u003eperfect sensitivity\u003c/strong\u003e across all datasets, DeDupli ensured that no duplicate was missed, while offering both an \u003cstrong\u003eautomatic mode\u003c/strong\u003e for rapid processing and a \u003cstrong\u003emanual mode\u003c/strong\u003e that improved precision by reducing false positives. Processing times were markedly shorter than those observed with EndNote and Rayyan, underscoring DeDupli\u0026rsquo;s efficiency advantage.\u003c/p\u003e\n\u003cp\u003eThese findings suggest that DeDupli offers a pragmatic balance between \u003cstrong\u003espeed, accuracy, and transparency\u003c/strong\u003e, addressing a long-standing methodological challenge in systematic review workflows. Its local, privacy-friendly design further enhances its suitability for sensitive or large-scale reviews. While external validation on real-world bibliographic datasets and broader comparisons with additional platforms are warranted, DeDupli represents a promising step toward \u003cstrong\u003estreamlining evidence synthesis\u003c/strong\u003e and reducing reviewer burden in systematic reviews and meta-analyses.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eTransparency and Ethical Requirements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDeclaration of funding\u003c/p\u003e\n\u003cp\u003eThis research received \u003cstrong\u003eno specific grant from any funding agency in the public, commercial, or not-for-profit sectors\u003c/strong\u003e. The development and evaluation of the DeDupli tool were undertaken as part of the authors\u0026rsquo; academic and professional activities without external sponsorship.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclaration of financial/other relationships\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors declare that they have \u003cstrong\u003eno financial, employment, or other relationships\u003c/strong\u003e that could be construed as a conflict of interest with respect to this study.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo assistance in the preparation of this article is to be declared.\u003cbr\u003e\u0026nbsp;This work has not been previously published or presented in any conference proceedings.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eConception and design of the study: Sambit Sagar, Satchit Sagar, Dikhra Khan\u003c/li\u003e\n \u003cli\u003eDevelopment of DeDupli tool: Sambit Sagar, Satchit Sagar, Dikhra Khan\u003c/li\u003e\n \u003cli\u003eData curation and ground truth preparation: Pooja Tiwari, Nivedita Kundu, Arup Roy, Lalitha Goriparthi, Jagatti Krishna, Aparna Mahalik\u003c/li\u003e\n \u003cli\u003ePerformance evaluation and analysis: Dikhra Khan, Amlan Rout, Hemant Khairwa, Rajashekhar CH T\u003c/li\u003e\n \u003cli\u003eDrafting of the manuscript: Dikhra Khan, Sambit Sagar, Satchit Sagar\u003c/li\u003e\n \u003cli\u003eCritical revision of the manuscript for important intellectual content: Arup Roy, Lalitha Goriparthi, Nivedita Kundu, Aparna Mahalik, Jagatti Krishna, Avnish Singh, Amlan Rout\u003c/li\u003e\n \u003cli\u003eFinal approval of the version to be published: All authors\u003c/li\u003e\n \u003cli\u003eAccountability: All authors agree to be accountable for all aspects of the work, ensuring accuracy and integrity.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cbr\u003e\u0026nbsp;\u003cstrong\u003eData availability statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data that support the findings of this study are available from the corresponding author, [Dr Sambit Sagar], upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eRathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module. \u003cem\u003eSystematic Reviews\u003c/em\u003e. 2015;4:6. doi:10.1186/2046-4053-4-6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-6\u003c/li\u003e\n\u003cli\u003eForbes C, Greenwood H, Carter M, Clark J. Automation of duplicate record detection for systematic reviews: Deduplicator. \u003cem\u003eSystematic Reviews\u003c/em\u003e. 2024;13(1):206. doi:10.1186/s13643-024-02619-9. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-024-02619-9\u003c/li\u003e\n\u003cli\u003eMcKeown S, Mir ZM. Comparison of methods for identifying duplicate records in systematic reviews. \u003cem\u003eSystematic Reviews\u003c/em\u003e. 2021;10(1):150. doi:10.1186/s13643-021-01583-y. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01583-y\u003c/li\u003e\n\u003cli\u003eJanka H, Metzendorf M-I. High precision but variable recall \u0026ndash; comparing the performance of five deduplication tools. \u003cem\u003eJournal of the European Association for Health Information and Libraries (JEAHIL)\u003c/em\u003e. 2024;20(1):12\u0026ndash;17. doi:10.32384/jeahil20607. Available from: https://ojs.eahil.eu/JEAHIL/article/view/607\u003c/li\u003e\n\u003cli\u003eBorissov N, Haas Q, Minder B, Kopp-Heim D, von Gernler M, Janka H, et al. Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. \u003cem\u003eSystematic Reviews\u003c/em\u003e. 2022;11(1):172. doi:10.1186/s13643-022-02045-9. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-022-02045-9\u003c/li\u003e\n\u003cli\u003eRathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. \u003cem\u003eSyst Rev\u003c/em\u003e. 2015;4:80. doi:10.1186/s13643-015-0067-6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-015-0067-6\u003c/li\u003e\n\u003cli\u003eJanka H, Metzendorf M-I. High precision but variable recall \u0026ndash; comparing the performance of five deduplication tools. \u003cem\u003eJ Eur Assoc Health Inf Libr\u003c/em\u003e. 2024;20(1):12\u0026ndash;17. doi:10.32384/jeahil20607. Available from: https://ojs.eahil.eu/JEAHIL/article/view/607\u003c/li\u003e\n\u003cli\u003evan der Mierden S, Tseng YJ, de Bruin J, Hooft L. Computer-assisted screening of records in systematic reviews: a systematic review. \u003cem\u003eSyst Rev\u003c/em\u003e. 2019;8:236. doi:10.1186/s13643-019-1175-7. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1175-7\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Table","content":"\u003cp\u003e\u003cstrong\u003eTable 1:\u0026nbsp;\u003c/strong\u003e\u003cem\u003ePerformance by tool variant\u003c/em\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"570\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTool Variant\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eN\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTP\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTN\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 22px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eFP\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 23px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eFN\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSensitivity\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSpecificity\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 65px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePrecision\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eF1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMean Time (s)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eDeDupli-Auto\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e782\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e214\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e520\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 22px;\"\u003e\n \u003cp\u003e24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 23px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e0.956\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 65px;\"\u003e\n \u003cp\u003e0.899\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003e0.947\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e5\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eDeDupli-Manual\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e782\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e214\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e538\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 22px;\"\u003e\n \u003cp\u003e15\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 23px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e0.973\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 65px;\"\u003e\n \u003cp\u003e0.934\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.966\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 93px;\"\u003e\n \u003cp\u003e320\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eEndNote\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e782\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e201\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e568\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 22px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 23px;\"\u003e\n \u003cp\u003e13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e0.939\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 65px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.969\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 93px;\"\u003e\n \u003cp\u003e2636\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eRayyan\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e782\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e201\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e568\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 22px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 23px;\"\u003e\n \u003cp\u003e13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e0.939\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 73px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 65px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.000\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.969\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 93px;\"\u003e\n \u003cp\u003e2580\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Systematic Review, Meta Analysis, Deduplication, Automatic deduplication","lastPublishedDoi":"10.21203/rs.3.rs-7548423/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7548423/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDuplicate records are a common challenge in systematic reviews, arising from database overlap and multiple citation formats. Failure to address duplicates increases reviewer workload and risks bias. Existing tools such as EndNote and Rayyan offer deduplication features, but evaluations highlight trade-offs between sensitivity and specificity. \u003cstrong\u003eDeDupli\u003c/strong\u003e, a novel Streamlit-based, privacy-friendly tool, was developed to balance speed, accuracy, and user oversight through automatic and manual deduplication modes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe generated three synthetic RIS datasets (262, 200, and 320 records) with known duplicate clusters to simulate real-world bibliographic search outputs. Three independent reviewers evaluated deduplication performance across \u003cstrong\u003eEndNote, Rayyan, and DeDupli (automatic and manual modes)\u003c/strong\u003e in a crossover design. Reviewers recorded duplicates removed, uniques retained, and processing time. Ground truth was used to calculate true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Performance metrics included sensitivity, specificity, precision, and F1-score.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAcross 1,564 records with 428 true duplicates, \u003cstrong\u003eDeDupli achieved perfect sensitivity (1.000)\u003c/strong\u003e, eliminating all duplicates. EndNote and Rayyan achieved perfect specificity and precision (1.000), but each missed 13 duplicates (sensitivity 0.939). \u003cstrong\u003eDeDupli-Auto\u003c/strong\u003eprocessed datasets in under 5 seconds but introduced more false positives (precision 0.899, F1 0.947). \u003cstrong\u003eDeDupli-Manual\u003c/strong\u003erequired ~5 minutes, reduced false positives, and improved precision (0.934) and F1 (0.966). By contrast, EndNote (2636 s) and Rayyan (2580 s) required ~40 minutes per dataset. Reviewer-level analysis showed minimal variability, confirming consistency across raters.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeDupli\u003c/strong\u003e demonstrated superior sensitivity and substantial time savings compared with EndNote and Rayyan. Its dual-mode design offers flexibility, allowing users to prioritize either speed (automatic) or accuracy (manual). These results highlight DeDupli as a promising, privacy-friendly solution to streamline deduplication in systematic reviews and meta-analyses. Validation on real-world datasets and integration into review workflows are recommended for future work.\u003c/p\u003e","manuscriptTitle":"Speed and Sensitivity in Deduplication: Evaluating DeDupli Against Established Tools for Systematic Reviews","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-09 13:52:02","doi":"10.21203/rs.3.rs-7548423/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b0f0fa90-adea-40e0-8df0-bb3f61cf6fc6","owner":[],"postedDate":"September 9th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-15T14:24:01+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-09 13:52:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7548423","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7548423","identity":"rs-7548423","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}