Accelerating metadata annotation in collaborative research centers: A hybrid AI workflow for biomedical entities

Accelerating metadata annotation in collaborative research centers: A hybrid AI workflow for biomedical entities | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Accelerating metadata annotation in collaborative research centers: A hybrid AI workflow for biomedical entities Manuel Watter, Felix Engel, Aref Kalantari, Claudia Giuliani, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9231981/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Collaborative Research Centers rely on FAIR-compliant, richly structured metadata, yet manual annotation is a major bottleneck. We implemented an AI- and search-augmented large language model (LLM) workflow within a local research data management system to pre-annotate biomedical entities, using human-in-the-loop verification to ensure data quality. Methods The pipeline uses Gemini 3.0 Pro for a two-step prompting strategy: (1) identify dataset deposits and stable identifiers in articles converted to Markdown; (2) extract structured fields from curated repository landing pages rendered via a headless browser. To respect a highly hierarchical metadata schema, we flattened the schema for prompting and remapped outputs to strict JSON, with granular provenance tags. Authors received pre-filled metadata and could accept, edit, or delete entries (TP, FP, FN mapping). Performance metrics (precision, recall, F1) were estimated as proportions and synthesized via random-effects meta-analysis. The workflow was rolled out in December 2025 with reminders at 5 and 10 weeks. Results Among 51 screened articles (40 original articles, 11 review articles), the LLM identified a repository deposit in 31 articles; authors responded for 17 of these (55%), yielding 39 datasets with human verification. On the 39 verified datasets, the number of true positives averaged 13.15 (SD 4.57; range 6–27). False positives were rare, with a mean of 0.23 (SD 0.58; range 0–2). False negatives were also low, with a mean of 1.46 (SD 1.93; range 0–6). Precision was consistently high across datasets with an overall random-effects estimate of 99.65% (95% CI 98.42% to 100.00%) and no detectable heterogeneity (I² = 0.00%). Recall showed more variability, with an overall estimate of 93.75% (95% CI 89.79% to 96.96%) and moderate heterogeneity (I² = 55.08%). The combined performance, expressed as the F1 score, yielded an overall estimate of 96.17% (95% CI 93.78% to 98.11%). Conclusions The hybrid workflow achieved very high precision with moderately variable recall, effectively shifting effort from drafting to reviewing while preserving schema compliance. However, the modest author response rate limits sample size and generalizability; broader engagement and multi-site validation are needed to confirm robustness across domains. Figures Figure 1 Figure 2 Figure 3 Introduction Long-term research hubs like Germany's Collaborative Research Centers (CRCs) rely on structured data sharing for FAIR-compliant reuse [ 1 , 2 ]. This necessity has intensified due to the massive growth of digital information, often termed a 'data deluge', which has shifted research data management from an optional task to a core scientific requirement [ 3 ]. In biomedicine, tagging entities such as organisms or genes boosts discoverability. Since manual annotation requires considerable effort, automated named entity recognition is often used to classify terms (for instance, identifying "mouse" as an "organism") [ 4 ]. Because each CRC has distinct scientific requirements, a dedicated metadata schema must first be developed to reflect its specific experimental context [ 3 ]. The specific targets depend on the experimental focus. Cell lines matter for in-vitro assays, but they're irrelevant in population studies. Only once this schema is in place can researchers begin manually annotating their datasets. This 'metadata bottleneck' frequently leads to the reality of 'empty archives,' as researchers often prioritize immediate research demands over labor-intensive documentation [ 3 , 5 ]. While research data management is indispensable, it should be designed to be as effortless and unobtrusive as possible, enabling researchers to concentrate on their scientific work. We outline the implementation of the AI-augmented workflow, into an existing research data management system [ 5 ]. The AI system is an LLM-based, search-augmented pipeline designed to identify and annotate biomedical entities, particularly focusing on handling multiple datasets across different public repositories. This workflow uses a two-step strategy that first identifies datasets, then extracts data primarily from curated repository landing pages as the source of truth. The target environment, fredato, is a standardized metadata collection system built as a wrapper over GitLab and Nextcloud [ 5 ], utilized by Collaborative Research Centers. This existing system already relies on a specialized, hierarchical metadata schema defined using a bottom-up approach with domain experts [ 6 , 7 ]. This strategy balances the need for consortium-wide standardization with the high granularity required for specialized biomedical data, such as disease models and readouts [ 3 , 5 ]. We aim to implement a hybrid model [ 8 ], where LLMs accelerate the data documentation process [ 5 , 8 , 9 ]. By deploying the model as a pre-annotation assistant, the system overcomes the 'blank slate problem' and significantly reduces 'creation fatigue' for professional researchers [ 10 ]. To be effective, the AI workflow must adhere to the existing schema's logic [ 7 ], while maintaining data quality through human validation [ 8 , 9 ]. Methods The implementation of an AI-augmented metadata annotation workflow is based on the search-augmented LLM pipeline that was tested previously [ 7 ], and the existing metadata annotation framework within the fredato research data management system [ 5 ]. The study follows a human-in-the-loop approach (see Appendix S1 for the detailed operational workflow), where artificial intelligence functions as a pre-annotation assistant to accelerate traditionally labor-intensive documentation processes while remaining under the epistemic oversight of the responsible scientist [ 8 , 10 ]. The pipeline uses Gemini 3.0 Pro to identify and annotate biomedical entities from public repositories via a two-step prompting strategy. To enable accurate text processing, PDF manuscripts were first converted into Markdown format (using Mistral OCR or Docling). First, the model scans the formatted text and references to identify dataset deposits and capture stable identifiers such as GEO or SRA URLs. In the second step, the model visits the curated repository landing pages, treated as the primary source of truth. Retrieval was performed using a headless browser (Playwright) to render dynamic JavaScript content. The extracted HTML was programmatically cleaned to remove boilerplate elements (e.g., navigation bars, styling) and truncated to fit within the model's context window (detailed cleaning specifications and fallback mechanisms are described in Appendix S2) to populate fields in the specialized, hierarchical metadata schema. The AI-generated suggestions are then integrated into fredato, a web-based metadata editor built as a wrapper over GitLab and Nextcloud that utilizes JSON schema definitions. To ensure data integrity and distinguish between human and machine inputs, the system implements a granular provenance tracking mechanism that tags individual metadata objects with their origin and validation status (see Appendix S4). To prevent context poisoning caused by the schema's complexity, we employed a schema flattening strategy. Instead of feeding the full JSON structure to the LLM, the schema was converted into a tree-like textual representation using short identifiers (see Appendix S3). The model’s output was then mapped back to the strict JSON format. This implementation respects the hierarchical and conditional logic of the domain-specific metadata schema [ 6 ], where specific entries (e.g., selecting a particular "tissue source") trigger the requirement for refined sub-fields. Within the interface, authors are presented with pre-filled metadata fields. In an email-based interactive validation loop, the model’s outputs are presented to the authors of the articles as editable suggestions rather than final records. Authors act as the "human in the loop," reviewing the suggestions with the ability to accept, modify, or delete entries. This step ensures the final data remains accurate, allowing scientists to correct potential model hallucinations or generalizations [ 8 , 10 ]. The primary objective of the evaluation is to validate the LLM's performance against high-quality labels generated by subject matter experts (the authors) during the validation phase. By tracking how authors interact with the pre-filled forms, we derive effect measures based on specific user actions. We mapped user actions to performance metrics: accepted suggestions count as True Positives (TP), deleted suggestions as False Positives (FP), manually added entities as False Negatives (FN). Based on these interactions, the following performance metrics are calculated: Precision Proportion of accepted suggestions. \(\:\:Precision=\frac{TP}{TP+FP}\) ​ Recall Ability to identify required entities. \(\:\:Recall=\frac{TP}{TP+FN}\) ​ F1 Score Harmonic mean of precision and recall. \(\:F1=\frac{2*TP}{2*TP+FN+FP}\) ​ This methodology allows the study to overcome the typical "open-ended" limitation of recall measurements in scientific annotation by providing a structured environment where human experts act as the "gold standard" for every required field [ 7 , 11 ]. Because the evaluation is performed over a highly hierarchical metadata schema (342 levels, see [ 3 , 7 ]), most fields are correctly left blank, yielding a very large number of true negatives. Metrics that heavily incorporate true negatives (such as accuracy) are therefore not reported, as they are less informative for the intended decision context. To evaluate the effectiveness of the AI-augmented workflow, the three performance metrics, Precision, Recall, and F1 Score are analyzed using the statistical methodology used in previous approaches [ 7 , 12 , 13 ]. In detail, these metrics are treated as proportions derived from the comparison between the initial AI-generated suggestions and the final "gold standard" labels validated by the authors within the fredato system. To estimate the overall performance across the different research articles and datasets, meta-analyses of single proportions are conducted [ 14 ]. Due to the expected small sample sizes per article and the likelihood of proportions being near 1, the Freeman–Tukey double arcsine transformation is applied to stabilize the variance [ 15 ]. A random-effects model using the restricted maximum likelihood (REML) estimator is employed to calculate aggregate estimates and their corresponding 95% confidence intervals, accounting for potential variability between different scientific domains or metadata tasks. Results The workflow was rolled out in December 2025, with reminder emails sent at 5 weeks and 10 weeks after rollout. Figure 1 summarizes the yield of the multi-step workflow across 51 screened articles (40 original articles and 11 review articles). A repository deposit was identified by the LLM for 31 articles; for 17 of these articles, authors provided feedback, enabling human-in-the-loop verification of the LLM suggestions for 39 datasets. For the remaining pathway, 9 articles had no repository identified by the LLM; in this branch, only one author provided additional information for 1 unpublished dataset; however, from four articles in the repository pathway, 11 unpublished datasets were added on top of the datasets already published in public repositories. No human verification was obtained at the time of writing for a total of 22 articles (Fig. 1 ). Performance evaluation was based on the 39 datasets with human verification. Across these datasets, the number of true positives (TP) averaged 13.15 (SD 4.57; range 6–27). False positives (FP) were rare, with a mean of 0.23 (SD 0.58; range 0–2). False negatives (FN) were also low, with a mean of 1.46 (SD 1.93; range 0–6). Precision was consistently high across datasets (mostly 100%), with an overall random-effects estimate of 99.65% (95% CI 98.42% to 100.00%) and no detectable heterogeneity (I² = 0.00%; Fig. 2 ). Recall showed more variability, with an overall estimate of 93.75% (95% CI 89.79% to 96.96%) and moderate heterogeneity (I² = 55.08%; Fig. 3 ). The combined performance, expressed as the F1 score, yielded an overall estimate of 96.17% (95% CI 93.78% to 98.11%) with moderate heterogeneity (I² = 61.16%; Supplemental Fig. 1). Discussion In the initial evaluation of the LLM suggestions for 39 distinct datasets, the workflow maintained robust performance comparable to the exploratory phase. Preliminary analysis of user interactions reveals that the Gemini 3.0 Pro model achieved an aggregate precision of approximately 99.65%, meaning authors accepted the vast majority of pre-filled fields without modification. By contrast, the recall was lower at roughly 93.75%. This gap reflects instances where the model missed entities that weren't explicitly listed on the repository landing page, requiring authors to manually add specific details. Therefore, while the high precision ensured that authors rarely had to delete incorrect suggestions (false positives), the recall metrics show that human expertise is still necessary to catch missing information (false negatives). Ultimately, the workflow successfully shifted the user's role from drafting to reviewing, significantly reducing the time required for metadata documentation while ensuring high schema compliance. Integrating this AI workflow into fredato illustrates the practical benefits of a hybrid approach. By pre-filling identifiers and entity tags, the model addresses the "blank slate" issue, reducing the workload for researchers authoring multiple records [ 10 ]. Crucially, the workflow maintains data integrity by requiring scientists to review the output, providing a checkpoint to correct errors or nuances the model may miss [ 7 , 9 , 10 ]. This approach balances automation with expert verification, compressing handling time while capturing necessary corrections [ 10 ]. The technically forced human-in-the-loop design not only represents a significant step in addressing the "metadata bottleneck" [ 3 , 16 ], but also follows established frameworks for responsible automation, positioning artificial intelligence as a context-sensitive amplifier intended to support rather than supplant human expertise [ 9 , 16 ]. Such structured interactions are critical safeguards against model hallucinations, ensuring that high-stakes research data remain trustworthy and auditable [ 16 , 17 ]. The use of a highly granular, hierarchical metadata schema is instrumental to this process, as it acts as a semantic backbone that constrains model inference through typed fields and controlled vocabularies [ 3 , 17 ]. Our results support the paradigm shift where schemas evolve from static data entry forms into high-dimensional " instruction sets " for agentic workflows [ 3 ]. In this model, the AI does not merely extract text but actively validates its decisions against schema constraints in real-time [Engel]. Furthermore, reformatting complex metadata dictionaries into structured JSON schemas has been shown to significantly improve a model's ability to correctly process and extract numerous features simultaneously [ 18 ]. These schema-based predictions allow for high-accuracy concept extraction that was previously unfeasible in purely manual routines [ 19 ]. Regarding model selection, while the debate between general-purpose and specialized LLMs remains nuanced, our findings suggest that generalist models like Gemini, when properly grounded through appropriate data, can achieve performance comparable to trained human annotators [Balasubramanian Wood]. While domain-specific fine-tuning often yields superior accuracy on niche terminology, generalist foundation models inherit strong generalization capabilities that are frequently lost in over-specialized architectures [ 18 , 20 ] The high precision (99.65%) achieved in our pilot is vital for knowledge discovery, as it minimizes the risk of misleading hypotheses [ 21 ]. However, the moderate recall (93.75%) highlights a persistent challenge where models may overlook entities in documents with high information density, reinforcing the necessity for expert post-validation to capture missing entries [ 10 , 11 , 21 ]. Despite these promising results, significant limitations exist regarding the isolated situation within a CRC. The schemas utilized are often highly granular but specific to individual research centers, which can lead to limited vocabulary overlap and potential friction when datasets deviate from the consortium's primary focus [ 3 , 7 ]. Such isolation runs the risk of creating siloed datasets that remain difficult to harmonize across broader institutional networks. Finally, the external validity of this pilot is constrained by its relatively small and homogeneous sample size [ 17 , 22 ]. To ensure the long-term utility and safe adoption of these AI-augmented routines, broader validation across multi-site cohorts and more diverse scientific domains is required to account for the heterogeneous reporting practices found across the wider biomedical literature [ 17 , 23 ]. One of the primary concerns identified in recent pilots is the non-deterministic nature of Large Language Models (LLMs), which produce variable outputs that are highly version-dependent [ 7 , 12 , 13 ]. This lack of deterministic consistency makes exact replication of metadata extraction results difficult unless specific model versions and prompts are strictly pinned [ 7 , 12 ]. Additionally, the "black box" phenomenon of AI reasoning continues to impact trust; without transparent and explainable decision pathways, expert clinicians and researchers may be hesitant to rely on AI-generated metadata for high-stakes evidence synthesis [ 8 , 24 , 25 ]. Establishing robust post-deployment monitoring is therefore essential to detect model drift and ensure that the AI continues to perform safely and accurately as documentation practices evolve [ 23 ]. Declarations Acknowledgements The article processing charge is funded by the German Research Foundation (DFG) and the Albert Ludwig University of Freiburg in the funding program Open Access Publishing. Funding This work was supported by the DFG (German Research Foundation) through the following projects: Project-ID 441891347 – SFB 1479, Project-ID 499552394 – SFB 1597, Project-ID 491676693 – TRR 359, Project-ID 256073931 – SFB 1160, Project-ID 514483642 – TRR 384, Project-ID 259373024 – TRR 167, Project-ID 431984000 – SFB 1453 and EXC 2189. The funding body has had no role in the design of the study or collection, analysis, or interpretation of data or in writing the manuscript. Authors’ contributions KK and MW designed the study. MW developed the software. KK and MW analyzed the data and wrote the manuscript. All authors (MW, FE, AK, CG, KS, SWF, MS, HB and KK) contributed to interpretation of the data, read and approved the final version of the manuscript. Ethics approval and consent to participate This study did not involve human subjects research as no identifiable personal data were collected or processed. The analyses were based solely on aggregated usage data of a software tool. Therefore, ethical approval and informed consent were not required according to applicable institutional and national regulations. Consent for publication Not applicable. Competing interests The authors declare that they have no competing interests. Availability of data and materials The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. References Wilkinson MD, Dumontier M, Aalbersberg IjJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:1–9. Kanza S, Knight NJ. Behind every great research project is great data management. BMC Res Notes. 2022;15:20. https://doi.org/10.1186/s13104-022-05908-5 . Engel F, Giuliani C, Watter M, Kalantari A, Schuller K, Binder H, et al. From manual entry to machine precision: challenges and evolution of metadata schema development in collaborative research centers. 2026. ResearchSquare. https://doi.org/10.21203/rs.3.rs-8711752/v1 Nadeau D, Sekine S. A survey of named entity recognition and classification. In: Sekine S, Ranchhod E, editors. Benjamins Current Topics. Amsterdam: John Benjamins Publishing Company; 2009. p. 3–28. https://doi.org/10.1075/bct.19.03nad . Watter M, Kahle L, Brunswiek B, Fichtner U, Pfaffenlehner M, Werner F, et al. Standardized metadata collection to reinforce collaboration in Collaborative Research Centers. E-Sci-Tage 2023 Empower Your Res - Preserve Your Data. 2023; https://doi.org/10.11588/heidok.00033131 . Engel F, Benadi G, Giuliani C, Werner J, Watter M, Zeiser R, et al. Development of Metadata Schemas For Collaborative Research Centers. FreiData. 2025; https://doi.org/10.60493/k1xe3-npc10 . https://doi.org/10.60493/k1xe3-npc10. Kaier K, Engel F, Benadi G, Giuliani C, Watter M, Kalantari A, et al. Identification of biomedical entities from multiple repositories using a specialized metadata schema and search-augmented large language models. BMC Res Notes. 2026;19:43. Anthuvan T, Prabhuram S, Maheshwari K, Rathi S. SLR AI : A Systematic Review and Methodological Framework for AI-Augmented Evidence Synthesis with Human-in-the-Loop Integration. Available SSRN 5394910. 2025. Reddy S. Generative AI in healthcare: an implementation science informed translational path on application, integration and governance. Implement Sci. 2024;19:27. https://doi.org/10.1186/s13012-024-01357-9 . Wang Y, Stevens D, Shah P, Jiang W, Liu M, Chen X, et al. Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs. 2024. https://doi.org/10.48550/arXiv.2409.10702 . Pangakis N, Wolken S, Fasching N. Automated Annotation with Generative AI Requires Validation. 2023. https://doi.org/10.48550/arXiv.2306.00176 . Watter M, Giuliani C, Benadi G, Engel F, Binder H, Kaier K. Automated Identification of Contextually Relevant Biomedical Entities with Grounded LLMs. Sci Rep. 2026;16:1952. https://doi.org/10.1038/s41598-026-35492-8 . Giuliani C, Benadi G, Engel F, Werner J, Watter M, Schwarzer G, et al. Identifying Biomedical Entities for Datasets in Scientific Articles: 4-Step Cache-Augmented Generation Approach Using GPT-4o and PubTator 3.0. JMIR Form Res. 2025;9:e73822. https://doi.org/10.2196/73822 . Schwarzer G, Rücker G. Meta-Analysis of Proportions. In: Evangelou E, Veroniki AA, editors. Meta-Research. New York, NY: Springer US; 2022. p. 159–72. https://doi.org/10.1007/978-1-0716-1566-9_10 . Miller JJ. The Inverse of the Freeman – Tukey Double Arcsine Transformation. Am Stat. 1978;32:138–138. https://doi.org/10.1080/00031305.1978.10479283 . Wood V, Jeffryes M, Green AF, Blum M, Orchard S, Panni S, et al. Empowering biological knowledgebases: advances in human-in-the-loop AI-driven literature curation. Bioinforma Adv. 2026;6. Mortezaagha P, Shaw J, Sun B, Rahgozar A. From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs. 2025. https://doi.org/10.48550/arXiv.2601.14267 . Balasubramanian JB, Adams D, Roxanis I, de Gonzalez AB, Coulson P, Almeida JS, et al. Leveraging large language models for structured information extraction from pathology reports. J Pathol Inform. 2025;:100521. Ikeda S, Zou Z, Bono H, Moriya Y, Kawashima S, Katayama T, et al. Extraction of biological terms using large language models enhances the usability of metadata in the BioSample database. GigaScience. 2025;14:giaf070. Verbitsky A, Boutet P, Eslami M. Metadata harmonization from biological datasets with language models. Bioinforma Adv. 2025;5:vbaf241. Brokman A, Ai X, Jiang Y, Gupta S, Kavuluru R. A benchmark for end-to-end zero-shot biomedical relation extraction with LLMs: experiments with OpenAI models. In: Proceedings of the Third Workshop for Artificial Intelligence for Scientific Publications. 2025. p. 44–55. Jabal MS, Warman P, Zhang J, Gupta K, Jain A, Mazurowski M, et al. Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports. 2024. https://doi.org/10.48550/arXiv.2409.10576 . Artsi Y, Sorin V, Glicksberg BS, Korfiatis P, Nadkarni GN, Klang E. Large language models in real-world clinical workflows: a systematic review of applications and implementation. Front Digit Health. 2025;7:1659134. Hassan M, Kushniruk A, Borycki E. Barriers to and facilitators of artificial intelligence adoption in health care: scoping review. JMIR Hum Factors. 2024;11:e48633. Trinkley KE, An R, Maw AM, Glasgow RE, Brownson RC. Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions. Implement Sci. 2024;19:17. https://doi.org/10.1186/s13012-024-01346-y . Additional Declarations No competing interests reported. Supplementary Files SupplementalFigure1.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9231981","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":613204490,"identity":"41917d5f-c8a2-41c8-999c-69a98956502e","order_by":0,"name":"Manuel Watter","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Manuel","middleName":"","lastName":"Watter","suffix":""},{"id":613204491,"identity":"333ccaae-2426-4c09-bf0d-dea7e3905089","order_by":1,"name":"Felix Engel","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Felix","middleName":"","lastName":"Engel","suffix":""},{"id":613204492,"identity":"38a1768f-0067-4c21-9df2-e043b96f2931","order_by":2,"name":"Aref Kalantari","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Aref","middleName":"","lastName":"Kalantari","suffix":""},{"id":613204493,"identity":"a9af4709-cc1d-476e-92f3-4635a6f531b7","order_by":3,"name":"Claudia Giuliani","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Claudia","middleName":"","lastName":"Giuliani","suffix":""},{"id":613204494,"identity":"f0fdf51a-31b2-4dd2-9834-8359c3fe63df","order_by":4,"name":"Karin Schuller","email":"","orcid":"","institution":"Helmholtz München","correspondingAuthor":false,"prefix":"","firstName":"Karin","middleName":"","lastName":"Schuller","suffix":""},{"id":613204495,"identity":"0041eea9-08b6-418f-8f1d-63b5aeb5e392","order_by":5,"name":"Claus-Werner Franzke","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Claus-Werner","middleName":"","lastName":"Franzke","suffix":""},{"id":613204496,"identity":"1fd80d75-43bc-4dd0-8eb2-d7144b80cba8","order_by":6,"name":"Markus Sperandio","email":"","orcid":"","institution":"Ludwig-Maximilians- Universität München","correspondingAuthor":false,"prefix":"","firstName":"Markus","middleName":"","lastName":"Sperandio","suffix":""},{"id":613204497,"identity":"96c95b72-6be7-48e3-850b-adc29b103b4d","order_by":7,"name":"Harald Binder","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Harald","middleName":"","lastName":"Binder","suffix":""},{"id":613204498,"identity":"9929ea68-64c4-407b-a5fa-36cd5c419ad8","order_by":8,"name":"Klaus Kaier","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIiWNgGAWjYBACCXYwlcDAxsDA+ADI4uEjqIUZoYXZAKSFjWgtQMAmASYJaZFsZn4m+YMhTZ6PvfdZ5dccOxk2BuaHj27g0SLNzGYmzcOQY9jGc9zstuy2ZKDD2IyNc/BokWNmMJNmYKhgbJNIY7stuY0ZqIWHTRq/FvZvQIdV2LfJP2MrltxWT1iLNDOPmQTQYYltEmxsjB+3HSasRbKZp9iaxyAtuY0njVmacdtxHjZmAn6RON6+8eaPimTb+e3HGD/+3FZtz8/e/PAxPi0QYAChmHnAJEHlSIDxBymqR8EoGAWjYMQAAOxXNnGeGPrtAAAAAElFTkSuQmCC","orcid":"","institution":"University of Freiburg","correspondingAuthor":true,"prefix":"","firstName":"Klaus","middleName":"","lastName":"Kaier","suffix":""}],"badges":[],"createdAt":"2026-03-26 09:23:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9231981/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9231981/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105729266,"identity":"14d37e9a-3eea-4ca0-b349-b4e1cb42c6f0","added_by":"auto","created_at":"2026-03-30 11:14:05","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":151811,"visible":true,"origin":"","legend":"\u003cp\u003eFlow chart of the multi-step LLM workflow\u003c/p\u003e","description":"","filename":"image1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-9231981/v1/3049d5ea51ca57cb1545799d.jpeg"},{"id":105695232,"identity":"a757beed-5b89-4f27-9572-de8f43f67d4e","added_by":"auto","created_at":"2026-03-30 03:44:52","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1308435,"visible":true,"origin":"","legend":"\u003cp\u003ePrecision of the multi-step LLM workflow according to the human-in-the-loop verification of LLM suggestions\u003c/p\u003e","description":"","filename":"image2.png","url":"https://assets-eu.researchsquare.com/files/rs-9231981/v1/711cb800fd02ffd59b83390a.png"},{"id":105695230,"identity":"21645b1e-5afc-4df5-b902-bb62b7323e78","added_by":"auto","created_at":"2026-03-30 03:44:52","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1287103,"visible":true,"origin":"","legend":"\u003cp\u003eRecall of the multi-step LLM workflow according to the human-in-the-loop verification of LLM suggestions\u003c/p\u003e","description":"","filename":"image3.png","url":"https://assets-eu.researchsquare.com/files/rs-9231981/v1/efc5daa9c41cc506dee7120a.png"},{"id":106402202,"identity":"7cf6d122-a8ff-4a7d-97bd-72d0b3ae5fd6","added_by":"auto","created_at":"2026-04-08 09:11:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2715099,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9231981/v1/dd7a8276-3816-4482-b22c-3b974c463840.pdf"},{"id":105728926,"identity":"edb90cd3-db72-4396-985f-48a741492177","added_by":"auto","created_at":"2026-03-30 11:13:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":409190,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementalFigure1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9231981/v1/2a2cb8b3345d8cbef7829c8b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Accelerating metadata annotation in collaborative research centers: A hybrid AI workflow for biomedical entities","fulltext":[{"header":"Introduction","content":"\u003cp\u003eLong-term research hubs like Germany's Collaborative Research Centers (CRCs) rely on structured data sharing for FAIR-compliant reuse [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. This necessity has intensified due to the massive growth of digital information, often termed a 'data deluge', which has shifted research data management from an optional task to a core scientific requirement [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. In biomedicine, tagging entities such as organisms or genes boosts discoverability. Since manual annotation requires considerable effort, automated named entity recognition is often used to classify terms (for instance, identifying \"mouse\" as an \"organism\") [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Because each CRC has distinct scientific requirements, a dedicated metadata schema must first be developed to reflect its specific experimental context [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. The specific targets depend on the experimental focus. Cell lines matter for in-vitro assays, but they're irrelevant in population studies. Only once this schema is in place can researchers begin manually annotating their datasets. This 'metadata bottleneck' frequently leads to the reality of 'empty archives,' as researchers often prioritize immediate research demands over labor-intensive documentation [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. While research data management is indispensable, it should be designed to be as effortless and unobtrusive as possible, enabling researchers to concentrate on their scientific work.\u003c/p\u003e \u003cp\u003eWe outline the implementation of the AI-augmented workflow, into an existing research data management system [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The AI system is an LLM-based, search-augmented pipeline designed to identify and annotate biomedical entities, particularly focusing on handling multiple datasets across different public repositories. This workflow uses a two-step strategy that first identifies datasets, then extracts data primarily from curated repository landing pages as the source of truth.\u003c/p\u003e \u003cp\u003eThe target environment, fredato, is a standardized metadata collection system built as a wrapper over GitLab and Nextcloud [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], utilized by Collaborative Research Centers. This existing system already relies on a specialized, hierarchical metadata schema defined using a bottom-up approach with domain experts [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. This strategy balances the need for consortium-wide standardization with the high granularity required for specialized biomedical data, such as disease models and readouts [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe aim to implement a hybrid model [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], where LLMs accelerate the data documentation process [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. By deploying the model as a pre-annotation assistant, the system overcomes the 'blank slate problem' and significantly reduces 'creation fatigue' for professional researchers [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. To be effective, the AI workflow must adhere to the existing schema's logic [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], while maintaining data quality through human validation [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eThe implementation of an AI-augmented metadata annotation workflow is based on the search-augmented LLM pipeline that was tested previously [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], and the existing metadata annotation framework within the fredato research data management system [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The study follows a human-in-the-loop approach (see Appendix S1 for the detailed operational workflow), where artificial intelligence functions as a pre-annotation assistant to accelerate traditionally labor-intensive documentation processes while remaining under the epistemic oversight of the responsible scientist [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe pipeline uses Gemini 3.0 Pro to identify and annotate biomedical entities from public repositories via a two-step prompting strategy. To enable accurate text processing, PDF manuscripts were first converted into Markdown format (using Mistral OCR or Docling). First, the model scans the formatted text and references to identify dataset deposits and capture stable identifiers such as GEO or SRA URLs. In the second step, the model visits the curated repository landing pages, treated as the primary source of truth. Retrieval was performed using a headless browser (Playwright) to render dynamic JavaScript content. The extracted HTML was programmatically cleaned to remove boilerplate elements (e.g., navigation bars, styling) and truncated to fit within the model's context window (detailed cleaning specifications and fallback mechanisms are described in Appendix S2) to populate fields in the specialized, hierarchical metadata schema.\u003c/p\u003e \u003cp\u003eThe AI-generated suggestions are then integrated into fredato, a web-based metadata editor built as a wrapper over GitLab and Nextcloud that utilizes JSON schema definitions. To ensure data integrity and distinguish between human and machine inputs, the system implements a granular provenance tracking mechanism that tags individual metadata objects with their origin and validation status (see Appendix S4). To prevent context poisoning caused by the schema's complexity, we employed a schema flattening strategy. Instead of feeding the full JSON structure to the LLM, the schema was converted into a tree-like textual representation using short identifiers (see Appendix S3). The model\u0026rsquo;s output was then mapped back to the strict JSON format. This implementation respects the hierarchical and conditional logic of the domain-specific metadata schema [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], where specific entries (e.g., selecting a particular \"tissue source\") trigger the requirement for refined sub-fields. Within the interface, authors are presented with pre-filled metadata fields. In an email-based interactive validation loop, the model\u0026rsquo;s outputs are presented to the authors of the articles as editable suggestions rather than final records. Authors act as the \"human in the loop,\" reviewing the suggestions with the ability to accept, modify, or delete entries. This step ensures the final data remains accurate, allowing scientists to correct potential model hallucinations or generalizations [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe primary objective of the evaluation is to validate the LLM's performance against high-quality labels generated by subject matter experts (the authors) during the validation phase. By tracking how authors interact with the pre-filled forms, we derive effect measures based on specific user actions. We mapped user actions to performance metrics: accepted suggestions count as True Positives (TP), deleted suggestions as False Positives (FP), manually added entities as False Negatives (FN).\u003c/p\u003e \u003cp\u003eBased on these interactions, the following performance metrics are calculated:\u003c/p\u003e \u003cp\u003e \u003cstrong\u003ePrecision\u003c/strong\u003e \u003cp\u003eProportion of accepted suggestions. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\:Precision=\\frac{TP}{TP+FP}\\)\u003c/span\u003e\u003c/span\u003e​\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eRecall\u003c/strong\u003e \u003cp\u003eAbility to identify required entities. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\:Recall=\\frac{TP}{TP+FN}\\)\u003c/span\u003e\u003c/span\u003e​\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eF1 Score\u003c/strong\u003e \u003cp\u003eHarmonic mean of precision and recall. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:F1=\\frac{2*TP}{2*TP+FN+FP}\\)\u003c/span\u003e\u003c/span\u003e​\u003c/p\u003e \u003c/p\u003e \u003cp\u003eThis methodology allows the study to overcome the typical \"open-ended\" limitation of recall measurements in scientific annotation by providing a structured environment where human experts act as the \"gold standard\" for every required field [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Because the evaluation is performed over a highly hierarchical metadata schema (342 levels, see [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]), most fields are correctly left blank, yielding a very large number of true negatives. Metrics that heavily incorporate true negatives (such as accuracy) are therefore not reported, as they are less informative for the intended decision context.\u003c/p\u003e \u003cp\u003eTo evaluate the effectiveness of the AI-augmented workflow, the three performance metrics, Precision, Recall, and F1 Score are analyzed using the statistical methodology used in previous approaches [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. In detail, these metrics are treated as proportions derived from the comparison between the initial AI-generated suggestions and the final \"gold standard\" labels validated by the authors within the fredato system. To estimate the overall performance across the different research articles and datasets, meta-analyses of single proportions are conducted [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Due to the expected small sample sizes per article and the likelihood of proportions being near 1, the Freeman\u0026ndash;Tukey double arcsine transformation is applied to stabilize the variance [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. A random-effects model using the restricted maximum likelihood (REML) estimator is employed to calculate aggregate estimates and their corresponding 95% confidence intervals, accounting for potential variability between different scientific domains or metadata tasks.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eThe workflow was rolled out in December 2025, with reminder emails sent at 5 weeks and 10 weeks after rollout. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e summarizes the yield of the multi-step workflow across 51 screened articles (40 original articles and 11 review articles). A repository deposit was identified by the LLM for 31 articles; for 17 of these articles, authors provided feedback, enabling human-in-the-loop verification of the LLM suggestions for 39 datasets. For the remaining pathway, 9 articles had no repository identified by the LLM; in this branch, only one author provided additional information for 1 unpublished dataset; however, from four articles in the repository pathway, 11 unpublished datasets were added on top of the datasets already published in public repositories. No human verification was obtained at the time of writing for a total of 22 articles (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003ePerformance evaluation was based on the 39 datasets with human verification. Across these datasets, the number of true positives (TP) averaged 13.15 (SD 4.57; range 6\u0026ndash;27). False positives (FP) were rare, with a mean of 0.23 (SD 0.58; range 0\u0026ndash;2). False negatives (FN) were also low, with a mean of 1.46 (SD 1.93; range 0\u0026ndash;6).\u003c/p\u003e \u003cp\u003ePrecision was consistently high across datasets (mostly 100%), with an overall random-effects estimate of 99.65% (95% CI 98.42% to 100.00%) and no detectable heterogeneity (I\u0026sup2; = 0.00%; Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Recall showed more variability, with an overall estimate of 93.75% (95% CI 89.79% to 96.96%) and moderate heterogeneity (I\u0026sup2; = 55.08%; Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). The combined performance, expressed as the F1 score, yielded an overall estimate of 96.17% (95% CI 93.78% to 98.11%) with moderate heterogeneity (I\u0026sup2; = 61.16%; Supplemental Fig.\u0026nbsp;1).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn the initial evaluation of the LLM suggestions for 39 distinct datasets, the workflow maintained robust performance comparable to the exploratory phase. Preliminary analysis of user interactions reveals that the Gemini 3.0 Pro model achieved an aggregate precision of approximately 99.65%, meaning authors accepted the vast majority of pre-filled fields without modification. By contrast, the recall was lower at roughly 93.75%. This gap reflects instances where the model missed entities that weren't explicitly listed on the repository landing page, requiring authors to manually add specific details. Therefore, while the high precision ensured that authors rarely had to delete incorrect suggestions (false positives), the recall metrics show that human expertise is still necessary to catch missing information (false negatives). Ultimately, the workflow successfully shifted the user's role from drafting to reviewing, significantly reducing the time required for metadata documentation while ensuring high schema compliance.\u003c/p\u003e \u003cp\u003eIntegrating this AI workflow into fredato illustrates the practical benefits of a hybrid approach. By pre-filling identifiers and entity tags, the model addresses the \"blank slate\" issue, reducing the workload for researchers authoring multiple records [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Crucially, the workflow maintains data integrity by requiring scientists to review the output, providing a checkpoint to correct errors or nuances the model may miss [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. This approach balances automation with expert verification, compressing handling time while capturing necessary corrections [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe technically forced human-in-the-loop design not only represents a significant step in addressing the \"metadata bottleneck\" [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], but also follows established frameworks for responsible automation, positioning artificial intelligence as a context-sensitive amplifier intended to support rather than supplant human expertise [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Such structured interactions are critical safeguards against model hallucinations, ensuring that high-stakes research data remain trustworthy and auditable [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe use of a highly granular, hierarchical metadata schema is instrumental to this process, as it acts as a semantic backbone that constrains model inference through typed fields and controlled vocabularies [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Our results support the paradigm shift where schemas evolve from static data entry forms into high-dimensional \u003cb\u003e\"\u003c/b\u003einstruction sets\u003cb\u003e\"\u003c/b\u003e for agentic workflows [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. In this model, the AI does not merely extract text but actively validates its decisions against schema constraints in real-time [Engel]. Furthermore, reformatting complex metadata dictionaries into structured JSON schemas has been shown to significantly improve a model's ability to correctly process and extract numerous features simultaneously [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. These schema-based predictions allow for high-accuracy concept extraction that was previously unfeasible in purely manual routines [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eRegarding model selection, while the debate between general-purpose and specialized LLMs remains nuanced, our findings suggest that generalist models like Gemini, when properly grounded through appropriate data, can achieve performance comparable to trained human annotators [Balasubramanian Wood]. While domain-specific fine-tuning often yields superior accuracy on niche terminology, generalist foundation models inherit strong generalization capabilities that are frequently lost in over-specialized architectures [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] The high precision (99.65%) achieved in our pilot is vital for knowledge discovery, as it minimizes the risk of misleading hypotheses [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. However, the moderate recall (93.75%) highlights a persistent challenge where models may overlook entities in documents with high information density, reinforcing the necessity for expert post-validation to capture missing entries [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eDespite these promising results, significant limitations exist regarding the isolated situation within a CRC. The schemas utilized are often highly granular but specific to individual research centers, which can lead to limited vocabulary overlap and potential friction when datasets deviate from the consortium's primary focus [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Such isolation runs the risk of creating siloed datasets that remain difficult to harmonize across broader institutional networks. Finally, the external validity of this pilot is constrained by its relatively small and homogeneous sample size [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. To ensure the long-term utility and safe adoption of these AI-augmented routines, broader validation across multi-site cohorts and more diverse scientific domains is required to account for the heterogeneous reporting practices found across the wider biomedical literature [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eOne of the primary concerns identified in recent pilots is the non-deterministic nature of Large Language Models (LLMs), which produce variable outputs that are highly version-dependent [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. This lack of deterministic consistency makes exact replication of metadata extraction results difficult unless specific model versions and prompts are strictly pinned [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Additionally, the \"black box\" phenomenon of AI reasoning continues to impact trust; without transparent and explainable decision pathways, expert clinicians and researchers may be hesitant to rely on AI-generated metadata for high-stakes evidence synthesis [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Establishing robust post-deployment monitoring is therefore essential to detect model drift and ensure that the AI continues to perform safely and accurately as documentation practices evolve [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eAcknowledgements\u003c/p\u003e\n\u003cp\u003eThe article processing charge is funded by the German Research Foundation (DFG) and the Albert Ludwig University of Freiburg in the funding program Open Access Publishing.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis work was supported by the DFG (German Research Foundation) through the following projects: Project-ID 441891347 \u0026ndash; SFB 1479, Project-ID 499552394 \u0026ndash; SFB 1597, Project-ID 491676693 \u0026ndash; TRR 359, Project-ID 256073931 \u0026ndash; SFB 1160, Project-ID 514483642 \u0026ndash; TRR 384, Project-ID 259373024 \u0026ndash; TRR 167, Project-ID 431984000 \u0026ndash; SFB 1453 and EXC 2189. The funding body has had no role in the design of the study or collection, analysis, or interpretation of data or in writing the manuscript.\u003c/p\u003e\n\u003cp\u003eAuthors\u0026rsquo; contributions\u003c/p\u003e\n\u003cp\u003eKK and MW designed the study. MW\u0026nbsp;developed the software. KK and MW\u0026nbsp;analyzed\u0026nbsp;the data and wrote the manuscript.\u0026nbsp;All authors (MW, FE, AK, CG, KS, SWF, MS, HB and KK) contributed to interpretation of the data, read and approved the final version of the manuscript.\u003c/p\u003e\n\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eThis study did not involve human subjects research as no identifiable personal data were collected or processed. The analyses were based solely on aggregated usage data of a software tool. Therefore, ethical approval and informed consent were not required according to applicable institutional and national regulations.\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials\u003c/p\u003e\n\u003cp\u003eThe datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eWilkinson MD, Dumontier M, Aalbersberg IjJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:1\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKanza S, Knight NJ. Behind every great research project is great data management. BMC Res Notes. 2022;15:20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13104-022-05908-5\u003c/span\u003e\u003cspan address=\"10.1186/s13104-022-05908-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEngel F, Giuliani C, Watter M, Kalantari A, Schuller K, Binder H, et al. From manual entry to machine precision: challenges and evolution of metadata schema development in collaborative research centers. 2026. ResearchSquare. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.21203/rs.3.rs-8711752/v1\u003c/span\u003e\u003cspan address=\"10.21203/rs.3.rs-8711752/v1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNadeau D, Sekine S. A survey of named entity recognition and classification. In: Sekine S, Ranchhod E, editors. Benjamins Current Topics. Amsterdam: John Benjamins Publishing Company; 2009. p. 3\u0026ndash;28. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1075/bct.19.03nad\u003c/span\u003e\u003cspan address=\"10.1075/bct.19.03nad\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWatter M, Kahle L, Brunswiek B, Fichtner U, Pfaffenlehner M, Werner F, et al. Standardized metadata collection to reinforce collaboration in Collaborative Research Centers. E-Sci-Tage 2023 Empower Your Res - Preserve Your Data. 2023;\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.11588/heidok.00033131\u003c/span\u003e\u003cspan address=\"10.11588/heidok.00033131\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEngel F, Benadi G, Giuliani C, Werner J, Watter M, Zeiser R, et al. Development of Metadata Schemas For Collaborative Research Centers. FreiData. 2025;\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.60493/k1xe3-npc10\u003c/span\u003e\u003cspan address=\"10.60493/k1xe3-npc10\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. https://doi.org/10.60493/k1xe3-npc10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaier K, Engel F, Benadi G, Giuliani C, Watter M, Kalantari A, et al. Identification of biomedical entities from multiple repositories using a specialized metadata schema and search-augmented large language models. BMC Res Notes. 2026;19:43.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnthuvan T, Prabhuram S, Maheshwari K, Rathi S. SLR\u003csup\u003eAI\u003c/sup\u003e: A Systematic Review and Methodological Framework for AI-Augmented Evidence Synthesis with Human-in-the-Loop Integration. Available SSRN 5394910. 2025.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eReddy S. Generative AI in healthcare: an implementation science informed translational path on application, integration and governance. Implement Sci. 2024;19:27. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13012-024-01357-9\u003c/span\u003e\u003cspan address=\"10.1186/s13012-024-01357-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Y, Stevens D, Shah P, Jiang W, Liu M, Chen X, et al. Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs. 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2409.10702\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2409.10702\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePangakis N, Wolken S, Fasching N. Automated Annotation with Generative AI Requires Validation. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2306.00176\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2306.00176\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWatter M, Giuliani C, Benadi G, Engel F, Binder H, Kaier K. Automated Identification of Contextually Relevant Biomedical Entities with Grounded LLMs. Sci Rep. 2026;16:1952. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-026-35492-8\u003c/span\u003e\u003cspan address=\"10.1038/s41598-026-35492-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGiuliani C, Benadi G, Engel F, Werner J, Watter M, Schwarzer G, et al. Identifying Biomedical Entities for Datasets in Scientific Articles: 4-Step Cache-Augmented Generation Approach Using GPT-4o and PubTator 3.0. JMIR Form Res. 2025;9:e73822. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2196/73822\u003c/span\u003e\u003cspan address=\"10.2196/73822\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchwarzer G, R\u0026uuml;cker G. Meta-Analysis of Proportions. In: Evangelou E, Veroniki AA, editors. Meta-Research. New York, NY: Springer US; 2022. p. 159\u0026ndash;72. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-1-0716-1566-9_10\u003c/span\u003e\u003cspan address=\"10.1007/978-1-0716-1566-9_10\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiller JJ. The Inverse of the Freeman \u0026ndash; Tukey Double Arcsine Transformation. Am Stat. 1978;32:138\u0026ndash;138. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/00031305.1978.10479283\u003c/span\u003e\u003cspan address=\"10.1080/00031305.1978.10479283\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWood V, Jeffryes M, Green AF, Blum M, Orchard S, Panni S, et al. Empowering biological knowledgebases: advances in human-in-the-loop AI-driven literature curation. Bioinforma Adv. 2026;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMortezaagha P, Shaw J, Sun B, Rahgozar A. From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs. 2025. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2601.14267\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2601.14267\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBalasubramanian JB, Adams D, Roxanis I, de Gonzalez AB, Coulson P, Almeida JS, et al. Leveraging large language models for structured information extraction from pathology reports. J Pathol Inform. 2025;:100521.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIkeda S, Zou Z, Bono H, Moriya Y, Kawashima S, Katayama T, et al. Extraction of biological terms using large language models enhances the usability of metadata in the BioSample database. GigaScience. 2025;14:giaf070.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVerbitsky A, Boutet P, Eslami M. Metadata harmonization from biological datasets with language models. Bioinforma Adv. 2025;5:vbaf241.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrokman A, Ai X, Jiang Y, Gupta S, Kavuluru R. A benchmark for end-to-end zero-shot biomedical relation extraction with LLMs: experiments with OpenAI models. In: Proceedings of the Third Workshop for Artificial Intelligence for Scientific Publications. 2025. p. 44\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJabal MS, Warman P, Zhang J, Gupta K, Jain A, Mazurowski M, et al. Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports. 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2409.10576\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2409.10576\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArtsi Y, Sorin V, Glicksberg BS, Korfiatis P, Nadkarni GN, Klang E. Large language models in real-world clinical workflows: a systematic review of applications and implementation. Front Digit Health. 2025;7:1659134.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHassan M, Kushniruk A, Borycki E. Barriers to and facilitators of artificial intelligence adoption in health care: scoping review. JMIR Hum Factors. 2024;11:e48633.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTrinkley KE, An R, Maw AM, Glasgow RE, Brownson RC. Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions. Implement Sci. 2024;19:17. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13012-024-01346-y\u003c/span\u003e\u003cspan address=\"10.1186/s13012-024-01346-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9231981/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9231981/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eCollaborative Research Centers rely on FAIR-compliant, richly structured metadata, yet manual annotation is a major bottleneck. We implemented an AI- and search-augmented large language model (LLM) workflow within a local research data management system to pre-annotate biomedical entities, using human-in-the-loop verification to ensure data quality.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThe pipeline uses Gemini 3.0 Pro for a two-step prompting strategy: (1) identify dataset deposits and stable identifiers in articles converted to Markdown; (2) extract structured fields from curated repository landing pages rendered via a headless browser. To respect a highly hierarchical metadata schema, we flattened the schema for prompting and remapped outputs to strict JSON, with granular provenance tags. Authors received pre-filled metadata and could accept, edit, or delete entries (TP, FP, FN mapping). Performance metrics (precision, recall, F1) were estimated as proportions and synthesized via random-effects meta-analysis. The workflow was rolled out in December 2025 with reminders at 5 and 10 weeks.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eAmong 51 screened articles (40 original articles, 11 review articles), the LLM identified a repository deposit in 31 articles; authors responded for 17 of these (55%), yielding 39 datasets with human verification. On the 39 verified datasets, the number of true positives averaged 13.15 (SD 4.57; range 6\u0026ndash;27). False positives were rare, with a mean of 0.23 (SD 0.58; range 0\u0026ndash;2). False negatives were also low, with a mean of 1.46 (SD 1.93; range 0\u0026ndash;6). Precision was consistently high across datasets with an overall random-effects estimate of 99.65% (95% CI 98.42% to 100.00%) and no detectable heterogeneity (I\u0026sup2; = 0.00%). Recall showed more variability, with an overall estimate of 93.75% (95% CI 89.79% to 96.96%) and moderate heterogeneity (I\u0026sup2; = 55.08%). The combined performance, expressed as the F1 score, yielded an overall estimate of 96.17% (95% CI 93.78% to 98.11%).\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eThe hybrid workflow achieved very high precision with moderately variable recall, effectively shifting effort from drafting to reviewing while preserving schema compliance. However, the modest author response rate limits sample size and generalizability; broader engagement and multi-site validation are needed to confirm robustness across domains.\u003c/p\u003e","manuscriptTitle":"Accelerating metadata annotation in collaborative research centers: A hybrid AI workflow for biomedical entities","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-30 03:44:47","doi":"10.21203/rs.3.rs-9231981/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"de367656-f8a9-424d-bf3e-feb027ae02f0","owner":[],"postedDate":"March 30th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-03T12:26:00+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-30 03:44:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9231981","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9231981","identity":"rs-9231981","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}